Using Python on CS Linux Machines

NOTE: As of Summer 2023, the default Python is 3.10. Please note the path has changed to /koko/system/anaconda3/bin

IMPORTANT:

When running GPU jobs, we required that you use Slurm Job Scheduling software to run your GPU jobs on iLab Servers. No GPU will be available without it.
Due to specific pytorch requirement of specific Cuda version, CS conda environment will not run pytorch with GPU. To run pytorch with GPUs, please run your pytoch in container.

WARNING: Python 2.7 thru 3.7 are end of life, Python environment prior to 3.10 are provided for compatibilities with older codes. No more changes will be provided on older environment. Please update to current Python 3.10.

Python has lots of modules that come in many different versions and many different dependencies of specific version. Due to version conflicts, many of these modules conflict with other modules and modules needed by the OS. To avoid these conflicts, we have created a minimal basic unified python virtual environment for beginners Computer Science Linux users to use.

For those who have been using python for a while, we recommend that you create your own python environment so you have all the flexibility to add any modules you like and it wont be affected by software changes we do.

Using Python in CS Preset Virtual Environments

Below instruction assumes you are using bash shell. If you are not using bash as your default shell, before you can activate specific python environment above, you need to run bash and type the following: bash

To use Python on CS machines, you simply need to follow 2 steps below:

1. Set a path to python environment

To add a path type:export PATH="$PATH:/koko/system/anaconda3/bin"
Hint: To skip this step on your next login, type: conda init and source .bashrc

2. Activate the python environment of your choice.

To activate python 3.10 environment (stable-current) type: conda activate python310

Other Environments

We also have Open Source Quantum development environment known as Qiskit. To activate this environment, type: conda activate qiskit

To see list of of available virtual environments, after activating an environment type: conda env list
To exit or deactivate python virtual environment, type: conda deactivate

Installed Modules

For a list of modules already installed in specific environment, type: conda list

Note: If python modules you use need GPU, please make sure to pick machines with GPU or it wont work. Here is a list of iLab machines and Graduate machines which list machines with GPU. If your codes (likely older codes) need specific older modules and/or CUDA version not installed in our system, please see Cuda/AI Learning Tools page where you are likely going to need to use your own containers to run your old codes.

Missing modules in the environment

We tried to put as many current working version of widely used modules as we can in these environments, If there are specific module you need that are not part of the virtual environment you are using, there are two options.

Create your own environment, see below. If your code needs lots of modules we don’t have, consider creating your own environment.
If your issue is older codes requiring specific version of python, older modules and/or CUDA version, please see Cuda/AI Learning Tools page.
If you need current version of modules not installed on our environment, Please let us know to request the module to be installed. We will try to install these module if they are not conflicting with existing software in a few days or so. Please be aware, major changes that requires major component upgrade, example CUDA version upgrade, will not be done mid semester.

Adding packages using pip

If you need a few packages that we don’ t have installed, you can use
pip install --user PACKAGE This will load the package into ~/.local/lib/pythonX.Y/site-packages in your home directory.

This will work both for the system python and the anaconda environments. While anaconda has its own package manager, conda, pip will still work. Conda does not have an equivalent of –user. So if you want to install anaconda packages for yourself, you’ll have to create your own environment, as described in the next section. The disadvantage is that Anaconda environments have their own separate installation of python, so they are big. pip install --user just installs the one package you need.

In either case, note that packages are specific to the version of python you are using. If you change to a different version you’ll have to install any packages you’ve installed via pip again.

Creating your own environment Using Conda

To create your own conda environment or clone an existing environment, you can follow steps below. If your issue is CUDA version, please see Cuda/AI Learning Tools page.

There are many ways to create virtual environment. In this example we will use conda. Note: for this to work, we assume you are using bash shell.

1. Set a path to python environment
if this is not already done, add a path by typing:
export PATH="$PATH:/koko/system/anaconda3/bin"
Hint: To skip this step on your next login, add this line to your .bashrc file

2. Activate the Environment

Lets assume we are going to clone existing python310 above. So lets activate the python319 environment by typing:
conda activate python310

3. Create a new python38 environment.
To create your own environement type:
conda create --name mypython310 python=3.10

For more info about Conda other commands see the Conda Cheat Sheet

3b. Cloning Existing Environment (skip if you are not cloning)

To clone existing environment, follow steps below.

A. Create a spec file for cloning.
Once activated, to clone existing environment, we need to get a list of all existing modules and store it in a file named python310.specs.txt. To get a list of current modules, type:

conda list --explicit > ~/python310.spec.txt

B. Cloning Existing Environment

You can clone a new environment based on existing specs. To create a clone environment based on the spec file, type:

conda create --name mypython310 --file ~/python310.spec.txt python=3.10

The above process will take time and when it finished you have just created a new environment named: mypython310.
To verify your new environment, you can type conda env list

C. Activating your cloned environment
To use your own environment now you simply type
conda activate mypython310

4. Adding Modules
To add modules to your own environment, you can use the conda command as follow:
conda install modulename
Notice, we are not using pip command as we are in a conda environment. Please refer to Conda Cheatsheets for more tips and tricks on how to use Conda.

5. Running out of storage
If you need more space, move your .conda to /common/users/your_netid/conda
Your conda environment is normally stored in your home directory under a hidden folder in~/.conda
If you run out of space in your home directory, you can move your conda storage in /common/users/your_netid disk where you have 100GB of quota using softlink. We are going to setup a link for your ~/.conda to point to /common/users/your_netid/conda using below steps:

- Move your existing conda folder named to /common/users/your_netid/conda using:
  mv ~/.conda /common/users/your_netid/conda
- Create a link for your ~/.conda to /common/users/your_netid/conda using:
  ln -s /common/users/your_netid/conda ~/.conda

Using Python in IDE.

Once your specific python environment activated, you may need Integrated Development Environment (IDE) to simplify your tasks. Here is a list of a few popular and free python IDEs:

A. Available on CS Linux machines

If you are doing DataScience, please see Data Science Facility page for more details.
idle3: a basic IDE for python3. To run idle in python3 environment type: idle3
spyder: a nice advanced IDE for python. To run spyder in your choice of python environment type: spyder
pyCharm: a limited community edition of pycharm is available. to run it, type pycharm
jupyter Notebook (iPython): an interactive way to run python code on a browser. If you are using your own environment, make sure you install a package named: jupyterlab before you can run jupyter. Jupyterlab is already installed in CS preconfigured environment.
To run jupyter, you need to open a terminal and depending on your session you would choose one of the following options. If you are in:
- Graphical Session like X2GO, XRDP. Weblogin.cs or Local login, type: jupyter notebook --ip=`hostname`
  this will start jupyter session and automatically open a browser on your graphical session.
- Non Graphical session like ssh type: jupyter notebook --ip=`hostname` --browser="none"
  this start jupyter session but tells you to copy paste URL to your local browser. NOTE: The ` ` around hostname are backwards quotes. The backward quote key is normally at the upper left of your keyboard. Important: due to firewall restriction, access to this URL is only accessible from the machine you run this notebook. It means accessing the URL from your own computer browser wont work.
- Web based Jupyter. We also have 100% Web Based system you can run on your browser by going to jupyter.cs.rutgers.edu
eclipse: a common and popular IDE. To run eclipse, type: eclipse or choose it from the Programming menu.

Using System Python

By default python 2.7 is installed on the system. To use it you simply typepython

Please be aware we don’t recommend you to use default system Python because a system update can brake your code and it does not contains many modules. If you must, Python allows you to install module for your own use that is site specific. To install your own module type pip install module_name --user

The module is installed in your own home directory in a hidden directory ~/.local/lib/python2.7/ and will use up a lot of space. To avoid running out of space, you should move your ~/.local folder to /common/users disk where you have 100GB of quota as follow:
mv ~/.local /common/users/~your_netid/local
and create a softlink to in your home directory for ~/.local using:
ln -s /common/users/~your_netid/local ~/.local

Note: Please be aware that your local modules may interfere with the modules already install in the environment you are using if they are not maintained or updated over time.

Further reading:

For help with our systems or If you need immediate assistant, visit LCSR Operator at CoRE 235 or call 848-445-2443. Otherwise, see CS HelpDesk. Don’t forget to include your NetID along with descriptions of your problem.

Department of Computer Science

Technical Services and Support

Using Python on CS Linux Machines