Using Python on CS Linux Machines

Python has lots of modules that come in many different versions and many different dependencies of specific version. Due to version conflicts, many of these modules conflict with other modules and modules needed by the OS. To avoid these conflicts, we have created a unified python virtual environment for Computer Science Linux users to use.

• Python 2.7 is end of life Dec 31, 2019, RU ready? Python 2.7 it is provided for compatibilities with older codes. No changes will be provided on python27 environment after Spring 2019. Please update to current version Python 3.8.

Using Python in CS Preset Virtual Environments

Below instruction assumes you are using bash shell. If you are not using bash as your default shell, before you can activate specific python environment above, you need to run bash and type the following: bash

To use Python on CS machines, you simply need to follow 2 steps below:

1. Set a path to python environment

To add a path type:export PATH="$PATH:/koko/system/anaconda/bin"
Hint: To skip this step on your next login, add this line to your .bashrc

2. Activate the python environment of your choice.

To activate python 3.6 environment (stable-old) type: source activate python36
To activate python 3.7 environment (stable-old) type: source activate python37
To activate python 3.8 environment (stable-current) type: source activate python38

Other Environments

We also have Open Source Quantum development environment known as Qiskit. To activate this environment, type: source activate qiskit

To see list of of available virtual environments, after activating an environment type: conda env list
To exit or deactivate python virtual environment, type: source deactivate

Installed Modules

For a list of modules already installed in specific environment, type: conda list

Note: If python modules you use need GPU, please make sure to pick machines with GPU or it wont work. Here is a list of iLab machines and Graduate machines which list machines with GPU. If your code needs CUDA version not installed in our system, please see Cuda/AI Learning Tools page.

Missing modules in the environment

We tried to put as many current working version of widely used modules as we can in these environments, If there are specific module you need that are not part of the virtual environment you are using, there are two options.

You can create your own environment, see below or request for the module to be installed so that everyone else benefit from it.
Please send email to help@cs.rutgers.edu to request the module to be installed. We will try to install these module if they are not conflicting with existing software in a few days or so. Please be aware, major changes that requires major component upgrade, example CUDA version upgrade, will not be done mid semester. If your issue is CUDA version, please see Cuda/AI Learning Tools page.

Adding packages using pip

If you need a few packages that we don’ t have installed, you can use
pip install --user PACKAGE This will load the package into ~/.local/lib/pythonX.Y/site-packages in your home directory.

This will work both for the system python and the anaconda environments. While anaconda has its own package manager, conda, pip will still work. Conda does not have an equivalent of –user. So if you want to install anaconda packages for yourself, you’ll have to create your own environment, as described in the next section. The disadvantage is that Anaconda environments have their own separate installation of python, so they are big. pip install --user just installs the one package you need.

In either case, note that packages are specific to the version of python you are using. If you change to a different version you’ll have to install any packages you’ve installed via pip again.

Creating your own environment Using Conda

If for some reason, you want to create your own environment based on the existing environment above, you can follow steps below. If your issue is CUDA version, please see Cuda/AI Learning Tools page.

There are many ways to create virtual environment. In this example we will use conda. Note: for this to work, we assume you are using bash shell.

1. Move your .conda to /common/users/your_netid/conda
Your new environment will be stored in your home directory under a hidden folder in~/.conda
The first step is to get you some disk space. Your home only have 4GB of quota and it will run out of space right away. To avoid running out of space, you should store these in /common/users/your_netid disk where you have 100GB of quota using softlink. We are going to setup a link for your ~/.conda to point to /common/users/your_netid/conda using below steps:

- Move your existing conda folder named to /common/users/your_netid/conda using:
  mv ~/.conda /common/users/your_netid/conda
- Create a link for your ~/.conda to /common/users/your_netid/conda using:
  ln -s /common/users/your_netid/conda ~/.conda

2. Set a path to python environment
if this is not already done, add a path by typing:
export PATH="$PATH:/koko/system/anaconda/bin"
Hint: To skip this step on your next login, add this line to your .bashrc file

3. Activate the Environment

Lets assume we are going to clone existing python38 above. So lets activate the python38 environment by typing:
source activate python38

4.You can skip this step if you are not cloning an existing environment. Once activated, to clone existing environment, we need to get a list of all existing modules and store it in a file named python36.specs.txt. To get a list of current modules, type:

conda list --explicit > ~/python36.spec.txt

5. Create a clone environment based on the spec file by typing

conda create --name mypython38 --file ~/python36.spec.txt python=3.8 If you dont want to clone, you can skip --file ~/python38.spec.txt

6. The above process will take time and when it finished you have just created a new environment named: mypython38.
To verify your new environment, you can type conda env list

7. To use your own environment now you simply type
source activate mypython38

8. To add modules to your own environment, you can use the conda command as follow:
conda install modulename
Notice, we are not using pip command as we are in a conda environment. Please refer to Conda Cheatsheets for more tips and tricks on how to use Conda.

Using Python in IDE.

Once your specific python environment activated, you may need Integrated Development Environment (IDE) to simplify your tasks. Here is a list of a few popular and free python IDEs:

A. Available on CS Linux machines

If you are doing DataScience, please see Data Science Facility page for more details.
idle3: a basic IDE for python3. To run idle in python3 environment type: idle3
spyder: a nice advanced IDE for python. To run spyder in your choice of python environment type: spyder
jupyter Notebook (iPython): an interactive way to run python code on a browser.
To run jupyter on you need to open a terminal and depending on your session you would choose one of the following options. If you are in:

– Graphical Session like X2Go, XRDP or Local login, type: jupyter notebook --ip=`hostname`
this will start jupyter session and automatically open a browser on your graphical session.

– Non Graphical session like ssh type: jupyter notebook --ip=`hostname` --browser="none"
this start jupyter session but tells you to copy paste URL to your local browser. NOTE: The ` ` around hostname are backwards quotes. The backward quote key is normally at the upper left of your keyboard.

Alternatively, we also have 100% Web Based system you can run on your browser by going to jupyter.cs.rutgers.edu
eclipse: a common and popular IDE. To run eclipse, type: eclipse or choose it from the Programming menu.

B. Not available on CS Linux machines

pycharm: A nice IDE free to install on your own. We don’t have licenses to install on our environment due to their license complex restrictions. Make sure you install it in /common/users/ or /common/home/storage of you will run out of space. See Storage Options for more detail.

Using System Python

By default python 2.7 is installed on the system. To use it you simply typepython

Please be aware we don’t recommend you to use default system Python because a system update can brake your code and it does not contains many modules. If you must, Python allows you to install module for your own use that is site specific. To install your own module type pip install module_name --user

The module is installed in your own home directory in a hidden directory ~/.local/lib/python2.7/ and will use up a lot of space. To avoid running out of space, you should move your ~/.local folder to /common/users disk where you have 100GB of quota as follow:
mv ~/.local /common/users/~your_netid/local
and create a softlink to in your home directory for ~/.local using:
ln -s /common/users/~your_netid/local ~/.local

Note: Please be aware that your local modules may interfere with the modules already install in the environment you are using if they are not maintained or updated over time.

Further reading:

Department of Computer Science

Technical Services and Support

Using Python on CS Linux Machines