Using Python on CS Linux Machines

Python has lots of modules that come in many different versions and many different dependencies of specific version. Due to version conflicts, many of these modules conflict with other modules  and modules needed by the OS. To avoid these conflicts, we have created a unified python virtual environment for Computer Science CentOS 7.x Linux users to use.

Using Python in CS Preset Virtual EnvironmentS

Note: Below instruction assumes you are using bash shell.  If you are not using bash as your default shell,  before you can activate specific python environment above, you need to run bash and type the following: bash

To use  Python on CS machines, you simply need to follow 2 steps below:

1. Set a path to python environment
– To add a path type:export PATH="$PATH:/koko/system/anaconda/bin"  
Hint: To skip this step on your next login, add this line to your .bashrc

2. Activate the python environment of your choice.

– To activate python 3.6 environment (stable) type: source activate python36
– To activate python 3.7 environment  (latest, missing modules) type: source activate python37
– To activate python 3.5 environment (old) type: source activate python35
– To activate python 2.7 environment (end of life)  type: source activate python27


Version Notices:
–  Python 2.7 is end of life Dec 31, 2019, RU ready? Python 2.7 it is provided for compatibilities with older codes. No changes will be provided on python27 environment after Spring 2019. Please update to Python 3.6
– Please use python36 environment if module(s) you are looking for is not in python35 or python37 environment. Many modules now support python3.6 and skipping python3.5 and not ready for python3.7. Warning: python35 environment  will be removed December 2019.

3. Other options
– To see list of of available virtual environments, after activating an environment type: conda env list
– To exit or deactivate python virtual environment, type:  source deactivate

4. Using Python in IDE.

Once your specific python environment activated, you may need Integrated Development Environment (IDE) to simplify your tasks. Here is a list of a few popular and free python IDEs:

A. Available on CS Linux machines

  • idle : A basic IDE  for python2. To run idle in python2 environment type: idle
  • idle3: a basic IDE for python3. To run idle in python3 environment type: idle3
  • spyder: a nice advanced IDE for python. To run spyder in your choice of python environment  type: spyder
  • jupyter Notebook (iPython): an interactive way to run python code on a browser.
    To run jupyter on you need to open a terminal and depending on your session you would choose one of the following options.
    If you are in:
    – Graphical Session like X2Go, XRDP or Local login,  type: jupyter notebook --ip=`hostname`
    this will start jupyter session and automatically open a browser on your graphical session.
    – Non Graphical session like ssh type: jupyter notebook --ip=`hostname` --browser="none"
    this start jupyter session but tells you to copy paste URL to your local browser.NOTE: The ` ` around hostname are backwards quotes. The backward quote key is normally at the upper left of your keyboard.
  • eclipse: a common and popular IDE. To run eclipse, type: eclipse or choose it from the Programming menu.

B. Not available on CS Linux machines

  • netbeans. A popular free Java IDE but python language support seems to be neglected on current release of Netbeans.
  • pycharm: A nice IDE free to install for your own computer.  We don’t have licenses to install on our environment due to their license complex restrictions.
  • Rodeo: A nice RStudio feel IDE but have issues that  have not been resolved since the founders have left. We dont have this on our system.

5. Installed Modules

For a list of modules already installed in specific environment, type:  conda list
 
Note: If python modules you use need GPU, please make sure to pick machines with GPU or it wont work. Here is a list of iLab machines and Graduate machines which list machines with GPU.
 

6. Missing modules in the environment.

We tried to put as many current working version of widely used modules as we can in these environments, If there are specific module you need that are not part of the virtual environment you are using, there are two options.
1.  You can create your own environment, see below or request for the module to be installed so that everyone else benefit from it.
2.  Please send email to help@cs.rutgers.edu to request the module to be installed. We will try to install these module if they are not conflicting with existing software in a few days or so. Please be aware, major changes that requires major component upgrade, example CUDA version upgrade,  will not be done mid semester.
 

Creating your own environment Using Conda

If for some reason, you want to create your own environment  based on the existing environment above, you can follow steps below.  There are many ways to create virtual environment. In this example we will use conda. 

Note: for this to work, we assume you are using bash shell.

  1. Move your .conda to /common/users/your_netid/conda
    Your new environment will be stored in your home directory under a hidden folder in~/.conda The first step is to get you some disk space. Your home only have 4GB of quota and it will run out of space right away. To avoid running out of space, you should store these in /common/users/your_netid disk where you have 100GB of quota using softlink. We are going to setup a link  for your ~/.conda to point to  /common/users/your_netid using below steps:
  •  create a folder named conda in /common/users/your_netid/conda using: 
     mkdir /common/users/your_netid/conda
  • Create a link for your ~/.conda to /common/users/your_netid/conda using:
    ln -s /common/users/your_netid/conda   ~/.conda 
 
2. Set a path to python environment  if this is not already done.
– To add a path type:export PATH="$PATH:/koko/system/anaconda/bin"  
Hint: To skip this step on your next login, add this line to your .bashrc file
 
3. Lets assume we are going to clone existing python36 above.  So lets activate the python35 environment by typing:
 source activate python36
 
4.You can skip this step if you are not cloning an existing environment. Once activated, to clone existing environment, we need to get a list of all existing modules and store it in a file named  python36.specs.txt.  To get a list of current modules,  type:
 conda list --explicit > ~/python36.spec.txt 
 
5. Create a clone environment based on the spec file by typing

conda create --name mypython36 --file ~/python36.spec.txt python=3.6 If you dont want to clone, you can skip --file ~/python36.spec.txt

5. The  above process will take time and when it finished you have just created a new environment named: mypython36.
To verify your new environment, you can type conda env list 

7. To use your own environment now you simply type 
source activate mypython36

8. To add modules to your own environment, you can use the conda command as follow: 
conda install modulename 
Notice,  we are not using pip command as we are in a conda environment. Please refer to Conda Cheatsheets for more tips and tricks on how to use Conda.

 

Using System Python

By default python 2.7 is installed on the system. To use it you simply typepython
Please be aware we don’t recommend you to use default system Python because a system update can brake your code and it does not contains many modules.  If you must, Python allows you to install module for your own use that is site specific.   To install your own module type  pip install module_name --user
 

The module is installed in your own home directory in a hidden directory ~/.local/lib/python2.7/ and will use up a lot of space.  To avoid running out of space, you should move your ~/.local folder to /common/users disk where you have 100GB of quota as follow:
    mv ~/.local /common/users/~your_netid/local 
and create a softlink to in your home directory for ~/.local using:
    ln -s /common/users/~your_netid/local ~/.local

Note: Please be aware that  your local modules may interfere with the modules already install in the environment you are using if they are not maintained or updated over time.
 
Happy Coding!