Python has lots of modules that come in many different versions and many different dependencies of specific version. Due to version conflicts, many of these modules conflict with other modules and modules needed by the OS. To avoid these conflicts, we have created a unified python virtual environment for Computer Science Linux users to use.
Using Python in CS Preset Virtual Environments
Below instruction assumes you are using bash shell. If you are not using bash as your default shell, before you can activate specific python environment above, you need to run bash and type the following: bash
To use Python on CS machines, you simply need to follow 2 steps below:
1. Set a path to python environment
To add a path type:export PATH="$PATH:/koko/system/anaconda/bin"
Hint: To skip this step on your next login, add this line to your .bashrc
2. Activate the python environment of your choice.
To activate python 3.6 environment (stable) type: source activate python36
To activate python 3.7 environment (stable) type: source activate python37
To activate python 3.8 environment (stable) type: source activate python38
Other Environments
We also have Open Source Quantum development environment known as Qiskit. To activate this environment, type: source activate qiskit
To see list of of available virtual environments, after activating an environment type: conda env list
To exit or deactivate python virtual environment, type: source deactivate
Installed Modules
conda list
Missing modules in the environment
- You can create your own environment, see below or request for the module to be installed so that everyone else benefit from it.
- Please send email to help@cs.rutgers.edu to request the module to be installed. We will try to install these module if they are not conflicting with existing software in a few days or so. Please be aware, major changes that requires major component upgrade, example CUDA version upgrade, will not be done mid semester. If your issue is CUDA version, please see Cuda/AI Learning Tools page.
Adding packages using pip
pip install --user PACKAGE
This will load the package into ~/.local/lib/pythonX.Y/site-packages
in your home directory.pip install --user
just installs the one package you need.Creating your own environment Using Conda
If for some reason, you want to create your own environment based on the existing environment above, you can follow steps below. If your issue is CUDA version, please see Cuda/AI Learning Tools page.
There are many ways to create virtual environment. In this example we will use conda. Note: for this to work, we assume you are using bash shell.
1. Move your .conda to /common/users/your_netid/conda
Your new environment will be stored in your home directory under a hidden folder in~/.conda
The first step is to get you some disk space. Your home only have 4GB of quota and it will run out of space right away. To avoid running out of space, you should store these in /common/users/your_netid disk where you have 100GB of quota using softlink. We are going to setup a link for your ~/.conda to point to /common/users/your_netid/conda using below steps:
- Move your existing conda folder named to /common/users/your_netid/conda using:
mv ~/.conda /common/users/your_netid/conda
- Create a link for your ~/.conda to /common/users/your_netid/conda using:
ln -s /common/users/your_netid/conda ~/.conda
- Move your existing conda folder named to /common/users/your_netid/conda using:
if this is not already done, add a path by typing:
export PATH="$PATH:/koko/system/anaconda/bin"
Hint: To skip this step on your next login, add this line to your .bashrc file
source activate python36
conda list --explicit > ~/python36.spec.txt
conda create --name mypython36 --file ~/python36.spec.txt python=3.6
If you dont want to clone, you can skip --file ~/python36.spec.txt
6. The above process will take time and when it finished you have just created a new environment named: mypython36.
To verify your new environment, you can type conda env list
7. To use your own environment now you simply typesource activate mypython36
8. To add modules to your own environment, you can use the conda command as follow:conda install modulename
Notice, we are not using pip command as we are in a conda environment. Please refer to Conda Cheatsheets for more tips and tricks on how to use Conda.
Using Python in IDE.
Once your specific python environment activated, you may need Integrated Development Environment (IDE) to simplify your tasks. Here is a list of a few popular and free python IDEs:
A. Available on CS Linux machines
- If you are doing DataScience, please see Data Science Facility page for more details.
- idle : A basic IDE for python2. To run idle in python2 environment type:
idle
- idle3: a basic IDE for python3. To run idle in python3 environment type:
idle3
- spyder: a nice advanced IDE for python. To run spyder in your choice of python environment type:
spyder
- jupyter Notebook (iPython): an interactive way to run python code on a browser.
To run jupyter on you need to open a terminal and depending on your session you would choose one of the following options.
If you are in:
– Graphical Session like X2Go, XRDP or Local login, type:jupyter notebook --ip=`hostname`
this will start jupyter session and automatically open a browser on your graphical session.
– Non Graphical session like ssh type:jupyter notebook --ip=`hostname` --browser="none"
this start jupyter session but tells you to copy paste URL to your local browser. NOTE: The ` ` around hostname are backwards quotes. The backward quote key is normally at the upper left of your keyboard. - eclipse: a common and popular IDE. To run eclipse, type:
eclipse
or choose it from the Programming menu.
B. Not available on CS Linux machines
- netbeans. A popular free Java IDE but python language support seems to be neglected on current release of Netbeans.
- pycharm: A nice IDE free to install for your own computer. We don’t have licenses to install on our environment due to their license complex restrictions.
- Rodeo: A nice RStudio feel IDE but have issues that have not been resolved since the founders have left. We dont have this on our system.
Using System Python
python
pip install module_name --user
The module is installed in your own home directory in a hidden directory ~/.local/lib/python2.7/ and will use up a lot of space. To avoid running out of space, you should move your ~/.local folder to /common/users disk where you have 100GB of quota as follow:mv ~/.local /common/users/~your_netid/local
and create a softlink to in your home directory for ~/.local using:ln -s /common/users/~your_netid/local ~/.local
Further reading:
- Cuda/AI Learning Tools in CS Linux systems.
- Using Virtual Environments in Jupyter Notebook and Python
- Jupyter Notebook for Beginners: A Tutorial
- Adding An Environment to Jupyter Notebooks
- 10 tips on using Jupyter Notebook