NOTE: As of Summer 2023, the default Python is 3.10. Please note the path has changed to /koko/system/anaconda3/bin
IMPORTANT: When running GPU jobs, we required that you use Slurm Job Scheduling software to run your GPU jobs on iLab Servers. No GPU will be available without it.
Python has lots of modules that come in many different versions and many different dependencies of specific version. Due to version conflicts, many of these modules conflict with other modules and modules needed by the OS. To avoid these conflicts, we have created a minimal basic unified python virtual environment for beginners Computer Science Linux users to use.
For those who have been using python for a while, we recommend that you create your own python environment so you have all the flexibility to add any modules you like and it wont be affected by software changes we do.
Using Python in CS Preset Virtual Environments
Below instruction assumes you are using bash shell. If you are not using bash as your default shell, before you can activate specific python environment above, you need to run bash and type the following: bash
To use Python on CS machines, you simply need to follow 2 steps below:
1. Set a path to python environment
To add a path type:export PATH="$PATH:/koko/system/anaconda3/bin"
Hint: To skip this step on your next login, add this line to your .bashrc
2. Activate the python environment of your choice.
To activate python 3.10 environment (stable-current) type: source activate python310
Other Environments
We also have Open Source Quantum development environment known as Qiskit. To activate this environment, type: source activate qiskit
To see list of of available virtual environments, after activating an environment type: conda env list
To exit or deactivate python virtual environment, type: source deactivate
Installed Modules
conda list
Missing modules in the environment
- Create your own environment, see below. If your code needs lots of modules we don’t have, consider creating your own environment.
- If your issue is older codes requiring specific version of python, older modules and/or CUDA version, please see Cuda/AI Learning Tools page.
- If you need current version of modules not installed on our environment, Please let us know to request the module to be installed. We will try to install these module if they are not conflicting with existing software in a few days or so. Please be aware, major changes that requires major component upgrade, example CUDA version upgrade, will not be done mid semester.
Adding packages using pip
pip install --user PACKAGE
This will load the package into ~/.local/lib/pythonX.Y/site-packages
in your home directory.pip install --user
just installs the one package you need.Creating your own environment Using Conda
To create your own conda environment or clone an existing environment, you can follow steps below. If your issue is CUDA version, please see Cuda/AI Learning Tools page.
There are many ways to create virtual environment. In this example we will use conda. Note: for this to work, we assume you are using bash shell.
if this is not already done, add a path by typing:
export PATH="$PATH:/koko/system/anaconda/bin"
Hint: To skip this step on your next login, add this line to your .bashrc file
source activate python38
To create your own environement type:
conda create --name mypython38 python=3.8
3b. Cloning Existing Environment (skip if you are not cloning)
Once activated, to clone existing environment, we need to get a list of all existing modules and store it in a file named python38.specs.txt. To get a list of current modules, type:
conda list --explicit > ~/python38.spec.txt
conda create --name mypython38 --file ~/python38.spec.txt python=3.8
The above process will take time and when it finished you have just created a new environment named: mypython38.
To verify your new environment, you can type conda env list
C. Activating your cloned environment
To use your own environment now you simply typesource activate mypython38
4. Adding Modules
To add modules to your own environment, you can use the conda command as follow:conda install modulename
Notice, we are not using pip command as we are in a conda environment. Please refer to Conda Cheatsheets for more tips and tricks on how to use Conda.
5. Running out of storage
If you need more space, move your .conda to /common/users/your_netid/conda
Your conda environment is normally stored in your home directory under a hidden folder in~/.conda
If you run out of space in your home directory, you can move your conda storage in /common/users/your_netid disk where you have 100GB of quota using softlink. We are going to setup a link for your ~/.conda to point to /common/users/your_netid/conda using below steps:
- Move your existing conda folder named to /common/users/your_netid/conda using:
mv ~/.conda /common/users/your_netid/conda
- Create a link for your ~/.conda to /common/users/your_netid/conda using:
ln -s /common/users/your_netid/conda ~/.conda
- Move your existing conda folder named to /common/users/your_netid/conda using:
Using Python in IDE.
Once your specific python environment activated, you may need Integrated Development Environment (IDE) to simplify your tasks. Here is a list of a few popular and free python IDEs:
A. Available on CS Linux machines
- If you are doing DataScience, please see Data Science Facility page for more details.
- idle3: a basic IDE for python3. To run idle in python3 environment type:
idle3
- spyder: a nice advanced IDE for python. To run spyder in your choice of python environment type:
spyder
- pyCharm: a limited community edition of pycharm is available. to run it, type
pycharm
- jupyter Notebook (iPython): an interactive way to run python code on a browser. If you are using your own environment, make sure you install a package named: jupyterlab before you can run jupyter. Jupyterlab is already installed in CS preconfigured environment.
To run jupyter, you need to open a terminal and depending on your session you would choose one of the following options. If you are in:- Graphical Session like X2GO, XRDP. Weblogin.cs or Local login, type:
jupyter notebook --ip=`hostname`
this will start jupyter session and automatically open a browser on your graphical session. - Non Graphical session like ssh type:
jupyter notebook --ip=`hostname` --browser="none"
this start jupyter session but tells you to copy paste URL to your local browser.
NOTE: The ` ` around hostname are backwards quotes. The backward quote key is normally at the upper left of your keyboard. - Web based Jupyter. We also have 100% Web Based system you can run on your browser by going to jupyter.cs.rutgers.edu
- Graphical Session like X2GO, XRDP. Weblogin.cs or Local login, type:
- eclipse: a common and popular IDE. To run eclipse, type:
eclipse
or choose it from the Programming menu.
Using System Python
python
pip install module_name --user
The module is installed in your own home directory in a hidden directory ~/.local/lib/python2.7/ and will use up a lot of space. To avoid running out of space, you should move your ~/.local folder to /common/users disk where you have 100GB of quota as follow:mv ~/.local /common/users/~your_netid/local
and create a softlink to in your home directory for ~/.local using:ln -s /common/users/~your_netid/local ~/.local
Further reading:
- Cuda/AI Learning Tools in CS Linux systems.
- Scheduler for Long GPU Jobs
- Using Virtual Environments in Jupyter Notebook and Python
- Jupyter Notebook for Beginners: A Tutorial
- Adding An Environment to Jupyter Notebooks
- 10 tips on using Jupyter Notebook
For help with our systems or If you need immediate assistant, visit LCSR Operator at CoRE 235 or call 848-445-2443. Otherwise, see CS HelpDesk. Don’t forget to include your NetID along with descriptions of your problem.