Skip to Main Content

Department of Computer Science

Technical Services and Support

Using Python on CS Linux Machines

IMPORTANT:

  • Due to restricted licensing, please refrain from using the Anaconda environment at Rutgers unless it is used as part of a course. To avoid license violations, anaconda.org may be blocked from CS machines. If you like the Anaconda environment, there is an open-source alternative called Miniforge you should consider.
  • Due to specific pytorch requirements of specific OS and Cuda versions, the CS python environment may not run pytorch with GPU. To run pytorch with GPUs, please run your pytoch in the container.
  • When running GPU jobs, we require that you use Slurm Job Scheduling software to run your GPU jobs on iLab Servers. No GPU will be available without it.

There are many Python Tools for Managing Virtual Environments. As commonly used in many tutorials and also used by a peer education institution in NJ, we recommend venv for our users. You are not limited to venv. As you get familiar with other options, you may choose whatever fits your needs, but please read the license carefully before using it to avoid issues.

Using Python with CS Preset venv Environments 

The instructions below assume you are using a bash shell.  If you do not use bash as your default shell, you must run and type bash before activating the Python environment above.

As of Summer 2024, there are five preset Python venv environments for Python: 3.8 / 3.9 / 3.10 / 3.11, and 3.12. You need to activate the correct version to use it on CS machines. Warning: Python 3.8 (end-of-life Sept 2024) is provided just in case people are not ready to move to a newer version.

Why use a Python environment?
 
Python has many modules/packages with many different versions and dependencies. In addition, many of these modules conflict with other modules and with modules/packages needed by the OS. To avoid these conflicts, we do not install modules in the system. Instead, we have created a minimal, basic, unified Python virtual environment for beginners to use immediately.

The best practice we recommend is to create your own Python environment for each project you work on, so you have all the flexibility to add any modules you like, and it won’t be affected by the software changes we do.

 
Activate/deactivate the Python environment of your choice

We only run actively supported Python. To activate:

  • python 3.8 environment type: (Warning: end-of-life: Sept 2024)
    source /common/system/venv/python39/bin/activate
  • python 3.9 environment type: (Warning: end-of-life: Sept 2025)
    source /common/system/venv/python39/bin/activate
  • python 3.10 environment type:
    source /common/system/venv/python310/bin/activate
  • python 3.11 environment type:
    source /common/system/venv/python311/bin/activate
  • python 3.12 environment  type:
    source /common/system/venv/python312/bin/activate

To deactivate or exit your Python virtual environment, type:
deactivate

Installed modules/packages
 
For a list of modules/packages already installed in a specific environment, type:  pip list

However, if you want to build an environment with the same contexts, you can use the file requirements.txt at the top level of the venv as a starting point. Some of the syntax is odd, but it works with pip install.

 
Missing packages/modules in the environment
 
We tried to put as many popular and current versions of widely used modules/packages in preset environments as possible. If there are specific modules you need that are not part of the virtual environment you are using, there are a few options.
    1. If your code needs different modules, consider creating your environment with your packages.
    2. If you want to start with the same packages we have, you can see the list of the packages we install in “/common/system/venv/python3XY/packages.txt” as a starting point. It is intended to be used with one “pip install” command per package. It will not work as a requirements file using -r
    3. If you need modules for a class or think certain modules should be part of our default environment, please let us know to request the module be installed. We will install these modules if they do not conflict with existing software in a few days or so. Please be aware that major changes that require major component upgrades, such as the CUDA version upgrade,  will not be done mid-semester and will likely be done in the summer.
Creating your environment Using venv
 

To create your environment, you can follow the steps below.

Important Compatibility Issues
    • If you plan to use your environment on multiple versions of UbuntuOS, please make sure you create it using the oldest version of UbuntuOS. The older version won’t be able to use the environment created using a newer Operating System. You can find out the version of OS you are running by typing: lsb_release -a
    • You can not mix the Operating System. Example: environment created on Ubuntu will not run on CentOS

There are many ways to create a virtual environment. In this example, we will use venv.   Note: for this to work, we assume you are using bash shell, and we are using Python. To create your python3.12 environment type:
/usr/bin/python3.12 -m venv ~/mypython312

For help about venv type:
/usr/bin/python3.12 -m venv help

Once created, to activate it, type:
source ~/mypython312/bin/activate

To add modules/packages, type:
pip install packagename

For more info, see: Understanding Python virtual environments using venv and virtualenv

GPU Usage on CS Machines
  1. If your issue is older codes requiring a specific version of Python, older modules, and CUDA versions, please see the Cuda/AI Learning Tools page.
  2. If you need powerful GPUs on GPU Servers, we require that you use Slurm Job Scheduling software to run your GPU jobs on iLab Servers (iLab1-4, rLab1-6).  No server GPU will be available without Slurm. Desktop GPU does not require Slurm and can be used for small jobs or testing.
Using Python in IDE.

Once your specific Python environment is activated, you may need an Integrated Development Environment (IDE) to simplify your tasks. Here is a list of a few popular and free Python IDEs:

  • If you are doing data science, please see the Data Science Facility page for more details.
  • vscode: a popular IDE with support in multiple languages. To run vscode, type: code
  • spyder: a nice advanced IDE for python. To run Spyder in your choice of Python environment  type: spyder
  • pyCharm: a limited community edition of pycharm is available. To run it, type pycharm
  • eclipse: a common and popular IDE. To run Eclipse, type: eclipse or choose it from the Programming menu.
  • Jupyter Notebook (iPython): an interactive way to run Python code on a browser. If you are using your environment, install a package named jupyterlab before you can run jupyter. Jupyterlab is already installed in the CS preconfigured environment.
    To run jupyter, you need to open a terminal, and depending on your session, you would choose one of the following options. If you are in:
    • Graphical Session like Weblogin.cs or Local login, X2GO, XRDP. Type: jupyter notebook --ip=`hostname`
      This will start the jupyter session and automatically open a browser for your graphical session.
    • Non Graphical session like ssh type: jupyter notebook --ip=`hostname` --browser="none"
      This starts a jupyter session and tells you to copy and paste the URL to your local browser.
      NOTE: The ` ` around the hostname are backward quotes. The backward quote key is normally at the upper left of your keyboard.  Important: Due to firewall restrictions, this URL is only accessible from the machine you run in this notebook. This means that accessing the URL from your computer browser won’t work.
    • Web-based Jupyter.  We also have a 100% Web Based system you can run on your browser by going to jupyter.cs.rutgers.edu
Using System Python 
 
By default, on Ubuntu 22.04, python 3.10 is installed on the system. To use it, you type: python3
 

Please be aware we don’t recommend you to use system Python because 
1. It is not portable
2 A system update can break your code
3. It does not contain many modules.

The best practice is to create your own or use an existing environment.

 
If you must, Python allows you to install a site-specific module.   To install your module type:
pip install module_name --user

The module is installed in your home directory in a hidden directory, for example, for python3.10,  ~/.local/lib/python3.10/.

Storage Issue

If you have a big project or are in a tight space, you can move your local Python modules to another place.  User Python modules are stored in the ~/.local folder by default. To move it to /common/users disk where you have 100GB of quota as follows:
mv ~/.local /common/users/~your_netid/local
And create a soft link to your home directory for ~/.local using:
ln -s /common/users/~your_netid/local ~/.local

Note: Please be aware that your local modules may interfere with those already installed in your environment if they are not maintained or updated over time.
 
Removing your existing Anaconda settings

When Anaconda was set up initially, it added a code section in your~/.bashrc

To clean up,

    • Edit your~/.bashrc and remove this line
export PATH="$PATH:/koko/system/anaconda/bin"
    • Remove the anaconda section as shown below:
# <<< conda initialize >>>
...
...

# <<< conda initialize <<<
    • Logout and login again

Further reading:

  1. Cuda/AI Learning Tools in CS Linux systems.
  2. Scheduler for Long GPU Jobs
  3. Using Virtual Environments in Jupyter Notebook and Python
  4. Jupyter Notebook for Beginners: A Tutorial
  5. Adding An Environment to Jupyter Notebooks
  6. 10 tips on using Jupyter Notebook
For help with our systems or immediate assistance, visit LCSR Operator at CoRE 235 or call 848-445-2443. Otherwise, see CS HelpDesk. Don’t forget to include your NetID along with descriptions of your problem.