• iLab1.cs.rutgers.edu – iLab4.cs.rutgers.edu
• rLab1.cs.rutgers.edu – rLab6.cs.rutgers.edu

with a total of 75 GPUs.

• • 32 RTX A4000
  • 16  RTX A4500
  • 11 RTX A4500 ADA
  • 8 GeForce GTX 1080 TI
  • 8 Nvidia TITAN X.

You can use slurm job scheduler in interactive and batch mode. Each one has its own pros and cons.

NOTE: nvidia-smi command won’t show you anything on a system that has slurm running, except if you run it within a slurm job. If you want to know if there is a GPU available for your batch more, you can run srun -G 1 nvidia-smi. This should give you indication that a single GPU is available.  A special copy, nvidia-smi-priv, can be used outside slurm to see what GPUs current machine has.

• Interactive session must be run of a command line or a terminal.
• The simplest approach to using it, is to ask for an interactive session. Type

srun -G 4 --pty python3 prog.py

This allocates first available 4 gpus even though there might be other machines with free gpus.

-G indicates how many GPUs you want. Currently you can get anything from 1 to 4. If GPUs are available, you’ll get them. Note that you may end up on a different computer from where you typed the srun command, depending upon where there are free GPUs. Important: You must specify -G option or no GPU is assigned to you.

• If no GPU is free, it will wait for free GPUs and then start. In this case you might prefer to submit a batch job. (See next section.)
• Note that the command will run in a completely different context. It will start by doing a “cd” to the directory you’re currently in, but if other setup is needed, create a script that does the setup and run the script rather than runing the program directly. Of course if your setup is done automatically by .bashrc, that will work.

• If you need to use graphics, use srun --x11=first -G 4 --pty python3 prog.py. Of course you’ll need to use srun in a graphical session. Login using RDP or https://weblogin.cs.rutgers.edu.

Batch mode requires you to put your commands in a file and run it as a batch job. Once submitted, your job will start as soon as GPUs are available:

• • Put the commands you want to execute in a file, e.g. myJob
  • Submit the job using sbatch -G 4 myJob, where the number after -G is the number of GPUs you want. (See below for large-memory jobs.)
  • You can see what jobs are running using the command squeue
  • You can cancel a job using scancel NNN, where NNN is the job number shown in squeue.
  • If there are a lot of jobs in the queue, you might want to test your job to make sure you haven’t made a mistake in the file. You can use sbatch myJob, i.e. without -G. However please cancel the job once you verify that it starts properly. These systems should only be used for jobs that use GPUs

#!/bin/bash -l
#SBATCH --output=logfile
cd YOURDIR
activate YOURENV
python YOURPROGRAM  

#!/bin/bash -l 
#SBATCH --output=logfile 
cd YOURDIR 
#Important: make sure your code autorun upon container execution 
#           and terminate upon code completion 
singularity run --nv  SINGULARITY_CONTAINER.sif

• The maximum number of GPUs we let you allocate in a single job is currently 4. That will go up as we add more computers to the system. If you want to use more than 4, submit several jobs.
• If a job runs longer than a week it will be killed if others want to use the GPUs. (There have been a few cases where someone needs to run a job longer than a week. If that’s essential, let us know and we’ll find a way to avoid killing the job.)
• The scheduler also controls memory. By default, we allocate jobs 80GB of memory. As of Jan 1, 2024, the default memory will be 40GB. However you can specify less. E.g. when using sbatch add --mem=32g. In a few cases that might allow a job to run that otherwise couldn’t. You can specify up to 1TB, but if you do that your job can only run on one of the four systems, and it may have to wait if other jobs are using memory. Please do not specify large amounts of memory unless you absolutely need it, as it will limit what other people can do.
• If you look at Slurm documentation, you’ll see lots of examples where all the commands in the file start with srun. That’s not necessary or even a good idea here. Usesbatchwhenever possible!
• Because we have many kinds of GPUs, to make it easy for you to specify, we have defined the feature. To see a list of all nodes and their specific features, use sinfo -o "%25N %50f" which shows:
  

  NODELIST	AVAIL_FEATURES
  ilab[1-2]	a4500,ampere
  ilab3,rlab1	rtx4500,ada
  ilab4,rlab[2,4]	a4000,ampere
  rlab3	a4000,ampere
  rlab5	1080ti,pascal
  rlab6	titanx,pascal

• If you wanted to use RTX A4000 you could specify -C a4000 or “-C ampere. If you want to use either a 1080 TI or a TITAN X, you could specify -C '1080ti|titanx' or -C pascal. Note that OR is specified by | . Pascal and Ampere are the architectures. Cards with the same architectures have the same features, but differ in the amount of memory and number of cores.
• You can also request a specific node using -w NODE, e.g. -w rlab2.
• The only resources we control are GPUs and memory. The scheduler makes no attempt to schedule CPUs. Slurm has the ability to run a single job across multiple computers. We don’t recommend using that. Instead, use multiple jobs. If you wanted to know more details on CPUs, memory, state, weight and features, use sinfo -Nel
• You will probably want output of your program to go into a file. You can use -o FILENAME in the sbatch command to specify an output file.
• In case you need  notification both sbatch and srun have option --mail-type and --mail-user which you can utilize to notify you for certain type of events and where to send the email to.
• If you login to one of the systems and don’t use sbatch or srun, you won’t have access to any GPUs. However nvidia-smi will show you all the GPUs, so you can see what’s going on. From within a batch or srun job, nvidia-smi will only show you the GPUs you have allocated.
• You can put options in the file. E.g. rather than using sbatch -G 4 -o logfile, you could put

       #SBATCH -G 4
       #SBATCH -o logfile

in the file. All #SBATCH lines must be at the beginning of the file (right after the #!/bin/bash).

• To see list of jobs currently managed by slurm, type squeue
• To see info about your jobs, type: scontrol show job jobidwhere jobid is obtained from squeue
• To see how much memory and other parameters you had used in the past 6 months shown in MaxVMSize, type: sacct -u $USER -S now-180days -o JobID,User,MaxRSS,MaxVMsize,ReqMem,Submit,Start,State,AllocTRES,Nodelist,Reason