Scheduler for GPU jobs

• iLab1.cs.rutgers.edu – iLab4.cs.rutgers.edu
• rLab[1-4].cs.rutgers.edu – rLab[6-7].cs.rutgers.edu
• A total of 71 GPUs.
  • 32 RTX A4000
  • 16 RTX A4500
  • 11 RTX A4500 ADA
  • 4 RTX A5000
  • 8 RTX A6000

You can use the slurm job scheduler in interactive and batch mode. Each one has its pros and cons. To see what each node configuration and current usage, type snodes This will give you full detail and configuration of all nodes.

NOTE: The nvidia-smi command won’t show you anything on a system with slurm running except if you run it within a slurm job. A special copy, nvidia-smi-priv, can be used outside the slurm to see the current machine’s GPUs.

If you want to know if a GPU is available for your batch, you can run
srun -G 1 nvidia-smi This should indicate that any single GPU is available.

To get a specific type of GPU specify something like: srun -G a6000:2 which says run using a two A6000 GPU.

• Interactive session must be run on a command line or a terminal.
• The simplest approach to using it is to ask for an interactive session. Type

srun -G 4 --pty python3 prog.py

This allocates the first available four gpus even though other machines might have free gpus.

-G indicates how many GPUs you want. Currently, you can get anything from 1 to 4. If GPUs are available, you’ll get them. Note that you may end up on a different computer from where you typed the srun command, depending on where there are free GPUs. Important: You must specify -G option or no GPU is assigned to you.

• If no GPU is free, it will wait for free GPUs and then start. In this case, you might prefer to submit a batch job. (See next section.)
• Note that the command will run in a completely different context. It will start by doing a “cd” to the directory you’re currently in, but if another setup is needed, create a script that does the setup and runs the script rather than running the program directly. Of course, if your setup is done automatically by .bashrc, that will work.

• If you need to use graphics, use srun --x11=first -G 4 --pty python3 prog.py. Of course, you’ll need to use srun in a graphical session. Login using RDP or https://weblogin.cs.rutgers.edu.

Batch mode requires you to put your commands in a file and run it as a batch job. Once submitted, your job will start as soon as GPUs are available:

• • Put the commands you want to execute in a file, e.g., myJob
  • Submit the job using sbatch -G 4 myJob, where the number after -G is the number of GPUs you want. (See below for large-memory jobs.)
  • You can see what jobs are running using the command squeue
  • You can cancel a job using scancel NNN, where NNN is the job number shown in squeue.
  • If there are a lot of jobs in the queue, you might want to test your job to ensure you haven’t made a mistake in the file. You can use sbatch myJob, i.e., without -G. However, please cancel the job once you verify it has started properly. These systems should only be used for jobs that use GPUs

#!/bin/bash -l
#SBATCH --output=logfile
cd YOURDIR
activate YOURENV
python YOURPROGRAM  

#!/bin/bash -l 
#SBATCH --output=logfile 
cd YOURDIR 
#Important: make sure your code autorun upon container execution 
#           and terminate upon code completion 
singularity run --nv  SINGULARITY_CONTAINER.sif

• The maximum number of GPUs we let you allocate in a single job is currently 4. That will go up as we add more computers to the system. If you want to use more than 4, submit several jobs.
• If a job runs longer than a week, it will be killed if others want to use the GPUs. (There have been a few cases where someone needs to run a job longer than a week. If that’s essential, let us know, and we’ll find a way to avoid killing the job.)
• You can use the -c option in sbatch if you need more CPUs. see sbatch documentation for more details.
• The scheduler also controls memory. By default, we allocate jobs 40GB of memory. However, you can specify less. E.g., when using sbatch, add. --mem=32g. In a few cases, that might allow a job that otherwise couldn’t run. You can specify up to 1TB, but if you do that, your job can only run on one of the limited systems, and it may have to wait if other jobs are using memory. Please do not specify large amounts of memory unless needed, as it will limit what other people can do.
• If you look at Slurm documentation, you’ll see many examples where all the commands in the file start with srun. That’s not necessary or even a good idea here. Usesbatchwhenever possible!
• Because we have many kinds of GPUs, we have defined the feature to make it easy for you to specify. To see a list of all nodes and their specific features, use sinfo -o "%25N %50f" which shows available features of each node.
• If you wanted to use RTX A4000 you could specify -C a4000 or “-C ampere . Note: ampere is the architectures. Cards with the same architectures have the same features but differ in the amount of memory and number of cores.
• You can also request a specific node using -w NODE, e.g. -w rlab2.
• The only resources we control are GPUs and memory. The scheduler does not attempt to schedule CPUs. Slurm can run a single job across multiple computers. We don’t recommend using that. Instead, use multiple jobs. If you want to know more details on CPUs, memory, state, weight, and features, use sinfo -Nel
• You will probably want the output of your program to go into a file. To specify an output file, you can use -o FILENAME in the sbatch command.
• In case you need notification, both sbatch and srun have the option --mail-type and --mail-user, which you can utilize to notify you of certain events and where to send the email.
• If you log in to one of the systems and don’t use sbatch or srun, you won’t have access to any GPUs. However, nvidia-smi will show you all the GPUs, so you can see what’s happening. From within a batch or srun job, nvidia-smi will only show you the GPUs you have allocated.
• You can put options in the file. For, rather than using sbatch -G 4 -o logfile, you could put

       #SBATCH -G 4
       #SBATCH -o logfile

in the file. All #SBATCH lines must be at the beginning of the file (right after the #!/bin/bash).

• To see a list of jobs currently managed by slurm, type squeue
• To see info about your jobs, type: scontrol show job jobid where jobid is obtained from squeue
• To see how much memory and other parameters you had used in the past six months shown in MaxVMSize, type: sacct -u $USER -S now-180days -o JobID,User,MaxRSS,MaxVMsize,ReqMem,Submit,Start,State,AllocTRES,Nodelist,Reason