Skip to Main Content

Department of Computer Science

Technical Services and Support

Scheduler for GPU jobs

Scheduler for GPU Jobs

We have a set of machines intended for GPU jobs and jobs that take large memory. Usage is coordinated by a scheduler, using the Slurm workload manager. Without Slurm, user wont be able to access any GPUs. The systems are on iLab1.cs.rutgers.edu – iLab4.cs.rutgers.edu and  rLab1.cs.rutgers.edurLab6.cs.rutgers.edu with a total of 70 GPUs – 32 RTX A4000, 16x RTX A4500, 24 GeForce GTX 1080 TI and 8 Nvidia TITAN X.

It doesn’t matter which system you log into. The scheduler will put your job on a system with free resources. The scheduler tries to give each user a fair share of the system. It also gives priority to jobs that are shorter or use fewer GPUs. If the resources are available, you may run several jobs, up to its individual limit, at the same time.

With Slurm, the Limitation Enforced on CS Linux Machines for long job and memory do not apply to jobs scheduled via sbatch or srun. You tell the scheduler how many GPUs (up to 4), how much memory you need (default to 80GB and up to about 1TB) and how long your job will last (up to 7 days), It enforces the limits. As of Jan 1, 2024, the default memory will be 40GB.

Special Version Requirements

You may need to use a specific version of Conda, pytorch, or other software that is different from what we have installed. For this we recommend using a container using Singularity.

Machine without Scheduler

If you need a machine with GPUs with first come and first served policy, you can use iLabU.cs.rutgers.edu which has 8 x GeForce GTX 2080ti These 8 GPUs are not managed by Slurm.

Getting Started

You can use slurm job scheduler in interactive and batch mode. Each one has its own pros and cons.

NOTE: nvidia-smi won’t show you anything on a system that has slurm running, except if you run it within a slurm job. A special copy, nvidia-smi-priv, can be used outside slurm.

Interactive Session (for testing)

Interactive session must be run of a command line or a terminal.

    • The simplest approach to using it, is to ask for an interactive session. Type

srun -G 4 --pty python3 prog.py

This allocates first available 4 gpus even though there might be other machines with free gpus.

-G indicates how many GPUs you want. Currently you can get anything from 1 to 4. If GPUs are available, you’ll get them. Note that you may end up on a different computer from where you typed the srun command, depending upon where there are free GPUs.

    • If no GPU is free, it will wait for free GPUs and then start. In this case you might prefer to submit a batch job. (See next section.)
    • Note that the command will run in a completely different context. It will start by doing a “cd” to the directory you’re currently in, but if other setup is needed, create a script that does the setup and run the script rather than runing the program directly. Of course if your setup is done automatically by .bashrc, that will work.
    • If you need to use graphics, use srun --x11=first -G 4 --pty python3 prog.py. Of course you’ll need to use srun in a graphical session. Login using RDP or https://weblogin.cs.rutgers.edu.
Batch Mode (recommended)

Batch mode requires you to put your commands in a file and run it as a batch job. Once submitted, your job will start as soon as GPUs are available:

    • Put the commands you want to execute in a file, e.g. myJob
    • Submit the job using sbatch -G 4 myJob, where the number after -G is the number of GPUs you want. (See below for large-memory jobs.)
    • You can see what jobs are running using the command squeue
    • You can cancel a job using scancel NNN, where NNN is the job number shown in squeue.
    • If there are a lot of jobs in the queue, you might want to test your job to make sure you haven’t made a mistake in the file. You can use sbatch myJob, i.e. without -G. However please cancel the job once you verify that it starts properly. These systems should only be used for jobs that use GPUs

What goes in your batch file

The file you submit with sbatch must contain every command you need to execute your program.

    • Remember, it may run on a different computer. It needs all the commands you’d have to type after logging in to get to the point where you can run.
    • It must begin with #!/bin/bash. We recommend using #!/bin/bash -l  (That’s a lowercase L, not a one.) That will cause it to read your .bash_profile, etc.
    • At a minimum it needs a cd to get to the directory with your files.
    • If you’re using python in an anaconda environment it needs “activate” for that environment.
    • You should probably include #SBATCH --output=FILE unless you prefer to type --output when you submit the job
    • Here’s an example
#!/bin/bash -l
#SBATCH --output=logfile

cd YOURDIR
activate YOURENV
python YOURPROGRAM
 
Additional Details
  • The maximum number of GPUs we let you allocate in a single job is currently 4. That will go up as we add more computers to the system. If you want to use more than 4, submit several jobs.
  • If a job runs longer than a week it will be killed if others want to use the GPUs. (There have been a few cases where someone needs to run a job longer than a week. If that’s essential, let us know and we’ll find a way to avoid killing the job.)
  • The scheduler also controls memory. By default, we allocate jobs 80GB of memory. As of Jan 1, 2024, the default memory will be 40GB. However you can specify less. E.g. when using sbatch add --mem=32g. In a few cases that might allow a job to run that otherwise couldn’t. You can specify up to 1TB, but if you do that your job can only run on one of the four systems, and it may have to wait if other jobs are using memory. Please do not specify large amounts of memory unless you absolutely need it, as it will limit what other people can do.
  • If you look at Slurm documentation, you’ll see lots of examples where all the commands in the file start with srun. That’s not necessary or even a good idea here. Usesbatchwhenever possible!
  • We have three kinds of GPU, which are known as RTX A4000, GeForce GTX 1080 TI and Nvidia TITAN X. However to make it easy for you to specify, we have defined the feature as below:
    • RTX A4000 as a4000 and ampere. These cards are in iLab1.cs – iLab4.cs
    • GeForce GTX 1080 TI as 1080ti and pascal. These cards are in rLab1.cs – rLab3.cs
    • Nvidia TITAN X as titanx and pascal. These cards are in rlab4.cs
  • If you wanted to use RTX A4000 you could specify -C a4000 or “-C ampere. If you want to use either a 1080 TI or a TITAN X, you could specify -C '1080ti|titanx' or -C pascal. Note that OR is specified by | . Pascal and Ampere are the architectures. Cards with the same architectures have the same features, but differ in the amount of memory and number of cores. To see a list of all nodes and their specific features, use sinfo -Nel. Note that the column with features is truncated.
  • You can also request a specific node using -w NODE, e.g. -w rlab2.
  • The only resources we control are GPUs and memory. The scheduler makes no attempt to schedule CPUs. Slurm has the ability to run a single job across multiple computers. We don’t recommend using that. Instead, use multiple jobs.
  • You will probably want output of your program to go into a file. You can use -o FILENAME in the sbatch command to specify an output file.
  • In case you need  notification both sbatch and srun have option --mail-type and --mail-user which you can utilize to notify you for certain type of events and where to send the email to.
  • If you login to one of the systems and don’t use sbatch or srun, you won’t have access to any GPUs. However nvidia-smi will show you all the GPUs, so you can see what’s going on. From within a batch or srun job, nvidia-smi will only show you the GPUs you have allocated.
  • You can put options in the file. E.g. rather than using sbatch -G 4 -o logfile, you could put
       #SBATCH -G 4
       #SBATCH -o logfile

in the file. All #SBATCH lines must be at the beginning of the file (right after the #!/bin/bash).

Jobs Information
  • To see list of jobs currently managed by slurm, type squeue
  • To see info about your jobs, type: scontrol show job jobidwhere jobid is obtained from squeue
  • To see how much memory and other parameters you had used in the past 6 months shown in MaxVMSize, type: sacct -u $USER -S now-180days -o JobID,User,MaxRSS,MaxVMsize,ReqMem,Submit,Start,State,AllocTRES,Nodelist,Reason
Common Slurm commands
  • sacct: show accounting data for all jobs and job steps
  • sacctmgr: view and modify Slurm account information
  • salloc: set an interactive job allocation
  • sattach: attach to a running job step
  • sbatch: submit a batch script to Slurm
  • scancel: cancel jobs, job arrays or job steps
  • scontrol: view or modify Slurm configuration and state.
  • sdiag: show scheduling statistics and timing parameters
  • sinfo: view information about Slurm nodes and partitions.
  • sprio: show the components of a job’s scheduling priority
  • squeue: show  jobs queues
  • sreport: show reports from job accounting  and statistics
  • srun: run task(s) across requested resources
  • sshare: show the shares and usage for each user
  • sstat: show the status information of a running job/step.
  • sview: graphical user interface to view and modify Slurm state
 
 
Further Reading: 

For help with our systems or If you need immediate assistant, visit LCSR Operator at CoRE 235 or call 848-445-2443. Otherwise, see CS HelpDesk. Don’t forget to include your NetID along with descriptions of your problem.