Using the queue system on skd-cyclone
The machine skd-cyclone.klientdrift.uib.no is used as a general-purpose linux computer with several connected storage. Researchers, PostDocs, PhD students and Master students use the machine for calculations. On several occasions, this has lead shortages in calculation resources, concerning both memory and CPU. The machine has started to "swap" memory to the disk, slowing down all calculations on the computer to an extend where it can not be used any more.
The queuing system on skd-cyclone is in place to schedule jobs according to the available resources on the machine. This leads to a more balanced load of the machine, and avoids swapping situations.
The main advantage of the queue system is that is makes calculations on skd-cyclone more stable and reliable, and thus overall faster.
Further advantages are:
- Interactive run of programs still possible
- Lightweight wrapper scripts required to run jobs on queue
- Email on fail/completion of job
- Queues with different priority
- The queue at GFI is a way of introducing already MSc students to requirements of a HPC environment
- No use of screen or similar commands to interactively connect to running jobs
- Waiting time when queue is full
- Need to write "wrapper script" (see below)
Rules and procedures
- The queue system is opt-in basis. That means that there is no obligation to use the queue system. Small jobs, for example, can run directly on the machine. Users who repeatedly run large jobs, however, such as NWP models or diagnostic code, will be asked to submit their job to the queue system.
1. create a basic wrapper script for your job
#!/bin/bash # # SLURM wrapper script # start this job with sbatch NAME_OF_THIS_FILE directly on the machine # #SBATCH --job-name=my_slurm_job #SBATCH --workdir=/scratch/$USER #SBATCH --partition=normal #SBATCH --output=results.%j.txt #SBATCH --error=errors.%j.out #SBATCH --mail-type=END #SBATCH --mail-type=FAIL #SBATCH --ntasks=1 #SBATCH --cpus-per-task=4 #apply any required path settings export PATH=$PATH:/Data/gfi/met/bin #change to your working directory cd /Data/gfi/met/batch # command to start your job ./run_flexpart_noresm.sh # done
2. submit the script to the queue
3. monitor your jobs and the general load on the queue
4. examine what resources are used by your job
5. cancel a job if required
6. receive an email when the job is finished
For example with the following header:
From: email@example.com To: <username>@uib.no Subject: SLURM Job_id=34 Name=WaterSip3.0 Ended, Run time 00:01:00
Converting a script to a queue script
- add elements to the header
Description of each element
The SLURM queue is controlled by a set of parameters at the beginning of a shell script:
These parameters control for example the output directory, emails, resource allocation and emailing.
The most important aspect is to determine the memory demands of your job. This is nicely described here: Finding out which memory limits to use
The most common script parameters are explained here in short:
- mem: how much memory your job will use at most (in MB).
- workdir: output files will be created in this directory.
- queue: which queue the output should be processed on.
- job-name: provide a meaningful name to your job that will be visible on the queue
- output: normal output will be written to this file. %j is a placeholder for the jobid
- error: error messages will be written to this file. %j is a placeholder for the jobid
- mail-type: an email will be send on normal completion (parameter END) or error (parameter FAIL) or for all occations (parameter ALL). For a combination of parameters (e.g. END and FAIL) the line must be included several times.
- time: maximum requested computation time (12h in the example below)
- ntasks: number of cores requested, for example for a MPI parallelisation
- cpus-per-task: number of parallel threads to be run, for example for OpenMP parallelisation
- mem-per-cpu: requested memory per CPU (100MB in the example below)
Best practices, Q&A
In case of questions, please contact Idar Hessevik or Harald Sodemann at GFI