Using the queue system on skd-cyclone: Difference between revisions

From gfi
No edit summary
 
(17 intermediate revisions by 2 users not shown)
Line 34: Line 34:


<pre>
<pre>
  #!/bin/bash
  #
   # SLURM wrapper script
   # SLURM wrapper script
   # start this job with sbatch NAME_OF_THIS_FILE directly on the machine
   # start this job with sbatch NAME_OF_THIS_FILE directly on the machine
 
  #
   #SBATCH --job-name=my_slurm_job
   #SBATCH --job-name=my_slurm_job
   #SBATCH --workdir=/scratch/$USER
   #SBATCH --workdir=/scratch/$USER
   #SBATCH --partition=default
   #SBATCH --partition=normal
   #SBATCH --output=results.%j.txt
   #SBATCH --output=results.%j.txt
   #SBATCH --error=errors.%j.out
   #SBATCH --error=errors.%j.out
   #SBATCH --mail-type=END
   #SBATCH --mail-type=END
   #SBATCH --time=12:00:00
   #SBATCH --mail-type=FAIL
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=4


   #apply any required path settings
   #apply any required path settings
Line 65: Line 69:
<code>squeue</code>
<code>squeue</code>


4. receive an email when the job is finished
4. examine what resources are used by your job
 
<code>sstat <jobid></code>
 
5. cancel a job if required
 
<code>scancel <jobid></code>
 
6. receive an email when the job is finished


For example with the following header:
For example with the following header:
Line 84: Line 96:
  #SBATCH --<parameter>=<argument>
  #SBATCH --<parameter>=<argument>
These parameters control for example the output directory, emails, resource allocation and emailing.
These parameters control for example the output directory, emails, resource allocation and emailing.
'''The most important aspect is to determine the memory demands of your job.''' This is nicely described here:
[https://rc.fas.harvard.edu/resources/documentation/slurm-memory/ Finding out which memory limits to use]


The most common script parameters are explained here in short:
The most common script parameters are explained here in short:


*'''work_dir''': output files will be created in this directory.  
*'''mem''': how much memory your job will use at most (in MB).
Example: <code> --mem 1000</code>
*'''workdir''': output files will be created in this directory.  
Example: <code> --workdir=/scratch/$USER</code>
Example: <code> --workdir=/scratch/$USER</code>
*'''queue''': which queue the output should be processed on.
*'''queue''': which queue the output should be processed on.
Example: <code>--partition=default</code>
Example: <code>--partition=default</code>
*'''job-name''':
*'''job-name''': provide a meaningful name to your job that will be visible on the queue
Example: <code>--job-name=my_slurm_job</code>
Example: <code>--job-name=my_slurm_job</code>
*'''output''':
*'''output''': normal output will be written to this file. %j is a placeholder for the jobid
Example: <code>--output=results.%j.txt</code>
Example: <code>--output=results.%j.txt</code>
*'''error''':
*'''error''': error messages will be written to this file. %j is a placeholder for the jobid
Example: <code>--error=errors.%j.out</code>
Example: <code>--error=errors.%j.out</code>
*'''mail-type''':
*'''mail-type''': an email will be send on normal completion (parameter END) or error (parameter FAIL) or for all occations (parameter ALL). For a combination of parameters (e.g. END and FAIL) the line must be included several times.
Example: <code>--mail-type=END</code>
Example: <code>--mail-type=ALL</code>
*'''time''':
*'''time''': maximum requested computation time (12h in the example below)
Example: <code>--time=12:00:00</code>
Example: <code>--time=12:00:00</code>
*'''mem-per-cpu''':
*'''ntasks''': number of cores requested, for example for a MPI parallelisation
Example:<code>--ntasks=1</code>
*'''cpus-per-task''': number of parallel threads to be run, for example for OpenMP parallelisation
Example: <code>--cpus-per-task=4</code>
*'''mem-per-cpu''': requested memory per CPU (100MB in the example below)
Example: <code>--mem-per-cpu=100</code>
Example: <code>--mem-per-cpu=100</code>


Line 108: Line 129:
== Further information ==
== Further information ==


# Contacts
==== Contacts ====
 
In case of questions, please contact Idar Hessevik or Harald Sodemann at GFI
 
==== Links ====


Idar Hessevik
[http://tanviralamspace.blogspot.no/2015/01/slurm-tutorial-basic-commands.html A very readable blog about the SLURM queue]
Harald Sodemann


# Links
[http://www.ceci-hpc.be/slurm_tutorial.html A easy-to-understand SLURM documentation]


[[http://tanviralamspace.blogspot.no/2015/01/slurm-tutorial-basic-commands.html|A very readable blog about the SLURM queue]]
[http://slurm.schedmd.com/ The full SLURM documentation]

Latest revision as of 10:56, 22 May 2017

General information

Purpose

The machine skd-cyclone.klientdrift.uib.no is used as a general-purpose linux computer with several connected storage. Researchers, PostDocs, PhD students and Master students use the machine for calculations. On several occasions, this has lead shortages in calculation resources, concerning both memory and CPU. The machine has started to "swap" memory to the disk, slowing down all calculations on the computer to an extend where it can not be used any more.

The queuing system on skd-cyclone is in place to schedule jobs according to the available resources on the machine. This leads to a more balanced load of the machine, and avoids swapping situations.

Advantages

The main advantage of the queue system is that is makes calculations on skd-cyclone more stable and reliable, and thus overall faster.

Further advantages are:

  • Interactive run of programs still possible
  • Lightweight wrapper scripts required to run jobs on queue
  • Email on fail/completion of job
  • Queues with different priority
  • The queue at GFI is a way of introducing already MSc students to requirements of a HPC environment

Disadvantages

  • No use of screen or similar commands to interactively connect to running jobs
  • Waiting time when queue is full
  • Need to write "wrapper script" (see below)

Rules and procedures

  • The queue system is opt-in basis. That means that there is no obligation to use the queue system. Small jobs, for example, can run directly on the machine. Users who repeatedly run large jobs, however, such as NWP models or diagnostic code, will be asked to submit their job to the queue system.

Short tutorial

1. create a basic wrapper script for your job

  #!/bin/bash
  #
  # SLURM wrapper script
  # start this job with sbatch NAME_OF_THIS_FILE directly on the machine
  #
  #SBATCH --job-name=my_slurm_job
  #SBATCH --workdir=/scratch/$USER
  #SBATCH --partition=normal
  #SBATCH --output=results.%j.txt
  #SBATCH --error=errors.%j.out
  #SBATCH --mail-type=END
  #SBATCH --mail-type=FAIL
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=4

  #apply any required path settings
  export PATH=$PATH:/Data/gfi/met/bin

  #change to your working directory
  cd /Data/gfi/met/batch

  # command to start your job
  ./run_flexpart_noresm.sh

  # done
  

2. submit the script to the queue

sbatch ./my_queue_job.sh

3. monitor your jobs and the general load on the queue

squeue

4. examine what resources are used by your job

sstat <jobid>

5. cancel a job if required

scancel <jobid>

6. receive an email when the job is finished

For example with the following header:

From: slurm@klientdrift.uib.no
To: <username>@uib.no
Subject: SLURM Job_id=34 Name=WaterSip3.0 Ended, Run time 00:01:00

Converting a script to a queue script

  1. add elements to the header

Description of each element

The SLURM queue is controlled by a set of parameters at the beginning of a shell script:

#SBATCH --<parameter>=<argument>

These parameters control for example the output directory, emails, resource allocation and emailing.

The most important aspect is to determine the memory demands of your job. This is nicely described here: Finding out which memory limits to use

The most common script parameters are explained here in short:

  • mem: how much memory your job will use at most (in MB).

Example: --mem 1000

  • workdir: output files will be created in this directory.

Example: --workdir=/scratch/$USER

  • queue: which queue the output should be processed on.

Example: --partition=default

  • job-name: provide a meaningful name to your job that will be visible on the queue

Example: --job-name=my_slurm_job

  • output: normal output will be written to this file. %j is a placeholder for the jobid

Example: --output=results.%j.txt

  • error: error messages will be written to this file. %j is a placeholder for the jobid

Example: --error=errors.%j.out

  • mail-type: an email will be send on normal completion (parameter END) or error (parameter FAIL) or for all occations (parameter ALL). For a combination of parameters (e.g. END and FAIL) the line must be included several times.

Example: --mail-type=ALL

  • time: maximum requested computation time (12h in the example below)

Example: --time=12:00:00

  • ntasks: number of cores requested, for example for a MPI parallelisation

Example:--ntasks=1

  • cpus-per-task: number of parallel threads to be run, for example for OpenMP parallelisation

Example: --cpus-per-task=4

  • mem-per-cpu: requested memory per CPU (100MB in the example below)

Example: --mem-per-cpu=100

Best practices, Q&A

Further information

Contacts

In case of questions, please contact Idar Hessevik or Harald Sodemann at GFI

Links

A very readable blog about the SLURM queue

A easy-to-understand SLURM documentation

The full SLURM documentation