HPC and NIRD toolkit user course March 2021

In March 2021, the Metacenter partners Sigma2, NTNU, UiB, UiO & UiT are jointly offering a hands-on course for current and future users of national HPC systems (Saga, Fram, Betzy) and the NIRD Toolkit. The course is open to all users of these systems. For individual sessions we list recommended prerequisites and learning outcomes. Recommended means, we believe you may struggle following the session if you do not know or are not familiar with what is listed. However, we leave the decision to you which session(s) you want to attend, and you're certainly welcome to just attend all sessions you are interested in.

On each course day, we will start at 08:50 and conclude at around 12:15. Each "morning" will include presentations and hands-on time on one of the actual HPC systems or the NIRD Toolkit. We recommend that participants use the afternoon on each day to repeat examples from the morning to get more experience on the systems.

Participants require access to a computer (not provided by the course organisers) with Internet connectivity and pre-installed programs to participate in the video meeting of the course (zoom, might work from a browser), to access the HPC systems (e.g., ssh or Putty), to transfer data to/from an HPC system (e.g., scp or WinSCP) and to access the NIRD Toolkit (web browser).

Note, registration will close minimum 14 days before the first lecture day to have sufficient time in providing access to HPC systems or the NIRD Toolkit if necessary.

Note 2, in order to use the NIRD Toolkit during the course you will need either a Feide account or a Feide OpenIDP guest account. You can create a Feide OpenIDP guest account at openidp.feide.no. If you're unsure about this, please contact us well in advance - minimum two weeks before the NIRD Toolkit sessions.

You can always contact us by sending an email to support@metacenter.no.

If you want to receive further information about training events, and other announcements about IT resources and services for researchers, there are a couple of information channels you can subscribe to:

The courses are free of charge.

HPC and NIRD Toolkit user course

Dates: March 16, 17, 18 and 19, 2021
Time: each day 08:50-12:15 with 3 short breaks
Location: video meeting (information will be send to registered participants), Note, sessions will be recorded and made publicly available after the course!

Participants may choose the lectures which they are interested in.

Registration

The course is free of charge and is organised by the Metacenter partners Sigma2, UiB, UiO, UiT & NTNU.

Registration is closed

Agenda

Day 1 - Welcome, Infrastructure news, HPC system details & Software installation - 16.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting

Session 1: 09:00-09:50 Welcome, Infrastructure news & HPC system details

  • Prerequisites: None
  • Learning outcomes: get to know what infrastructure is available/coming soon; understand details & differences of available HPC systems

Session 2: 10:00-10:50 Software installation - basics & sharing: EasyBuild

  • Prerequisites: Account on HPC system, know how to use UNIX shell
  • Learning outcomes: know how to find, install & use an EasyBuild package under a user-writeable location, know how to share a self-installed EasyBuild package with other users

Session 3: 11:00-11:50 Software installation - basics & sharing: pip/conda & R

  • Prerequisites: Account on HPC system, know how to use UNIX shell
  • Learning outcomes: know how to find, install & use a Python or R package under a user-writeable location, know how to share a self-installed Python or R package with other users

Session Q&A: 12:00-12:15 Questions & Answers

  • Prerequisites: none

Day 2 - Jobs, Determining resource requirements & Using storage - 17.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting

Session 1: 09:00-09:50 How to improve job scripts for better resource usage (cores and memory)

  • Prerequisites: Know what a job is; know a scripting language (e.g. bash) which is used for job scripts; know how to use UNIX shell and an editor
  • Learning outcomes: assess how many resources a job has used; be able to adjust resource requirements of a job as accurate as possible

Session 2: 10:00-10:50 Jobs: layouts, dependencies, array jobs, parameter sweep runs, access to internet, job priority, accounting

  • Prerequisites: Know what a job is and have used them already on an HPC system; know what Slurm is; know how to use UNIX shell and an editor
  • Learning outcomes: understand more details about jobs; know when to use nodes, tasks, threads and how to specify them; know when and how to use array jobs and dependencies between jobs and how these are scheduled; be able to run simple parameter sweep runs; understand job priorities; be able to determine the cost of a job (accounting); be able to access the internet within a job, e.g., to download data from a web server or clone some GitHub repository

Session 3: 11:00-11:50 Storage - Disk quota and usage, data archiving

  • Prerequisites: Account on HPC system and/or NIRD; know how to use UNIX shell; know how to do remote data transfers
  • Learning outcomes: understand characteristics of different storage areas; figure out how much disk you use and how to solve problems (e.g., "disk quota exceeded"); be able to transfer data to/from NIRD archive

Session Q&A: 12:00-12:15 Questions & Answers

  • Prerequisites: none

Day 3 - NIRD Toolkit, How to get help? & Login via ssh keys - 18.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting

Session 1: 09:00-09:50 NIRD Toolkit: Introduction & basics

  • Prerequisites: account to access the NIRD Toolkit; web browser
  • Learning outcomes: understand what the NIRD Toolkit is, what it can be used for
  • Note, limited resources for hands-on exercises. Everyone can join the session, but only first 30 registered may be able to do hands-on exercises during the course.

Session 2: 10:00-10:50 NIRD Toolkit: demos & setting up student group exercises

  • Prerequisites: account to access the NIRD Toolkit; web browser
  • Learning outcomes: how to launch an application/a service with the NIRD Toolkit; how to setup a group (e.g., of students) and give them access to an application/a service
  • Note, limited resources for hands-on exercises. Everyone can join the session, but only first 30 registered may be able to do hands-on exercises during the course.

Session 3: 11:00-11:50 Asking for help & Login via ssh keys

  • Prerequisites: 1st part (asking for help): none; 2nd part: laptop/desktop with tools to create keys and login into a remote machine (ssh-keygen, ssh-agent, ssh, PuTTY)
  • Learning outcomes: know how to get help, particularly how to send requests to the helpdesk/support and know about the existing documentation; know how to create and use ssh keys

Session Q&A: 12:00-12:15 Questions & Answers

  • Prerequisites: none

Day 4 - Containers - 19.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting

Session 1: 09:00-09:50 Containers: Essentials

  • Prerequisites: none
  • Learning outcomes: understand what a container is and how to use existing containers

Session 2: 10:00-10:50 Containers: How to use them on our HPC machines or the NIRD SP/Toolkit

  • Prerequisites: account on an HPC system or the NIRD Toolkit; web browser; understand basics of containers
  • Learning outcomes: how to launch a container on an HPC system or the NIRD Toolkit; how to access data on the file system from inside a container

Session 3: 11:00-11:50 Containers: Advanced topics

  • Prerequisites: account on an HPC system or the NIRD Toolkit; web browser; have used containers before
  • Learning outcomes: use containers more efficiently; be able to build your own containers or modify/extend existing containers

Session Q&A: 12:00-12:15 Questions & Answers

  • Prerequisites: none

Preparing your machine for the course

We assume you have the necessary tools installed on your machine and are able to use them. You need tools to login into a remote machine (e.g., ssh or PuTTY) and to transfer data to/from a remote machine (e.g., scp or WinSCP). If you need to install such tools, please see the pages of our course in October 2020.