Strumenti Utente

Strumenti Sito


calcoloscientifico:userguide

Scientific Computing of the University of Parma, in collaboration with INFN
HPC User Guide

Login

In order to access the resources, you must be included in the LDAP database of the HPC management server.

Any user needs to be associatated to a research group. The current list of research groups is available here .

Requests for new access must be sent to es_calcolo<at>unipr.it by the contact person of the research group.

Any request must be validated by the representative in the scientific committee . for the research area.

Once enabled, the login is done through SSH on the login host:

ssh <name.surname>@login.hpc.unipr.it 

Password access is allowed only within the University network (160.78.0.0/16). Outside this context it is necessary to use the University VPN or access with public key authentication.

Password-less access among nodes

In order to use the cluster it is necessary to eliminate the need to use the password among nodes, using public key authentication. It is necessary to generate on login.hpc.unipr.it the pair of keys, without passphrase, and add the public key in the authorization file (authorized_keys):

Key generation. Accept the defaults by pressing enter:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

Copy of the public key into authorized_keys:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Change the authorized_keys file permission:

chmod 0600 ~/.ssh/authorized_keys

External Access with public key Authentication

The key pair must be generated with the SSH client. The private key should be protected by an appropriate passphrase (it is not mandatory but recommended). The public key must be included in your authorized_keys file on the login host.

An authorized_keys file contains a list of OpenSSH public keys, one per line. There can be no linebreaks in the middle of a key, and the only acceptable key format is OpenSSH public key format, which looks like this:

 ssh-rsa AAAAB3N[... long string of characters ...]UH0= key-comment 

If you use the SSH client for Windows PuTTY (http://www.putty.org), you need to generate the public and private key pair with PuTTYgen and save them into 2 separate files.

The private key, with extension .ppk, must be included in the Putty (or WinSCP) configuration panel:

Configuration -> Connection -> SSH -> Auth -> Private key file for authentication

Once done on Putty, return to Session:

Session -> Host name: <name.surname>@login.hpc.unipr.it -> Copy and Paste into "Saved Session" -> Save -> Open

The public key can be copied from the PuttyGen window (see picture below) and included in the .ssh/authorized_keys file on login.hpc.unipr.it:

cd .ssh/
nano authorized_keys
ctrl + X - confirm and press Enter

PuttyGEN

File transfer

SSH is the only protocol for external communication and can also be used for file transfer.

If you use a Unix-like client (Linux, MacOS X) you can use the command scp or sftp.

On Windows systems, the most used tool is WinSCP (https://winscp.net/eng/docs/introduction). During the installation of WinSCP it is possible to import Putty profiles.

Hardware

The current cluster is composed of the new computing nodes.

Computing nodes

Partition Node Name CPU Type HT #Cores MEM(GB) GPU Totale
BDWwn01-wn08 2 INTEL XEON E5-2683v4 2.1GHz 16c NO 32 DDR4-2400 128 0 256 cores
BDWwn09-wn17 2 INTEL XEON E5-2680v4 2.4GHz 14c NO 28 128 0 252 cores
BDWwn33 2 INTEL XEON E5-2683v4 2.1GHz 16c NO 32 1024 0 32 cores
BDWwn34 4 INTEL XEON E7-8880v4 2.2GHz 22c NO 88 512 0 88 cores
GPUwn41 wn42 2 INTEL XEON E5-2683v4 2.1GHz 16c NO 32 128 7 P-100 14 GPU
KNLwn51-wn54 1 INTEL XEON PHI 7250 1.4GHz 68c YES 272 196 0 1088 cores
VRTwn61-wn64 2 INTEL XEON E5-2620v4 2.1GHz 8c YES 8 64 0 32 v-cores

Peak Performance (d.p.):

Device TFlops notes
1 node BDW E5-2683 0.5 2×16 (cores) x 2.1 (GHz) x 2 (add+mul) x 4 (256 bits AVX2 / 64 bits d.p.)
1 node SKL E5-6230 2.7 4×20 (cores) x 2.1 (GHz) x 2 (add+mul) x 8 (512 bits AVX512 / 64 bits d.p.)
1 GPU P-100 4.7
1 node KNL 1.5 68 (cores) x 1.4 (GHz) x 2 (add+mul) x 8 ( 512 bits AVX512 / 64 bits d.p.)

Nodes Usage (intranet only)

Node Interconnection Intel OmniPath

Peak performance:

Bandwidth: 100 Gb/s, Latency: 100 ns.

Benchmarks: IMB, NBODY, HPL

Private area

Software

The operating system for all types of nodes is CentOS 7.X.

Environment Software (libraries, compilers e tools): Listglobale o List

Some software components must be loaded in order to be used.

To list the available modules:

module avail

To upload / download a module (example intel):

module load   intel 
module unload intel

To list the loaded modules:

module list

To unload all the loaded module:

module purge 

Storage

The login node and computing nodes share the following storage areas:

Mount Point Env. Var. Backup Quota Note Support
/hpc/home $HOME yes 50 GB Programs and data SAN near line
/hpc/group $GROUP yes 100 GB Programs and data SAN near line
/hpc/scratch $SCRATCH no max 1 month run-time data SAN
/hpc/archive $ARCHIVE no Archive NAS/tape/cloud (2019)
/hpc/share Application software and database SAN near line

Checking personal disk quota and usage on /hpc/home :

 hpc-quota-user 

Checking disk quota and usage of the group disk on /hpc/group :

 hpc-quota-group

Private Area

Acknowledgement

Remember to mention the project in the publications among the Acknowlegements:

This research benefits from the HPC (High Performance Computing) facility of the University of Parma, Italy

Old sentence, do no use: Part of this research is conducted using the High Performance Computing (HPC) facility of the University of Parma.

The authors are requested to communicate the references of the publications, which will be listed on the site.

Job Submission with Slurm

The queues are scheduled with Slurm Workload Manager.

Slurm Partitions

Cluster Partition job resources TIMELIMIT Max Running per user Max Submit per user Max nodes per job
BDW bdw 2-628 core 3-00:00:00 24 2000 8
KNL knl 2-1088 core 5-00:00:00 16 2000
GPU gpu 1-14 GPU 0-24:00:00 6 2000
VRT vrt 1 core 10-00:00:00 24 2000 1
ALL mngt Reserved

Private area : Quota Management - PBSpro - Slurm

Useful commands

Custom commands:

hpc-sinfo
hpc-show-user-account
hpc-squeue

Slurm commands:

scontrol show partition    # Display the status of the partitions 
scontrol show job <jobid>  # Display the status of a individual job
sinfo                      # Groups of nodes and partitions 
sinfo -all                 # Detailed list of nodes groups
sinfo -all | grep idle     # list free nodes
sinfo -N                   # list of nodes and their status

srun <options>             # interactive mode
sbatch <options> script.sh # batch mode
squeue                     # display jobs in the queue
squeue -all                # display jobs details in the queue
sprio                      # show dynamic priority

Main SLURM options

Partition

-p --partition=<partition name>
Default   : bdw
Example:  --partition=gpu

Job Name

-J, --job-name=<jobname>
Default :  the name of the batch script

Nodes

-N, --nodes=<number of nodes>
Example:  --nodes=2   

Do not share nodes with other running jobs:
--exclusive

CPUs

-n, --ntasks=<global number of tasks>
Example:  -N2 -n4   # 4cpus on 2 nodes

-c --cpus-per-task=<cpu per task>
Example:  -N2 -c2   # 2 nodes, each with 2 cpus

--ntasks-per-node=<number of tasks per node>
Example:  --ntasks-per-node=8  ## 8 MPI tasks per node

GPUs

 --gres=[name:type:count] (list of consumable resources)
Example: --gres=gpu:tesla:2 

Memory limit

Memory required per allocated CPU
--mem-per-cpu=<size{units}>
Example: --mem-per-cpu=1G
Default: 512M

Memory required per node (as an alternative to --mem-per-cpu)
--mem =<size{units}>
Example:  --mem=4G

Time limit

Maximum execution time of the job. 
-t, --time=<days-hours:minutes:seconds>
Default: --time=0-01:00:00
Example: --time=1-00:00:00

Account

Specifies the account to be charged for the used resources
-A --account=<account_name>
Default: see the command  hpc-show-user-account
Example: --account=T_HPC18A

See also TeamWork

Output and error

Output and error file name
-o, --output=<filename pattern>
-e, --error=<filename pattern>
Example: --output=%x.o%j --error=%x.e%j 
###  %x=job name, %j=jobid
Default: stdout and stderr are written in a file named  slurm-<jobid>.out

E-mail

One or more e-mail addresses, separated by commas, that will receive the notifications from the queue manager.
--mail-user=<mail address>
Default: University e-mail address

Events  generating  the notification.
--mail-type=<FAIL, BEGIN, END, TIME_LIMIT_50, TIME_LIMIT_90, NONE, ALL>
Default:  NONE

Example:  --mail-user=john.smith@unipr.it   --mail-type=BEGIN,END

Export variables

Export new variable to the batch job.
--export=<environment variables>
Example: --export threads="8" 

Priority

The priority (from queue to execution) is dynamically defined by three parameters:

  • Timelimit
  • Aging (waiting time in partition)
  • Fair share (amount of resources used in last 14 days per account)

Advanced Resource Reservation

It is possible to define an advanced reservation for teaching activities or special requests.

To view active reservations:

 scontrol show reservation 

To use a reservation:

 sbatch  --reservation=<reservation name>    nomescript.sh  ## command line
 

or

 #SBATCH   --reservation=<reservation name>                 ## inside the script

Interactive example:

srun     --reservation=corso_hpc18a --account=T_HPC18A  -N1 -n1 -p vrt --pty bash
...
exit

Batch example:

sbatch  --reservation=corso_hpc18a --account=T_HPC18A   -N1 -n2  nomescript.sh

In order to request a reservation send an e-mail to es_calcolo@unipr.it.

Reporting/Accounting

Reporting

Synopsis:
  hpc-report [--user=<user>] [--partition=<partition>] [--accounts=<accounts>] [--starttime=<start_time>] [--endtime=<end_time>]
  hpc-report --help

Usage:
  hpc-report
  hpc-report --start=<start_time> --end=<end_time>

Example:
  hpc-report --start=2018-05-07 --end=2018-05-21
  hpc-report --start=2018-05-07 --end=2018-05-21 --user=$USER

Options:
  --user=<user>             Filter results per user
  --partition=<partition>   Filter results per partition
  --accounts=<accounts>     Filter results per accounts
  --starttime=<start_time>  Filter results per start date
  --endtime=<end_time>      Filter results per end date
  --help                    Display a help message and quit

To list user jobs executed in the last three days:

hpc-lastjobs

Accounting

Interactive jobs

srun --partition=<partition> --nodes=<nodes number> --nodelist=<nodename> --account=<accountname> <other slurm options> --pty bash
... some interactive activity on the node ...
exit

Examples:

# Reserve 4 cores on 2 node on cluster BDW 
srun -N2 --ntasks-per-node=4  --pty bash
echo "$SLURM_JOB_NODELIST"       # assigned nodes list
echo "$SLURM_NTASKS_PER_NODE"    # number of task per node
scontrol show job $SLURM_JOB_ID  # job details
exit
 
# reserve 1 core on node wn34. Defaults: partition=bdw, timelimit=72:00, account=user's default, mem=500MB 
srun -N1 --nodelist=wn34   --pty bash
 
# 1 chunk of 272 CPUs type KNL (a whole KNL node)
srun -N1 -n272 -p knl --mem=8G --pty bash
 
# 1 chunk with 1 GPU on GPU Cluster
srun -N1 -p gpu --gres=gpu:tesla:1  --mem=4G --pty bash
echo "#CUDA_VISIBLE_DEVICES : $CUDA_VISIBLE_DEVICES"
  #CUDA_VISIBLE_DEVICES : 6
exit
 
# 2 cores on different nodes
srun -N2 --ntasks-per-node=1  --pty bash
 
# 2 cores on the same node
srun -N1 --ntasks-per-node=2  --pty bash

Batch job

The commands to be executed on the node must be included in a shell script. To submit the job:

sbatch  scriptname.sh

Slurm options can be specified in the command line or in the first lines of the shell script (see examples below).

Each job is assigned a unique numeric identifier <Job Id>.

At the end of the execution a file containing stdout and stderr will be created in the directory from which the job was submitted.

By default the file in named slurm-<job id>.out

Serial jobs on VRT

OpenMP example program: calculation of the product of two matrices

cp /hpc/share/samples/serial/mm.cpp .

Script mm.bash for the submission stage:

#!/bin/bash
 
#SBATCH --output=%x.o%j
##SBATCH --error=%x.e%j  #If error is not specified stderr is redirected to stdout
#SBATCH --partition=vrt
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
 
## Uncomment the following line if you need an amount of memory other than default (512MB)
##SBATCH --mem=2G
 
## Uncomment the following line if your job needs a wall clock time other than default (1 hour) 
## Please note that priority of queued job decreases as requested time increases
#SBATCH --time=0-00:30:00 
 
## Uncomment the following line if you want to use an account other than your default account ( see hpc-show-user-account )
##SBATCH --account=<accountname> 
 
## Print the list of the assigned resources 
echo "#SLURM_JOB_NODELIST: $SLURM_JOB_NODELIST"
 
# default gcc version is 4.8.5. Load GNU module if you want to use a newer version of the GNU compiler
# module load gnu 
  g++ mm.cpp -o mm
  ./mm
 
#  Uncomment the following lines  to compile and run using the INTEL compiler
#  module load intel
#  icc mm.cpp -o mm_intel
#  ./mm_intel
 
# Uncomment the following lines  to compile and run using the PGI compiler
#  module load pgi
#  pgcc mm.cpp -o mm_pgi
#  ./mm_pgi

Submission

sbatch mm.bash

Monitor the job status:

hpc-squeue  ( or squeue )

To cancel the job:

scancel <Job id>

Other serial examples

ls  /hpc/share/samples/serial/

OpenMP jobs on BDW

OpenMP example Program:

cp /hpc/share/samples/omp/omp_hello.c .

Script omp_hello.bash with the request for 28 CPUs.

#!/bin/bash
 
#SBATCH --partition=bdw
#SBATCH --job-name="omp_hello-GNU_compiler"
#SBATCH --output=%x.o%j
 
##SBATCH --error=%x.e%j  #If error is not specified stderr is redirected to stdout
#SBATCH --nodes=1
#SBATCH --cpus-per-task=28
 
## Uncomment the following line if you need an amount of memory other than default (512MB)
##SBATCH --mem=2G
 
## Uncomment the following line if your job needs a wall clock time other than default (1 hour) 
## Please note that priority of queued job decreases as requested time increases
##SBATCH --time=0-00:30:00 
 
## Uncomment the following line if you want to use an account other than your default account ( see hpc-show-user-account )
##SBATCH --account=<accountname> 
 
##SBATCH --exclusive  # uncomment to require a whole node with at least 28 cores 
 
echo  "#SLURM_JOB_NODELIST      : $SLURM_JOB_NODELIST"
# Comment out the following line in case of exclusive request 
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
echo  "#OMP_NUM_THREADS        : $OMP_NUM_THREADS"
 
# Compile and run with the GNU compiler. Default gcc version is 4.8.5. 
# Load the gnu module if you want to use a newer version of the GNU compiler. 
# module load gnu 
gcc -fopenmp omp_hello.c -o omp_hello
./omp_hello
 
# Uncomment the following lines  to compile and run with the INTEL compiler
# module load intel
# icpc -qopenmp omp_hello.c -o omp_hello_intel
# ./omp_hello_intel
 
# Uncomment the following lines  to compile and run with the PGI compiler
# module load pgi 
# pgc++ -mp omp_hello.c -o omp_hello_pgi
#./omp_hello_pgi

Submission

sbatch omp_hello.bash

Other openMP examples

ls /hpc/share/samples/omp/ 

MPI Jobs on BDW

MPI example program:

cp /hpc/share/samples/mpi/mpi_hello.c .

Script mpi_hello.bash , 2 nodes with 4 CPUs per node :

#!/bin/bash
 
#SBATCH --partition=bdw
#SBATCH --job-name="mpi_hello-GNU_compiler"
#SBATCH --output=%x.o%j
##SBATCH --error=%x.e%j  #If error is not specified stderr is redirected to stdout
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4 
 
## Uncomment the following line if you need an amount of memory other than default (512MB)
##SBATCH --mem=2G
 
## Uncomment the following line if your job needs a wall clock time other than default (1 hour) 
## Please note that priority of queued job decreases as requested time increases
##SBATCH --time=0-00:30:00 
 
## Uncomment the following line if you want to use an account other than your default account ( see hpc-show-user-account )
##SBATCH --account=<accountname> 
 
echo  "#SLURM_JOB_NODELIST      : $SLURM_JOB_NODELIST"
echo  "#SLURM_CPUS_PER_TASK     : $SLURM_CPUS_PER_TASK"
echo  "#SLURM_JOB_CPUS_PER_NODE : $SLURM_JOB_CPUS_PER_NODE"
 
 
# Gnu compiler
module load gnu openmpi
mpicc mpi_hello.c -o mpi_hello
mpirun mpi_hello
 
#  Uncomment the following lines  to compile and run using the INTEL compiler
#  module load intel intelmpi
#  mpicc mpi_mm.c -o mpi_mm_intel
#  mpirun  mpi_hello_intel
 
# Uncomment the following lines  to compile and run using the PGI compiler
# module load pgi/2018 openmpi/2.1.2/2018 
# mpicc mpi_hello.c -o mpi_hello_pgi
# mpirun --mca mpi_cuda_support 0 mpi_hello_pgi

Submission

sbatch mpi_hello.bash

Other MPI examples

ls /hpc/share/samples/mpi/

MPI+OpenMP jobs on BDW

MPI + OpenMP example:

cp  /hpc/share/samples/mpi+omp/mpiomp_hello.c .

Script mpiomp_hello.bash :

#!/bin/sh
 
#< 2 nodes, 2 MPI processes per node, 6 OpenMP threads per MPI  process
 
#SBATCH --partition=bdw
#SBATCH --job-name="mpi_hello-GNU_compiler"
#SBATCH --output=%x.o%j
##SBATCH --error=%x.e%j      #If error is not specified stderr is redirected to stdout
 
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2    #< Number of MPI task per node
#SBATCH --cpus-per-task=6      #< Number of OpenMP threads per  MPI task
 
## Uncomment the following line if you need an amount of memory other than default (512MB)
##SBATCH --mem=2G
 
## Uncomment the following line if your job needs a wall clock time other than default (1 hour)
##SBATCH --time=0-00:30:00
## Please note that priority of queued job decreases as requested time increases
 
## Uncomment the following line if you want to use an account other than your default account
##SBATCH --account=<accountname>
 
echo  "#SLURM_JOB_NODELIST      : $SLURM_JOB_NODELIST"
echo  "#SLURM_CPUS_PER_TASK     : $SLURM_CPUS_PER_TASK"
 
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
 
module load gnu openmpi
mpicc -fopenmp mpiomp_hello.c -o mpiomp_hello
mpirun mpiomp_hello
 
 
## Uncomment the following lines  to compile and run with  the INTEL compiler
#module load intel intelmpi
#mpicc -qopenmp mpiomp_hello.c -o mpiomp_hello_intel
#mpirun mpiomp_hello_intel

MPI for PYTHON

Original site - examples

hello-mpi-world.py

#!/usr/bin/env python

from mpi4py import MPI

comm = MPI.COMM_WORLD
name=MPI.Get_processor_name()

print "Hello! I'm rank ",comm.rank," on node ", name, " from ", comm.size, " running in total"

comm.Barrier()   # wait for everybody to synchronize _here_

Submission example hello-mpi-world.sh:

#!/bin/bash

#SBATCH --partition=bdw
#SBATCH --output=%x.o%j
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --account=T_HPC18A 

module load gnu7  openmpi3  py2-mpi4py
#module load intel  openmpi3  py2-mpi4py

mpirun  python -W ignore hello-mpi-world.py

Matrix-Vector product:

cp /hpc/group/T_HPC18A/samples/mpi4py/matrix-vector-product.py .
cp /hpc/group/T_HPC18A/samples/mpi4py/matrix-vector-product.sh .
sbatch matrix-vector-product.sh

KNL jobs

The compiler to be used is Intel.

The selection of the KNL cluster is done by specifying “ -p knl”.

Each knl node includes 68 physical core and each physical core includes 4 virtual cores with hyperthreading technology. The maximum number of cores (ncpus) selectable per node is 272.

MMPI+openMP example on KNL:

 cp /hpc/share/samples/mpi+omp/mpiomp_hello.c .

Sumbission script mpiomp_hello_knl.bash

#!/bin/sh
 
#< 2  nodes. Executes 1 MPI process per node and 68 threads per process
 
#SBATCH --partition=knl
#SBATCH --nodes=2
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=68      #< Number of threads OpenMP for each process MPI
#SBATCH --time=0-00:30:00
#SBATCH --mem=4G
 
## Uncomment the following line if you want to use an account other than your default account 
##SBATCH --account=<accountname> 
 
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
 
module load intel intelmpi
mpicc -qopenmp mpiomp_hello.c -o mpiomp_hello_knl
mpirun  mpiomp_hello_knl

Submission

sbatch  mpiomp_hello_knl.bash 

GPU jobs

The GPU partition consists of 2 machines with 7 GPUs each. The GPUs of a single machine are identified by an integer ID that ranges from 0 to 6.

The compiler to use is nvcc:

NVIDIA CUDA Compiler

Compilation example:

cp /hpc/share/samples/cuda/hello_cuda.cu .
module load cuda
nvcc hello_cuda.cu -o hello_cuda

The GPU cluster selection is done by specifying “ -p gpu ” and “ –gres = gpu: <0-6> ” among the required resources.

Example of submission on 1 of the 7 GPUs available on a single node of the GPU cluster:

#!/bin/sh
 
#< 1 node with 1 GPU
 
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --gres=gpu:tesla:1
#SBATCH --time=0-00:30:00
#SBATCH --mem=4G
 
#< Charge resources to account
##SBATCH --account=<accountname>
 
module load cuda
 
echo "#SLURM_JOB_NODELIST   : $SLURM_JOB_NODELIST"
echo "#CUDA_VISIBLE_DEVICES : $CUDA_VISIBLE_DEVICES"
 
./hello_cuda

Example of submission of the N-BODY benchmark on all 7 GPUs available in a single node of the GPU cluster:

#!/bin/sh
 
#< 1 node with 7 GPU
 
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --gres=gpu:tesla:7
#SBATCH --time=0-00:30:00
#SBATCH --mem=3G
 
#< Charge resources to account 
##SBATCH --account=<accountname> 
 
module load cuda
 
echo "#SLURM_JOB_NODELIST   : $SLURM_JOB_NODELIST"
echo "#CUDA_VISIBLE_DEVICES : $CUDA_VISIBLE_DEVICES"
 
/hpc/share/samples/cuda/nbody/nbody -benchmark -numbodies=1024000 -numdevices=7

In the case of N-BODY, the number of GPUs to be used is specified using the -numdevices option (the specified value must not exceed the number of GPUs required with the ngpus option).

In general, the GPU IDs to be used are derived from the value of the CUDA_VISIBLE_DEVICES environment variable.

In the case of the last example we have:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6

Teamwork

To share activities with other member of your group you can enter the command

newgrp <GROUPNAME>

The newgrp command modifies the primary group and the permissions of the new files created during the current session.

Moreover newgrp on the HPC cluster automatically executes the command to enter the group directory (/hpc/group/<groupname>):

cd "$GROUP" 

In the batch mode you must indicate the account to be used with the following directive:

#SBATCH --account <account name>

To display your default account:

 hpc-show-user-account

Job Array

https://slurm.schedmd.com/job_array.html

Using a single SLURM script it is possible to submit a battery of Jobs, which can be executed in parallel, specifying a different numerical parameter for each submissive job.

The –array option specifies the numeric sequence of parameters. At each launch the value of the parameter is contained in the $ SLURM_ARRAY_TASK_ID variable

Example:

Starts N job for the computing of Pi with a number of intervals increasing from 100000 to 900000 with increment of 10000:

cp /hpc/share/samples/serial/cpi/cpi_mc.c .
gcc cpi_mc.c -o cpi_mc

Script slurm_launch_parallel.sh

#!/bin/sh
 
#SBATCH --partition=vrt
#SBATCH --array=1-10:1
 
## Charge resources to account 
##SBATCH --account=<accountname>
##SBATCH --mem=4G
 
N=$((${SLURM_ARRAY_TASK_ID}*1000000))
CMD="./cpi_mc -n $N"
echo "# $CMD"
eval $CMD
sbatch slurm_launch_parallel.sh

Gather the outputs:

grep -vh '^#' slurm_launch_parallel.sh.o*.*

MATLAB Jobs

MATLAB serial job

Execution of a MATLAB serial program

cp -p /hpc/share/samples/matlab/pi_greco.m .

Script matlab.sh:

#!/bin/sh
#SBATCH --job-name=matlab
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --partition=vrt
#SBATCH --mem=8G
#SBATCH --time=00:30:00
#SBATCH --account=<accountname>
 
module load matlab/R2017a
 
matlab -nodisplay -r pi_greco

Submitting matlab.sh:

sbatch matlab.sh

MATLAB parallel job

Execution of a parallel job with MATLAB

cp -p /hpc/share/samples/matlab/pi_greco_parallel.m .

Script matlab_parallel.sh:

#!/bin/sh
#SBATCH --job-name=matlab_parallel
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --tasks-per-node=4
#SBATCH --partition=bdw
#SBATCH --mem=8G
#SBATCH --time=0-00:30:00
#SBATCH --account=<accountname>
 
module load matlab/R2017a
 
matlab -nodisplay -r pi_greco_parallel

Submitting matlab_parallel.sh:

sbatch matlab_parallel.sh

MATLAB GPU job

Execution of a MATLAB program on GPU

In this example we see how to run the GPUBench MATLAB program on GPU. Requires the MATLAB Parallel Computing Toolbox and a GPU with CUDA Compute Capability.

Script slurm-matlab-gpu-bench.sh:

#!/bin/sh
#SBATCH --job-name=GPUBench
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --gres=gpu:tesla:1
#SBATCH --partition=gpu
#SBATCH --mem=8G
#SBATCH --time=0-00:30:00
#SBATCH --account=<accountname>
 
module load matlab/R2017a
 
cd 'GPUBench_v1.10'
 
matlab -nodisplay -nosplash -r "T=gpuBench;quit"

Submitting slurm-matlab-gpu-bench.sh:

sbatch slurm-matlab-gpu-bench.sh

Execution of two MATLAB programs on the same GPU

In this example we see how to run two instances of the GPUBench MATLAB program on the same GPU. Requires the MATLAB Parallel Computing Toolbox and a GPU with CUDA Compute Capability.

Script slurm-matlab-multi-gpu-bench.sh:

#!/bin/sh
#SBATCH --job-name=GPUBench
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --gres=gpu:tesla:1
#SBATCH --partition=gpu
#SBATCH --mem=8G
#SBATCH --time=0-00:30:00
#SBATCH --account=<accountname>
 
module load matlab/R2017a
 
cd 'GPUBench_v1.10'
 
pids=""
 
for i in $(seq 0 1); do
  matlab -nodisplay -nosplash -r "T=gpuBench;quit" \
  1>"$SLURM_SUBMIT_DIR/${SLURM_JOB_NAME}.o${SLURM_JOB_ID}.${i}" \
  2>"$SLURM_SUBMIT_DIR/${SLURM_JOB_NAME}.e${SLURM_JOB_ID}.${i}" &
  pids="${pids:+$pids }$!"
done
 
echo "Waiting PID(s): $pids"
 
wait $pids

Submitting slurm-matlab-multi-gpu-bench.sh:

sbatch slurm-matlab-multi-gpu-bench.sh

Mathematica Jobs

Mathematica serial job

Execution of a Mathematica serial program

Script math-vrt.m:

Integrate[1/(x^4 - a^4), x]

(*Prints all Mersenne prime numbers less than 10000*)
Print[Select[Range[10000],PrimeQ[2^#-1]&]];

A = Sum[i, {i,1,100}]
B = Mean[{25, 36, 22, 16, 8, 42}]
Answer = A + B

Quit[];

Script slurm-math-vrt.sh:

#!/bin/sh
#SBATCH --job-name=MathVRT
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --ntasks=1
#SBATCH --partition=vrt
#SBATCH --mem=8G
#SBATCH --time=0-00:30:00
#SBATCH --account=<accountname>
 
module load mathematica
 
math math-vrt.m

Submitting slurm-math-vrt.sh:

sbatch slurm-math-vrt.sh

Mathematica parallel job

Execution of a parallel job with Mathematica

Script math-bdw.m:

GetEnvironment["SLURM_NTASKS_PER_NODE"]

(*Limits Mathematica to requested resources*)
Unprotect[$ProcessorCount];
$ProcessorCount = ToExpression[Environment["SLURM_NTASKS_PER_NODE"]];

(*Prints the machine name that each kernel is running on*)
Print[ParallelEvaluate[$MachineName]];

(*Prints all Mersenne PRime numbers less than 10000*)
Print[Parallelize[Select[Range[10000],PrimeQ[2^#-1]&]]];

Quit[];

Script slurm-math-bdw.sh:

#!/bin/sh
#SBATCH --job-name=MathBDW
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --tasks-per-node=8
#SBATCH --partition=bdw
#SBATCH --mem=16G
#SBATCH --time=0-00:30:00
#SBATCH --account=<accountname>
 
module load mathematica
 
math math-bdw.m

Submitting slurm-math-bdw.sh:

sbatch slurm-math-bdw.sh

Crystal14 MPI Jobs

Script crystal14.sh for submitting the MPI version of Crystal14. Requires 4 nodes from 8 cores and starts 8 MPI processes per node:

#!/bin/sh
 
#SBATCH --job-name="crystal14"        #< Job name 
#SBATCH --partition=bdw               #< Resource request
#SBATCH --nodes=4
#SBATCH --ntasks=8
#SBATCH --time=0-168:00:00
 
#< Charge resources to account 
#SBATCH --account=<accountname>
#SBATCH --mem=64G
 
#< input files directory
CRY14_INP_DIR='input'
 
#< output files directory
CRY14_OUT_DIR='output'
 
#< ouput files prefix
CRY14_INP_PREFIX='test'
 
#< input wave function file prefix
CRY14_F9_PREFIX='test'
 
source /hpc/share/applications/crystal14

We recommend creating a folder for each simulation. In each folder there must be a copy of the crystal14.sh script.

The script contains the definition of four variables:
  • CRY14_INP_DIR: the input file or files must be in the 'input' subfolder of the current directory. To use the current directory, comment the line with the definition of the CRY14_INP_DIR variable. To change subfolder, change the value of the CRY14_INP_DIR variable.
  • CRY14_OUT_DIR: the output files will be created in the 'output' subfolder of the current folder. To use the current directory, comment the line with the definition of the CRY14_OUT_DIR variable. To change subfolder modify the value of the variable CRY14_OUT_DIR.
  • CRY14_INP_PREFIX: the file or input files have a prefix that must coincide with the value of the CRY14_INP_PREFIX variable. The string 'test' is purely indicative and does not correspond to a real case.
  • CRY14_F9_PREFIX: the input file, with extension 'F9', is the result of a previous processing and must coincide with the value of the variable CRY14_F9_PREFIX. The string 'test' is purely indicative and does not correspond to a real case.

The crystal14.sh script includes, in turn, the system script / hpc / software / bin / hpc-pbs-crystal14 . The latter can not be changed by the user.

Submission of the shell script

Navigate to the folder containing crystal14.sh and run the following command to submit the script to the job scheduler:

sbatch ./crystal14.sh
Analysis of files produced by Crystal14 during job execution

During execution of the job a temporary tmp folder is created which contains the two files:

nodes.par
machines.LINUX

The nodes.par file contains the names of the nodes that participate in the parallel computing.

The machines.LINUX file contains the names of the nodes that participate in the parallel computing with a multiplicity equal to the number of MPI processes started on the node.

To locate the temporary folders produced by Crystal14 during the execution of the job, run the following command directly from the login node:

eval ls -d1 /hpc/node/wn{$(seq -s, 81 95)}/$USER/crystal/* 2>/dev/null
Be careful because the previous command contains the names of the currently available calculation nodes. This list and the corresponding command may change in the future.

To check the contents of the files produced by Crystal14 during the execution of the job, the user can move to one of the folders highlighted by the previous command.

At the end of the execution of the job, the two files machines.LINUX and nodes.par are deleted. The temporary folder tmp is deleted only if it is empty.

It is therefore not necessary to log in with SSH to the nodes participating in the processing to check the contents of the files produced by Crystal14.

Job Gromacs

Several versions of Gromacs are available:

Compilation details:

Gromacs Plumed Compiler MPI CUDA SDK CUDA RTL CUDA Architectures
4.5.7 GNU 5.4.0 OpenMPI 1.10.6
4.5.7 2.3.2 GNU 5.4.0 OpenMPI 1.10.6
5.1.4 2.3.2 GNU 5.4.0 OpenMPI 1.10.6
2016.3 2.3.2 GNU 5.4.0 OpenMPI 1.10.6 8.0.61 8.0.61.2 6.0
2016.4 2.3.5 GNU 5.4.0 OpenMPI 1.10.6 8.0.61.2 8.0.61.2 6.0
2018.6 2.4.6 GNU 5.4.0 OpenMPI 1.10.6 10.0.130 10.0.130 6.0, 7.0
2018.6 2.5.3 GNU 7.3.0 OpenMPI 3.0.0 10.0.130 10.0.130 6.0, 7.0
2019.4 2.4.6 GNU 5.4.0 OpenMPI 1.10.6 10.0.130 10.0.130 6.0, 7.0
2019.4 2.5.3 GNU 7.3.0 OpenMPI 3.0.0 10.0.130 10.0.130 6.0, 7.0

For example Gromacs 2016.3 was patched with Plumed 2.3.2 and compiled with the GNU 5.4.0 tool chain and OpenMPI 1.10.6. The GPU enabled version was compiled with CUDA SDK (software development kit) 8.0.61 and is usable with CUDA RTL (run time library) 8.0.61.2. Code was generated for CUDA architecture 6.0.

Supported CUDA architectures:

GPU CUDA Architecture
Nvidia Tesla P100 6.0
Nvidia Tesla V100 7.0

Starting from the 2016.4 version, environment modules are available in different flavors:

Gromacs Prerequisites Flavor Module
2016.4 gnu/5.4.0 openmpi/1.10.6 bdw 2016.4-bdw
2016.4 gnu/5.4.0 openmpi/1.10.6 gpu 2016.4-gpu
2016.4 gnu/5.4.0 openmpi/1.10.6 knl 2016.4-knl
2018.6 gnu/5.4.0 openmpi/1.10.6 bdw 2018.6-bdw
2018.6 gnu/5.4.0 openmpi/1.10.6 gpu 2018.6-gpu
2018.6 gnu/5.4.0 openmpi/1.10.6 knl 2018.6-knl
2018.6 gnu/5.4.0 openmpi/1.10.6 skl 2018.6-skl
2018.6 gnu7/7.3.0 openmpi3/3.0.0 bdw 2018.6-bdw
2018.6 gnu7/7.3.0 openmpi3/3.0.0 gpu 2018.6-gpu
2018.6 gnu7/7.3.0 openmpi3/3.0.0 knl 2018.6-knl
2018.6 gnu7/7.3.0 openmpi3/3.0.0 skl 2018.6-skl
2019.4 gnu/5.4.0 openmpi/1.10.6 bdw 2019.4-bdw
2019.4 gnu/5.4.0 openmpi/1.10.6 gpu 2019.4-gpu
2019.4 gnu/5.4.0 openmpi/1.10.6 knl 2019.4-knl
2019.4 gnu/5.4.0 openmpi/1.10.6 skl 2019.4-skl
2019.4 gnu7/7.3.0 openmpi3/3.0.0 bdw 2019.4-bdw
2019.4 gnu7/7.3.0 openmpi3/3.0.0 gpu 2019.4-gpu
2019.4 gnu7/7.3.0 openmpi3/3.0.0 knl 2019.4-knl
2019.4 gnu7/7.3.0 openmpi3/3.0.0 skl 2019.4-skl

GMXLIB environment variable

To define the GMXLIB environment variable, add the following lines to the file $HOME/.bash_profile:

GMXLIB=$HOME/gromacs/top
 
export GMXLIB
The path $HOME/gromacs/top is purely indicative. Modify it according to your preferences.

Job Gromacs OpenMP

This will initiate a single process and will result in suboptimal performance.

Gromacs 5.1.4

Script mdrun-omp.sh to exclusively request a node and start multiple OpenMP threads (tested with –cpus-per-task=14 and –cpus-per-task=16):

#!/bin/bash                                                                        
#SBATCH --job-name=mdrun_omp
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=14
#SBATCH --exclusive
#SBATCH --time=0-24:00:00
#SBATCH --mem=120G
#SBATCH --partition=bdw
#
# Charge resources to account
#SBATCH --account=<accountname>
 
test -n "$SLURM_NODELIST" || exit
 
module load gnu/5.4.0
module load openmpi/1.10.6
 
source '/hpc/share/applications/gromacs/5.1.4/mpi_bdw/bin/GMXRC'
 
# Set OMP_NUM_THREADS to the same value as --cpus-per-task with a fallback in   
# case it isn't set. SLURM_CPUS_PER_TASK is set to the value of --cpus-per-task,
# but only if --cpus-per-task is explicitly set.                                 
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
 
gmx mdrun -deffnm topology -pin on

Gromacs 2016.4

Script mdrun-omp.sh to exclusively request a node and start multiple OpenMP threads (tested with –cpus-per-task=14 and –cpus-per-task=16):

#!/bin/bash                                                                        
#SBATCH --job-name=mdrun_omp
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=14
#SBATCH --exclusive
#SBATCH --time=0-24:00:00
#SBATCH --mem=120G
#SBATCH --partition=bdw
#
# Charge resources to account
#SBATCH --account=<accountname>
 
test -n "$SLURM_NODELIST" || exit
 
module load gnu/5.4.0
module load openmpi/1.10.6
module load gromacs/2016.4-bdw
 
# Set OMP_NUM_THREADS to the same value as --cpus-per-task with a fallback in   
# case it isn't set. SLURM_CPUS_PER_TASK is set to the value of --cpus-per-task,
# but only if --cpus-per-task is explicitly set.                                 
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
 
gmx mdrun -deffnm topology -pin on -plumed plumed.dat
Plumed module is automatically loaded.

Job Gromacs MPI OpenMP

This will initiate multiple MPI processes and will achieve optimal performance.

Gromacs 5.1.4

Script mdrun-mpi-omp.sh to exclusively request one or more nodes (tested with –nodes=1 and –nodes=2) and start multiple MPI processes (tested with –ntasks-per-node=6 and –ntasks-per-node=8; the number of OpenMP threads will be calculated automatically if –cpus-per-task is not explicitly set):

#!/bin/bash
#SBATCH --job-name=mdrun_mpi_omp
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=6
#SBATCH --exclusive
#SBATCH --time=0-24:00:00
#SBATCH --mem=120G
#SBATCH --partition=bdw
#
# Charge resources to account
#SBATCH --account=<accountname>
 
test -n "$SLURM_NODELIST" || exit
 
module load gnu/5.4.0
module load openmpi/1.10.6
 
source '/hpc/share/applications/gromacs/5.1.4/mpi_bdw/bin/GMXRC'
 
# Set OMP_NUM_THREADS to the same value as --cpus-per-task with a fallback in
# case it isn't set. SLURM_CPUS_PER_TASK is set to the value of --cpus-per-task,
# but only if --cpus-per-task is explicitly set.
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
    OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
else
    if [ -n "$SLURM_NTASKS_PER_NODE" ]; then
        OMP_NUM_THREADS=$((SLURM_CPUS_ON_NODE/SLURM_NTASKS_PER_NODE))
    else
        if [ -n "$SLURM_NTASKS" ] && [ -n "$SLURM_NNODES" ]; then
            OMP_NUM_THREADS=$((SLURM_CPUS_ON_NODE/(SLURM_NTASKS/SLURM_NNODES)))
        else
            OMP_NUM_THREADS=1
        fi
    fi
fi
 
export OMP_NUM_THREADS
 
mpirun gmx mdrun -deffnm topology -pin on

Gromacs 2016.4

Script mdrun-mpi-omp.sh to exclusively request one or more nodes (tested with –nodes=1 and –nodes=2) and start multiple MPI processes (tested with –ntasks-per-node=6 and –ntasks-per-node=8; the number of OpenMP threads will be calculated automatically if –cpus-per-task is not explicitly set):

#!/bin/bash
#SBATCH --job-name=mdrun_mpi_omp
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=6
#SBATCH --exclusive
#SBATCH --time=0-24:00:00
#SBATCH --mem=120G
#SBATCH --partition=bdw
#
# Charge resources to account
#SBATCH --account=<accountname>
 
test -n "$SLURM_NODELIST" || exit
 
module load gnu/5.4.0
module load openmpi/1.10.6
module load gromacs/2016.4-bdw
 
# Set OMP_NUM_THREADS to the same value as --cpus-per-task with a fallback in
# case it isn't set. SLURM_CPUS_PER_TASK is set to the value of --cpus-per-task,
# but only if --cpus-per-task is explicitly set.
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
    OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
else
    if [ -n "$SLURM_NTASKS_PER_NODE" ]; then
        OMP_NUM_THREADS=$((SLURM_CPUS_ON_NODE/SLURM_NTASKS_PER_NODE))
    else
        if [ -n "$SLURM_NTASKS" ] && [ -n "$SLURM_NNODES" ]; then
            OMP_NUM_THREADS=$((SLURM_CPUS_ON_NODE/(SLURM_NTASKS/SLURM_NNODES)))
        else
            OMP_NUM_THREADS=1
        fi
    fi
fi
 
export OMP_NUM_THREADS
 
mpirun gmx mdrun -deffnm topology -pin on -plumed plumed.dat
Plumed module is automatically loaded.

Job Gromacs OpenMP GPU

Gromacs 2016.4

The mdrun-omp-cuda.sh script runs a 4 core with 1 CUDA GPU Gromacs job:

#!/bin/bash
#SBATCH --job-name=mdrun_omp_cuda
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:tesla:1
#SBATCH --time=0-24:00:00
#SBATCH --mem=16G
#SBATCH --partition=gpu
#
# Charge resources to account
#SBATCH --account=<accountname>
 
test -n "$SLURM_NODELIST" || exit
 
module load gnu/5.4.0
module load openmpi/1.10.6
module load gromacs/2016.4-gpu
 
# Set OMP_NUM_THREADS to the same value as --cpus-per-task with a fallback in
# case it isn't set. SLURM_CPUS_PER_TASK is set to the value of --cpus-per-task,
# but only if --cpus-per-task is explicitly set.
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
 
gmx mdrun -deffnm topology -pin on -dlb auto -plumed plumed.dat
Plumed ed CUDA modules are automatically loaded.

Job Desmond

Job Desmond GPU

Example script desmond-md-gpu.sh to launch Desmond multisim on 1 node, 1 core, 1 GPU:

#!/bin/sh                                                                        
#SBATCH --job-name=desmond_md_gpu
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --gres=gpu:tesla:1
#SBATCH --time=0-24:00:00
#SBATCH --mem=10G
#SBATCH --partition=gpu
#
# Charge resources to account
#SBATCH --account=<accountname>
 
test -n "$SLURM_NODELIST" || exit
 
module load cuda
module load schrodinger
 
if [ -d '/node' ]; then
  SCHRODINGER_TMPDIR='/node'
else
  SCHRODINGER_TMPDIR=$(dirname "$SCRATCH")
fi
 
export SCHRODINGER_TMPDIR
 
cd "$SLURM_SUBMIT_DIR"
 
test -n "$STAGE" || STAGE=1
 
test $STAGE -eq 1 && \
multisim \
-WAIT \
-JOBNAME "$SLURM_JOB_NAME" \
-m "${SLURM_JOB_NAME}-in.msj" \
-o "${SLURM_JOB_NAME}-out.cms" \
-maxjob $SLURM_NTASKS_PER_NODE \
-cpu $SLURM_NTASKS_PER_NODE \
-set stage[1].set_family.md.jlaunch_opt=[-gpu]
 
test $STAGE -gt 1 && \
multisim \
-WAIT \
-JOBNAME "$SLURM_JOB_NAME" \
-RESTART "${SLURM_JOB_NAME}-multisim_checkpoint:${STAGE}" \
-d "${SLURM_JOB_NAME}_$((STAGE-1))-out.tgz" \
-o "${SLURM_JOB_NAME}-out.cms" \
-maxjob $SLURM_NTASKS_PER_NODE \
-cpu $SLURM_NTASKS_PER_NODE \
-set stage[1].set_family.md.jlaunch_opt=[-gpu]

For the first stage of the Desmond multisim workflows you can use:

sbatch desmond-md-gpu.sh

To pass the value of the variable STAGE into the job script named desmond-md-gpu.sh you can use:

sbatch --export=STAGE=2 desmond-md-gpu.sh

Job Abaqus

Before you run your batch script, you may execute

newgrp G_ABAQUS

Now your current working directory is /hpc/group/G_ABAQUS. Create and use your own sub-directory.

Load Abaqus module

module load abaqus

and submit from this new shell.

Command line options

  • job: job name
  • input: input file
  • user: user privided source file or object file
  • scratch: full path name of the directory to be used for scratch files. The default value on Linux is the value of the $TMPDIR environment variable or /tmp if $TMPDIR is not defined. During the analysis a subdirectory will be created under this directory to hold the analysis scratch files. The name of the subdirectory is constructed from the user's user name, the job ID, and the job's process number. The subdirectory and its contents are deleted upon completion of the analysis.

Job Abaqus MPI

Example script slurm-abaqus.sh to run Abacus on 1 node, 6 cores, 0 GPUs:

#!/bin/bash
#SBATCH --job-name=abaqus
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=28
#SBATCH --mem=94G
#SBATCH --time=3-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
echo "$SLURM_JOB_NODELIST"
 
# Modules required by Abacus (choose one of the following two possibilities)
#  module load intel intelmpi abaqus
module load gnu openmpi abaqus
 
ulimit -s unlimited
ulimit -s
 
abaqus job=testverita interactive ask_delete=OFF cpus=$SLURM_NTASKS_PER_NODE

Job Abaqus MPI with GPU

Example script slurm-abaqus-gpu.sh to run Abacus on 1 node, 6 cores, 1 GPU:

#!/bin/bash
#SBATCH --job-name=abaqus_gpu
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=6
#SBATCH --gres=gpu:tesla:1
#SBATCH --mem=30G
#SBATCH --time=24:00:00
#SBATCH --partition=gpu
#
# Edit next line
#SBATCH --account=<accountname> 
 
echo "$SLURM_JOB_NODELIST"
 
# Modules required by Abacus (choose one of the following two possibilities)
# module load intel intelmpi cuda abaqus
module load gnu openmpi cuda abaqus
 
ulimit -s unlimited
ulimit -s
 
abaqus job=testverita interactive ask_delete=OFF cpus=$SLURM_NTASKS_PER_NODE gpus=1

Job Abaqus MPI with Fortran

Example script slurm-abaqus-fort.sh to run Abacus on 1 node, 2 cores, 0 GPUs with Fortran subroutine:

#!/bin/bash
#SBATCH --job-name=abaqus_fort
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --mem=8G
#SBATCH --time=2-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
echo "$SLURM_JOB_NODELIST"
 
# Modules required by Abacus (choose one of the following two possibilities)
module load gnu intel/ps-xe-2017.2 intelmpi abaqus
# module load gnu openmpi abaqus
 
ulimit -s unlimited
ulimit -s
 
abaqus \
job="$SLURM_JOB_NAME" \
input=<input_file> \
user=<user_file> \
scratch="$SCRATCH" \
interactive \
ask_delete=OFF \
cpus=$SLURM_NTASKS_PER_NODE

Submission example (one node, more then one task per node):

sbatch --account=G_ABAQUS --job-name=MODEL12-01 --partition=bdw --ntasks-per-node=6 --mem=24G slurm-abaqus-fort.sh

Submission example (one node, one task per node):

sbatch --account=G_ABAQUS --job-name=MODEL12-02 --partition=vrt --ntasks-per-node=1 --mem=16G slurm-abaqus-fort.sh

Job Abaqus MPI with user subroutine

Example script slurm-abaqus-user.sh to run Abacus on 1 node, 2 cores, 0 GPUs with user provided subroutine:

#!/bin/bash
#SBATCH --job-name=abaqus_user
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --mem=8G
#SBATCH --time=2-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
echo "$SLURM_JOB_NODELIST"
 
# Modules required by Abacus (choose one of the following two possibilities)
module load gnu intel/ps-xe-2017.2 openmpi abaqus
#module load gnu openmpi abaqus
 
ulimit -s unlimited
ulimit -s
 
abaqus \
job="$SLURM_JOB_NAME" \
${abaqus_input:+input="$abaqus_input" }\
${abaqus_user:+user="$abaqus_user" }\
scratch="$SCRATCH" \
interactive \
ask_delete=OFF \
cpus=$SLURM_NTASKS_PER_NODE

Submission example (one node, more then one task per node):

abaqus_input=MODEL12
abaqus_user=UTIL56
sbatch --account=G_ABAQUS --job-name=${abaqus_input}-03 --partition=bdw --ntasks-per-node=6 --mem=24G abaqus_user.sh

Submission example (one node, one task per node):

abaqus_input=MODEL12
abaqus_user=UTIL56
sbatch --account=G_ABAQUS --job-name=${abaqus_input}-04 --partition=vrt --ntasks-per-node=1 --mem=16G abaqus_user.sh

Job Gaussian

Job Gaussian MPI

Example script slurm-gaussian16-mpi.sh to run Gaussian 16 on 1 node using 28 cores and at most 80 GB of memory:

#!/bin/bash
#SBATCH --job-name=gaussian16_mpi
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=28
#SBATCH --mem=80G
#SBATCH --time=1-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
test -n "$SLURM_NODELIST" || exit
test $SLURM_NNODES -eq 1 || exit
 
module load gaussian/16.B01
 
if [ ! -e "${SLURM_JOB_NAME}.com" ]; then
    echo "Error: file '${SLURM_JOB_NAME}.com' not found." 1>&2
    exit 1
fi
 
mkdir -p "$GAUSS_SCRDIR"
 
GAUSSIAN_MEM="$((SLURM_MEM_PER_NODE*90/100))MB"
 
echo -e "%NProcShared=${SLURM_TASKS_PER_NODE}\n%Mem=${GAUSSIAN_MEM}\n%chk=${SLURM_JOB_NAME}.${SLURM_JOB_ID}.chk" > "${SLURM_JOB_NAME}.${SLURM_JOB_ID}.com"
egrep -i -v '^%(NProcShared|Mem|chk)' "${SLURM_JOB_NAME}.com" >> "${SLURM_JOB_NAME}.${SLURM_JOB_ID}.com"
sed -i -r -e "s/(\-\-Link1\-\-)/\1\n%NProcShared=${SLURM_TASKS_PER_NODE}\n%Mem=${GAUSSIAN_MEM}\n%chk=${SLURM_JOB_NAME}.${SLURM_JOB_ID}.chk/g" \
"${SLURM_JOB_NAME}.${SLURM_JOB_ID}.com"
 
g16 "${SLURM_JOB_NAME}.${SLURM_JOB_ID}.com" "${SLURM_JOB_NAME}.${SLURM_JOB_ID}.log"

Example script slurm-gaussian16-mpi.sh to run Gaussian 16 on 1 node in exclusive mode:

#!/bin/bash
#SBATCH --job-name=gaussian16_mpi
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err
#SBATCH --nodes=1
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=1-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
test -n "$SLURM_NODELIST" || exit
test $SLURM_NNODES -eq 1 || exit
 
module load gaussian/16.B01
 
if [ ! -e "${SLURM_JOB_NAME}.com" ]; then
    echo "Error: file '${SLURM_JOB_NAME}.com' not found." 1>&2
    exit 1
fi
 
mkdir -p "$GAUSS_SCRDIR"
 
GAUSSIAN_MEM="$((SLURM_MEM_PER_NODE*90/100))MB"
 
echo -e "%NProcShared=${SLURM_TASKS_PER_NODE}\n%Mem=${GAUSSIAN_MEM}\n%chk=${SLURM_JOB_NAME}.${SLURM_JOB_ID}.chk" > "${SLURM_JOB_NAME}.${SLURM_JOB_ID}.com"
egrep -i -v '^%(NProcShared|Mem|chk)' "${SLURM_JOB_NAME}.com" >> "${SLURM_JOB_NAME}.${SLURM_JOB_ID}.com"
sed -i -r -e "s/(\-\-Link1\-\-)/\1\n%NProcShared=${SLURM_TASKS_PER_NODE}\n%Mem=${GAUSSIAN_MEM}\n%chk=${SLURM_JOB_NAME}.${SLURM_JOB_ID}.chk/g" \
"${SLURM_JOB_NAME}.${SLURM_JOB_ID}.com"
 
g16 "${SLURM_JOB_NAME}.${SLURM_JOB_ID}.com" "${SLURM_JOB_NAME}.${SLURM_JOB_ID}.log"

Submission example:

sbatch --job-name=test0322 slurm-gaussian16-mpi.sh
Use %nosave directive in .com input file to automatically delete the configuration binary file .chk

Job NAMD

NAMD is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems.

NAMD

Job NAMD on BDW

Job NAMD multicore (single BDW node)

Example script slurm-namd-bdw.sh to run NAMD multicore on 1 node using 16 cores and at most 32 GB of memory:

#!/bin/bash
#SBATCH --job-name=NAMD
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=16
#SBATCH --mem=32G
#SBATCH --time=1-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
module load namd/2.12/bdw
 
if [ ! -e "${SLURM_JOB_NAME}.conf" ]; then
    echo "Error: file '${SLURM_JOB_NAME}.conf' not found." 1>&2
    exit 1
fi
 
namd2 +p ${SLURM_TASKS_PER_NODE} "${SLURM_JOB_NAME}.conf" > "${SLURM_JOB_NAME}.log"

Submission example:

sbatch --job-name=t1r1_min1 slurm-namd-bdw.sh

Job NAMD multicore (single BDW node, simplified version)

Example script slurm-namd-bdw.sh to run NAMD multicore on 1 node using 16 cores and at most 32 GB of memory:

#!/bin/bash
#SBATCH --job-name=t1r1_min1
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=16
#SBATCH --mem=32G
#SBATCH --time=1-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
module load namd/2.12/bdw
 
namd2 +p ${SLURM_TASKS_PER_NODE} t1r1_min1.conf > t1r1_min1.log

Submission example:

sbatch slurm-namd-bdw.sh

Job NAMD multicore (single BDW node, exclusive mode)

Example script slurm-namd.sh to run NAMD multicore on 1 node in exclusive mode:

#!/bin/bash
#SBATCH --job-name=NAMD
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --mem=0
#SBATCH --exclusive
#SBATCH --time=1-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
module load namd/2.12/bdw
 
if [ ! -e "${SLURM_JOB_NAME}.conf" ]; then
    echo "Error: file '${SLURM_JOB_NAME}.conf' not found." 1>&2
    exit 1
fi
 
namd2 +p ${SLURM_TASKS_PER_NODE} "${SLURM_JOB_NAME}.conf" > "${SLURM_JOB_NAME}.log"

Submission example:

sbatch --job-name=t1r1_min1 slurm-namd-bdw.sh

Job NAMD multicore (single BDW node, exclusive mode, simplified version)

Example script slurm-namd-bdw.sh to run NAMD multicore on 1 node in exclusive mode:

#!/bin/bash
#SBATCH --job-name=t1r1_min1
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --mem=0
#SBATCH --exclusive
#SBATCH --time=1-00:00:00
#SBATCH --partition=bdw
#
# Edit next line
#SBATCH --account=<accountname>
 
module load namd/2.12/bdw
 
namd2 +p ${SLURM_TASKS_PER_NODE} t1r1_min1.conf > t1r1_min1.log

Submission example:

sbatch slurm-namd-bdw.sh

Job NAMD on KNL

Job NAMD multicore (single KNL node)

Example script slurm-namd-knl.sh to run NAMD multicore on 1 KNL node in exclusive mode using 64 cores:

#!/bin/bash
#SBATCH --job-name=NAMD
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=64
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=1-00:00:00
#SBATCH --partition=knl
#
# Edit next line
#SBATCH --account=<accountname>
 
module load namd/2.12/knl
 
if [ ! -e "${SLURM_JOB_NAME}.conf" ]; then
    echo "Error: file '${SLURM_JOB_NAME}.conf' not found." 1>&2
    exit 1
fi
 
namd2 +p ${SLURM_TASKS_PER_NODE} "${SLURM_JOB_NAME}.conf" > "${SLURM_JOB_NAME}.log"

Submission example:

sbatch --job-name=t1r1_min1 slurm-namd-knl.sh

Job NAMD on GPU

Job NAMD multicore (single GPU node)

Example script slurm-namd-gpu.sh to run NAMD multicore on 1 GPU node using 8 cores, 2 GPUs and at most 32 GB of memory:

#!/bin/bash
#SBATCH --job-name=NAMD
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:tesla:2
#SBATCH --mem=32G
#SBATCH --time=1-00:00:00
#SBATCH --partition=gpu
#
# Edit next line
#SBATCH --account=<accountname>
 
module load namd/2.12/gpu
 
if [ ! -e "${SLURM_JOB_NAME}.conf" ]; then
    echo "Error: file '${SLURM_JOB_NAME}.conf' not found." 1>&2
    exit 1
fi
 
namd2 +p ${SLURM_TASKS_PER_NODE} "${SLURM_JOB_NAME}.conf" > "${SLURM_JOB_NAME}.log"

Submission example:

sbatch --job-name=t1r1_min1 slurm-namd-gpu.sh

Submission example overriding the options specified in the launch script:

sbatch --job-name=t1r1_min1 --ntasks-per-node=16 --gres=gpu:tesla:4 --mem=64G slurm-namd-gpu.sh

Job FDS

FDS-SMV

Fire Dynamics Simulator (FDS) is a large-eddy simulation (LES) code for low-speed flows, with an emphasis on smoke and heat transport from fires.

Smokeview (SMV) is a visualization program used to display the output of FDS and CFAST simulations.

Job FSD multicore (single node)

Example script slurm-fds.sh to run FSD on 1 node using 28 cores and at most 32 GB of memory:

#!/bin/bash
#SBATCH --partition=bdw
#SBATCH --job-name=Modello_R04_SC1_R02
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --cpus-per-task=28
#SBATCH --mem=32G
#SBATCH --time=1-00:00:00 
#
# Edit next line
#SBATCH --account=<accountname>
 
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
 
module load intel/17.0.2.174
module load fds/6.7.0 
 
ulimit -s unlimited
ulimit -s
 
fds Modello_R04_SC1_R02.fds

Python

Two versions of python are available via modules:

  1. python/2.7.14
  2. python/3.6.3

Python virtualenv

Virtualenv is a tool to create isolated Python environments.

Python 2.7.14 virtualenv

Create a virtual environment for “machine-learning”:

module load gnu/5.4.0
module load python/2.7.14
module load cuda/8.0.61.2
module list
which python
python2 -m pip install --user --upgrade pip
python2 -m pip install --user virtualenv
mkdir -p "$HOME/local/tools/python/2.7.14/virtualenv/machine-learning"
python2 -m virtualenv "$HOME/local/tools/python/2.7.14/virtualenv/machine-learning"

Activate “machine-learning” virtual environment:

source "$HOME/local/tools/python/2.7.14/virtualenv/machine-learning/bin/activate"

Install packages in “machine-learning” virtual environment:

pip install --upgrade opencv-python Pillow sklearn h5py scikit-image cython easydict

Deactivate “machine-learning” virtual environment:

deactivate

Python 3.6.3 virtualenv

Available virtual environments for Python 3.6.3:

  • opencv
  • tensorflow
  • tensorflow-gpu
Site installation

Activate “opencv” virtual environment:

module load gnu/5.4.0
module load python/3.6.3

source '/hpc/share/tools/python/3.6.3/virtualenv/opencv/bin/activate'

Activate “tensorflow” virtual environment:

module load gnu/5.4.0
module load python/3.6.3

source '/hpc/share/tools/python/3.6.3/virtualenv/tensorflow/bin/activate'

Activate “tensorflow-gpu” virtual environment:

module load gnu/5.4.0
module load python/3.6.3
module load cuda/9.0.176
module load cudnn/7.1.1

source '/hpc/share/tools/python/3.6.3/virtualenv/tensorflow-gpu/bin/activate'

Deactivate the virtual environment:

deactivate
User installation

Create “opencv” virtual environment (just once):

module load gnu/5.4.0
module load python/3.6.3

hpc-make-virtualenv-opencv

Update “opencv” virtual environment (every time you want to update):

module load gnu/5.4.0
module load python/3.6.3

hpc-make-virtualenv-opencv

Activate “opencv” virtual environment:

module load gnu/5.4.0
module load python/3.6.3

source "$HOME/local/tools/python/3.6.3/virtualenv/opencv/bin/activate"

Create “tensorflow” virtual environment (just once):

module load gnu/5.4.0
module load python/3.6.3

hpc-make-virtualenv-tensorflow

Update “tensorflow” virtual environment (every time you want to update):

module load gnu/5.4.0
module load python/3.6.3

hpc-make-virtualenv-tensorflow

Activate “tensorflow” virtual environment:

module load gnu/5.4.0
module load python/3.6.3

source "$HOME/local/tools/python/3.6.3/virtualenv/tensorflow/bin/activate"

Create “tensorflow-gpu” virtual environment (just once):

module load gnu/5.4.0
module load python/3.6.3

hpc-make-virtualenv-tensorflow-gpu

Update “tensorflow-gpu” virtual environment (every time you want to update):

module load gnu/5.4.0
module load python/3.6.3

hpc-make-virtualenv-tensorflow-gpu

Activate “tensorflow-gpu” virtual environment:

module load gnu/5.4.0
module load python/3.6.3
module load cuda/9.0.176
module load cudnn/7.1.1

source '/hpc/share/tools/python/3.6.3/virtualenv/tensorflow-gpu/bin/activate'

Deactivate the virtual environment:

deactivate

Other stuff

VTune

VTune is a performance profiler from Intel and is available on the HPC cluster.

General information from Intel: https://software.intel.com/en-us/get-started-with-vtune-linux-os

VTune local guide (work in progress)

CINECA guides

Other resources

calcoloscientifico/userguide.txt · Ultima modifica: 25/10/2019 14:42 da roberto.alfieri