SLURM DRAFT

Job submission FAQ:

Waiting for training slides, please use this document (Slurm for users) : Slurm_for_users

[BATCH]

sbatch: submit a batch job to slurm (default workq partition).
sarray: submit a batch job-array to slurm.

[INTERACTIVE]

srun --pty bash : submit an interactive session with a compute node (default workq partition).

runVisuSession.sh: submit a TurboVNC / VirtualGL session with the graphical node (interq partition). Just for graphics jobs.

srun
-J job name  -> for change the jobname
-p partition  -> which partition(~ queue) to use
--time=HH:MM:SS -> max time of the job

-o (--output) = output_filename : to specify the stdout redirection. If -e (--error) is not specified both stdout ans stderr will be directed to the file name specified.
-e (--error) = error_filename : if specified, stderr will be redirected to different location as stdout

Without any parameter, on any partition, each job is limited to 1 cpu, 4G ram (cpus-per-task=1, mem=4G)

1 - First write a script (ex: myscript.sh) with the command line as following:

#!/bin/bash
#SBATCH -J test
#SBATCH -o output.out
#SBATCH -e error.out
#SBATCH -t 01:00:00
#SBATCH --mem=8G
#SBATCH --mail-type=BEGIN,END,FAIL
#Purge any previous modules
module purge

#Load the application
module load bioinfo/ncbi-blast-2.2.29+

# My command lines I want to run on the cluster
blastall ...

2 - To submit the job, use the sbatch command line as following:

sbatch myscript.sh

To change memory reservation, add this option to the submission command (sbatch, srun, sarray):

--mem=XG (default value is 4G)

With default parameters, each job is limited to 1 cpu.
To book more, use the following options:

# Book n cpus on the same node (up to 64)
-c=ncpus (--cpus-per-task=ncpus)

# Book n cpus on any nodes in case of MPI jobs
-N=nnodes (--nodes=nnodes)

-n=ntasks (--ntasks=ntasks)

-c=ncpus (--cpus-per-task)ncpus

Each job is submitted to a specific partition (the default one is the workq).
Each partition has a different priority considering the maximum time of execution allowed.

 

QueueAccessPriorityMax timeMax slots
workqeveryone1004 days (96h)3072
unlimitqeveryone1180 days500
interq (runVisuSession.sh)on demand1 day (24h)32
smpqon demand180 days96
wflowqspecific software180 days3072

To submit an array of jobs, use sarray command (same sbatch options):

sarray [sbatch options] shell_command_file

Not available for the moment

Use the following command line (on genologin server):

mmlsquota -u username

Example of a full bash script :
#!/bin/bash
#SBATCH -J mpi_job
#SBATCH --nodes=2
#SBATCH --tasks-per-node=6
#SBATCH --time=00:10:00
cd $SLURM_SUBMIT_DIR
module purge
module load compiler/intel-2018.0.128 mpi/openmpi-1.8.8-intel2018.0.128
mpirun -n $SLURM_NTASKS -npernode $SLURM_NTASKS_PER_NODE ./hello_world

To do so, you can use the squeue command, following are some usefull options:

squeue -u username : list only the specified user's jobs.
squeue -j job_id : provide several informations on the specified job.

For more detail:

scontrol show job job_id

You can also have access to a graphical user interface which provides the same informations.
This interface is accessible with the sview command.

To do so, use the sacct command line as following:

sacct -j job_id

To do so, you can use the scancel command, following are some usefull options:

# Kill the specified job
scancel job_id


# Kill all job launched by the specified user
scancel -u username

SGESLURMComments
qsub -b y "command"srun "command"submit a job in real time
qsub -l mem=XG -l h_vmem=YG -b y srun --mem=YGNo h_vmem parameters with Slurm.
qsub script.shsbatch script.shsbatch is only for script
qsub -m beasbatch/srun --mail-type=BEGIN,END,FAILNotify user by email when certain event types occur.
qsub -pe parallel_smp 8sbatch/srun -c=8 (--cpus-per-task=8)by default, job are on one node (-N=1 <--> --nodes=nnodes)
qsub -pe parallel_fill n or
qsub -pe parallel_rr n
sbatch/srun

-N=nnodes (--nodes=nnodes)

-n=ntasks (--ntasks=ntasks)
-c=ncpus (--cpus-per-task)ncpus
No parallel environnement with slurm
qstat -u loginsqueue -u loginSee all your submitted jobs
qstat -j job_idscontrol show job job_idRunning job details
qacct -j job_idsacct --unit=G --format JobID,jobname,NTasks,nodelist,CPUTime,ReqMem,MaxVMSize,Elapsed job_idFinished job details.

 

Default ressources FAQ

47 INTEL cluster nodes : node101... node147 (each 64 cores with hyperthreading, 4GB memory/core, 250G/node for SLURM)

1 SMP node : genosmp02 (96 cores with hyperthreading & 1,5TB of memory)

1 Visu node : genoview (64cores with hyperthreading, 128G of memory, Graphic Card: Nvidia K40)

/home/user: 100MB available to store your configuration files.
/work/user: 1TB available as working directory. You have read/write access from any cluster node. Files are automaticly deleted if they have not been accessed within the last 120 days (to know them: find repertoire/ -atime +120).
/save/user: 250GB available for data you want to save with 30 retention days. You have read only access on this directory from any cluster nodes.


If you need more space in /work or in /save you are invited to fill the resources request form.


/usr/local/bioinfo/src/: directory gathering all bioinformatic software (see Software FAQ)
/bank: biological banks in different format (see Databanks FAQ)

Academic account quota: 100 000 h/per calendar year
Beyond these 100,000 hours, you will need to submit a science project (by the resources request form) to estimate the real needs of the bioinformatics environment.

According to results from this evaluation, but also their geographical and institutional origin, users can then either continue their treatments or be invited to contribute financially to infrastructure, or be redirected to regional or national mésocentres calculation.

Non-academic account quota:  500 h/per calendar year for testing the infrastructure.
Overtime calculation will be charged (price on request).


To know your quota, use the command:
Soon available command login

Without any parameters, on any queue, all jobs are limited to:

  • 4GB (memory)
  • 1 CPU (thread)

It depends on our genotoul linux group (contributors, INRA and/or REGION, others).

Max slotsworkq
(group)
workq
(user)
unlimitq
(all users sum)
unlimitq
(user)
Contributors3072768500125
INRA/Region230457650094
Others76819250031

It depends on our genotoul linux group (contributors, INRA and/or REGION, others).

 

Soon available

Max jobs per user2500
Max job for all users10000

 

Slurm Cluster software FAQ:

  • On our Web site (Resources): Software
  • On genologinserver with ls command:

ls /usr/local/bioinfo/src|grep -i soft_name

  • On genologinserver with module command:

module avail : display all available software installed on the cluster

module avail category/soft_name : display the versions available for a particular application (with category in bioinfo,compiler,mpi or system) (case sensitive)

module avail -t 2>&1 | grep -i soft_name :display the versions available for a particular application  (case insensitive)

Examples :

$module avail bioinfo/cutadapt

$-------------/tools/share/Modules ----------------------------------------
bioinfo/cutadapt-1.14-python-2.7.2 bioinfo/cutadapt-1.14-python-3.4.3

 

$ module avail -t 2>&1 | grep -i openmpi
mpi/openmpi-1.6.5
mpi/openmpi-1.8.1
mpi/openmpi-1.8.8
mpi/openmpi-1.8.8-intel2018.0.128
mpi/openmpi-2.1.2
mpi/openmpi-2.1.2-intel2018.0.128

  • Bioinfo software: /usr/local/bioinfo/src/
  • Others tools and libraries: /tools/
  • Compilers: /tools/compilers

Software installation or update are performed by complete the following form: Software installation

  • Run a specific application

We  now use Environment Modules to dynamically set up environments for different applications (see "How to use module command ?" section of this FAQ for more information).

To run a software you need to load  the corresponding module.

Examples:

To use blast:

module load bioinfo/blast-2.2.26

To verify:

$ which blastall
/usr/local/bioinfo/src/NCBI_Blast/blast-2.2.26/bin/blastall

To use Python 2.7.2

$ module load system/Python-2.7.2
$ which python
/tools/python/2.7.2/bin/python

  • Documentation

Typically , documentation is provided by the soft in the installation folder (/usr/local/bioinfo/src/soft_name) and on the web site of the software (we give the link to the website of each software on the Software page).

You can find also specific information on a How_to_use_SLURM_ file in the software installation directory (/usr/local/bioinfo/src/soft_name) or on our web site Software page (Avaibility/Use column, click on SLURM cluster link) .

In the software installation directory, you can also find a basic example submission job in an example_on_cluster directory. 

Example with bowtie:

$ ls /usr/local/bioinfo/src/bowtie/
bowtie-1.2.1.1 bowtie2-2.2.9 bowtie2-2.3.3.1-linux-x86_64.zip How_to_use_SLURM_bowtie
bowtie-1.2.1.1-linux-x86_64.zip bowtie2-2.3.3.1 example_on_cluster

$ ls /usr/local/bioinfo/src/bowtie/example_on_cluster/
errot.txt lambda_virus.1.bt2 lambda_virus.3.bt2 lambda_virus.rev.1.bt2 output.txt
example lambda_virus.2.bt2 lambda_virus.4.bt2 lambda_virus.rev.2.bt2 test_v2-2.2.9.sh

The basic command to use is module:

module : (no arguments) print usage instructions
module avail
: list available software module
module load modulename : add a module to your environment
module unload modulename : unload remove a module
module purge : remove all modules
module show modulename : show what changes a module will make to your environment

For more documentation, you can see the Environment Module website.

Launch java onto the nodes for the java command:
java -Xmx4g

Choose a specific java version:
module load system/Java8 (for java 8)

To see available compiler:

module avail compiler

Exemple to load a compiler:

module load compiler/gcc-7.2.0