Getting started with the I2BC cluster
Objective 3
Learning how to use the software already available on the cluster.
Software should only be run on the salve nodes, never on the Frontale (everyone connects to the cluster through the Frontale, thus, running software on the Frontale directly will slow down the connection for all its users).
Once on a node, you can run your programmes and scripts, but where to find them?
Introduction to the module system
Many clusters manage all software through the module command, the I2BC cluster works the same way. There are 3 important commands to remember:
module availto list all software that is accessible on the clustermodule loadto “activate” a software in order to use itmodule unloadto remove a loaded software (for example, in order to load another version of that same software)
Finding the software
Use module avail to list all software that is accessible on the cluster:
john.doe@cluster-i2bc:/home/john.doe$ module avail
------------------------------------------- /usr/share/modules/modulefiles --------------------------------------------
454/454 nodes/python3-openssl-20.0.1 singularity/charmm42
AFG/afg nodes/python3-pyparsing-2.4.7 singularity/clever2
albula/albula3.2 nodes/python3-pysam-0.15.4 singularity/clevertk
amber16/amber16 nodes/python3-rpy2-3.4.2 singularity/CorsetLace
atsas/atsas-2.7.1-1 nodes/python3-scipy-1.6.0 singularity/cutadap
bcf/bcf1.3.1 nodes/python3-tk-3.9.2 singularity/fastqc
bcl2fast2/bcl2fastq2-2.15.0.4 nodes/python3-tz-2021.1 singularity/gatk
bcl2fast2/bcl2fastq2-2.18.12 nodes/python3-virtualenv-20.4.0 singularity/hicpro3.1.0
bedtools/bedtools2.26.0 nodes/python3-xlrd-1.2.0 singularity/hicstuff-3.2.1
bowtie/bowtie2-2.2.1 nodes/r-base-4.0.4 singularity/hisat2-2.2.1
bowtie/bowtie2-2.2.9 nodes/r-cran-ade4-1.7 singularity/R3.5
[...]
To search for a specific software, you can use the -C option followed by a search pattern (add -i to make it case insensitive). For example:
john.doe@cluster-i2bc:/home/john.doe$ module avail -C fast -i
------------------------------------------- /usr/share/modules/modulefiles --------------------------------------------
bcl2fast2/bcl2fastq2-2.15.0.4 fastp/fastp-0.23.2 fastqc/fastqc_v0.10.1 nodes/fastdnaml-1.2.2
bcl2fast2/bcl2fastq2-2.18.12 fastp/fastp-0.23.4 fastqc/fastqc_v0.11.5 singularity/fastqc
Loading the software
Loading the software will only work if you are on one of the slave nodes of the cluster.
john.doe@cluster-i2bc:/home/john.doe$ qsub -I
qsub: waiting for job 916684.pbsserver to start
qsub: job 916684.pbsserver ready
john.doe@node09:/home/john.doe$
Once this is done, you’ll be able to run module load. For example, to load version 0.11.5 of FastQC, copy the exact terms from the module avail command:
john.doe@node09:/home/john.doe$ module load fastqc/fastqc_v0.11.5
john.doe@node09:/home/john.doe$ fastqc --version
FastQC v0.11.5
To undo the load, just run module unload in the same way:
john.doe@node09:/home/john.doe$ module unload fastqc/fastqc_v0.11.5
john.doe@node09:/home/john.doe$ fastqc --version
-bash: fastqc: command not found
As you can see, the shell doesn’t recognise the fastqc command anymore as expected.
Adding software that isn't listed
If you have trouble finding your software, please send an email to the SICS with [cluster install] in the subject and specify what software you need installing.
Take home message
1) software on the cluster is managed by the module system
2) there are 3 key commands: module avail, module load and module unload
3) software should only be run on the slave nodes, never on the Frontale
4) missing software can be easily and rapidly installed by the SICS