===== Conda =====

Conda is an open source, cross-platform, language-agnostic package manager and environment management system. It is released under the Berkeley Software Distribution License by Continuum Analytics.

Conda allows users to easily install different versions of binary software packages and any required libraries appropriate for their computing platform. Also, it allows users to switch between package versions and download and install updates from a software repository.

Conda is written in the Python programming language, but can manage projects containing code written in other languages (e.g., R), including multi-language projects. Conda can install the Python programming language, while similar Python-based cross-platform package managers (such as wheel or pip) cannot.


A set of Conda [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#building-identical-conda-environments |environments]] has been defined on the HPC system.

[[https://www.hpc.unipr.it/dokuwiki/doku.php?id=calcoloscientifico:priv:cluster:conda | Conda configuration on the HPC cluster]] (private area)

==== Virtualenv list ====

To see a list of all available environments run:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda info --envs
</code>

==== Virtualenv "base" ====

Enable the "base" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate base
</code>

Disable the "base" environment:

<code>
conda deactivate
</code>

==== Virtualenv "statistical-genetics" ====

Available versions of the "statistical-genetics" environment:

^ condaenv                         ^ **build date**  ^
| statistical-genetics             | 2020.06.29      |
| statistical-genetics-2020.03.14  | 2020.03.14      |
| statistical-genetics-2020.06.29  | 2020.06.29      |

Enable the default version of the "statistical-genetics" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate statistical-genetics
</code>

Enable the 2020.03.14 version of the "statistical-genetics" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate statistical-genetics-2020.03.14
</code>

Packages in the "statistical-genetics" environment (any version):

<code>
conda list
</code>

Disable the "statistical-genetics" environment (any version):

<code>
conda deactivate
</code>

Now we will see how to run the scripts that you find on the [[calcoloscientifico:userguide:r|R]] page.

Script ''slurm-statistical-genetics-vrt.sh'':

<code bash slurm-statistical-genetics-vrt.sh>
#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem=512M
#SBATCH --partition=vrt
#SBATCH --time=0-00:05:00
##SBATCH --account=<account>

module load miniconda3

source "$CONDA_PREFIX/etc/profile.d/conda.sh"

conda activate statistical-genetics

Rscript --no-save hello.R

conda deactivate
</code>

Script ''slurm-statistical-genetics-bdw.sh'':

<code bash slurm-statistical-genetics-bdw.sh>
#!/bin/bash
#SBATCH --job-name=loop
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --mem=512M
#SBATCH --partition=bdw
#SBATCH --time=0-00:05:00
##SBATCH --account=<account>

module load miniconda3

source "$CONDA_PREFIX/etc/profile.d/conda.sh"

conda activate statistical-genetics

Rscript --no-save loop.R ${SLURM_NTASKS_PER_NODE:-1}

conda deactivate
</code>

==== Virtualenv "machine-learning-cuda" ====

Main packages included in 2022 condaenvs:

<code>
conda install:
 - cudatoolkit
 - pip
 - mpi4py
 - h5py
 - keras
 - opencv
 - opencv-python-headless
 - tensorflow
 - torchvision
 - easydict
 - pytorch 
 - statsmodels
 - flax
 - transformers

pip install:
 - keras.datasets
 - sklearn
</code>

Available versions of the "machine-learning-cuda" environment:

^ condaenv                                 ^ build date  ^
| machine-learning-cuda-11.6.0             | 2022.05.20  |
| machine-learning-cuda-11.6.0-2022.05.20  | 2022.05.20  |
| machine-learning-cuda-11.6.0-2022.03.09  | 2022.03.09  |
| machine-learning-cuda-11.4.2             | 2022.05.20  |
| machine-learning-cuda-11.4.2-2022.05.20  | 2022.05.20  |
| machine-learning-cuda-11.4.2-2022.03.09  | 2022.03.09  |

Main packages included in 2021 condaenvs:

<code>
conda install:
 - cudatoolkit
 - pip
 - mpi4py
 - h5py
 - keras
 - opencv
 - opencv-python-headless
 - tensorflow
 - torchvision
 - easydict
 - pytorch 

pip install:
 - keras.datasets
 - sklearn
</code>

Available versions of the "machine-learning-cuda" environment:

^ condaenv                                 ^ build date  ^
| machine-learning-cuda-11.2.2             | 2021.10.23  |
| machine-learning-cuda-11.2.2-2021.10.23  | 2021.10.23  |
| machine-learning-cuda-11.2.1             | 2021.10.23  |
| machine-learning-cuda-11.2.1-2021.10.23  | 2021.10.23  |
| machine-learning-cuda-11.2.0             | 2021.10.23  |
| machine-learning-cuda-11.2.0-2021.10.23  | 2021.10.23  |
| machine-learning-cuda-11.1               | 2021.10.22  |
| machine-learning-cuda-11.1-2021.10.22    | 2021.10.22  |
| machine-learning-cuda-11.0               | 2021.10.23  |
| machine-learning-cuda-11.0-2021.10.23    | 2021.10.23  |
| machine-learning-cuda-10.2               | 2021.10.22  |
| machine-learning-cuda-10.2-2021.10.22    | 2021.10.22  |
| machine-learning-cuda-10.1               | 2021.10.23  |
| machine-learning-cuda-10.1-2021.10.23    | 2021.10.23  |
| machine-learning-cuda-10.0               | 2021.10.23  |
| machine-learning-cuda-10.0-2021.10.23    | 2021.10.23  |
| machine-learning-cuda-9.0                | 2019.10.22  |
| machine-learning-cuda-9.0-2019.10.22     | 2019.10.22  |
| machine-learning-cuda-8.0                | 2019.10.22  |
| machine-learning-cuda-8.0-2019.10.22     | 2019.10.22  |

Creation scripts:

<code>
/hpc/share/tools/conda/bin/hpc-make-condaenv-machine-learning-cuda-10.0
/hpc/share/tools/conda/bin/hpc-make-condaenv-machine-learning-cuda-10.1
/hpc/share/tools/conda/bin/hpc-make-condaenv-machine-learning-cuda-10.2
/hpc/share/tools/conda/bin/hpc-make-condaenv-machine-learning-cuda-11.0
/hpc/share/tools/conda/bin/hpc-make-condaenv-machine-learning-cuda-11.1
/hpc/share/tools/conda/bin/hpc-make-condaenv-machine-learning-cuda-11.2.0
/hpc/share/tools/conda/bin/hpc-make-condaenv-machine-learning-cuda-11.2.1
/hpc/share/tools/conda/bin/hpc-make-condaenv-machine-learning-cuda-11.2.2
</code>

Enable the default version of the "machine-learning-cuda-10.1" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate machine-learning-cuda-10.1
</code>

Packages in the "machine-learning-cuda-XX" environment (any version):

<code>
conda list
</code>

Disable the "machine-learning-cuda" environment (any version):

<code>
conda deactivate
</code>

Script ''slurm-machine-learning-cuda.sh'':

<code bash slurm-machine-learning-cuda.sh>
#!/bin/bash
#SBATCH --job-name=conda
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:tesla:1
#SBATCH --partition=gpu
#SBATCH --mem=16G
#SBATCH --time=0-00:30:00
##SBATCH --account=<account>

module load miniconda3

source "$CONDA_PREFIX/etc/profile.d/conda.sh"

conda activate machine-learning-cuda-10.1

#...

conda deactivate
</code>

==== Virtualenv "cryolo-cpu" ====

Available versions of the "cryolo-cpu" environment:

| **condaenv**           ^ build date  ^
| cryolo-cpu             | 2020.09.18  |
| cryolo-cpu-2020.09.18  | 2020.09.18  |

Enable the default version of the "cryolo-cpu" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate cryolo-cpu
</code>

Enable the 2020.09.18 version of the "cryolo-cpu" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate cryolo-cpu-2020.09.18
</code>

Packages in the "cryolo-cpu" environment (any version):

<code>
conda list
</code>

Disable the "cryolo-cpu" environment (any version):

<code>
conda deactivate
</code>

Script ''slurm-cryolo-vrt.sh'':

<code bash slurm-cryolo-vrt.sh>
#!/bin/bash
#SBATCH --job-name=cryolo
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem=1G
#SBATCH --partition=vrt
#SBATCH --time=0-01:00:00
##SBATCH --account=<account>

module load miniconda3

source "$CONDA_PREFIX/etc/profile.d/conda.sh"

conda activate cryolo-cpu

cryolo_gui.py config config_cryolo.json 220 \
--filter LOWPASS \
--low_pass_cutoff 0.1 \
--num_cpu ${SLURM_NTASKS_PER_NODE:-1}

conda deactivate
</code>

Script ''slurm-cryolo-bdw.sh'':

<code bash slurm-cryolo-bdw.sh>
#!/bin/bash
#SBATCH --job-name=cryolo
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --mem=4G
#SBATCH --partition=bdw
#SBATCH --time=0-01:00:00
##SBATCH --account=<account>

module load miniconda3

source "$CONDA_PREFIX/etc/profile.d/conda.sh"

conda activate cryolo-cpu

cryolo_gui.py config config_cryolo.json 220 \
--filter LOWPASS \
--low_pass_cutoff 0.1 \
--num_cpu ${SLURM_NTASKS_PER_NODE:-1}

conda deactivate
</code>

==== Virtualenv "cryolo-gpu" ====

Available versions of the "cryolo-gpu" environment:

^ condaenv               ^ build date  ^
| cryolo-gpu             | 2020.09.18  |
| cryolo-gpu-2020.09.18  | 2020.09.18  |

Enable the default version of the "cryolo-gpu" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate cryolo-gpu
</code>

Enable the 2020.09.18 version of the "cryolo-gpu" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate cryolo-gpu-2020.09.18
</code>

Packages in the "cryolo-gpu" environment (any version):

<code>
conda list
</code>

Disable the "cryolo-gpu" environment (any version):

<code>
conda deactivate
</code>

Script ''slurm-cryolo-gpu.sh'':

<code bash slurm-cryolo-gpu.sh>
#!/bin/bash
#SBATCH --job-name=cryolo
#SBATCH --output=%x.o%j
#SBATCH --error=%x.e%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:1
#SBATCH --mem=1G
#SBATCH --partition=gpu
#SBATCH --time=0-01:00:00
##SBATCH --account=<account>

module load miniconda3

source "$CONDA_PREFIX/etc/profile.d/conda.sh"

conda activate cryolo-gpu

cryolo_gui.py config config_cryolo.json 220 \
--filter LOWPASS \
--low_pass_cutoff 0.1 \
--num_cpu ${SLURM_NTASKS_PER_NODE:-1}

conda deactivate
</code>

==== Virtualenv "mdanalysis" ====

Available versions of the "mdanalysis" environment:

^ condaenv          ^ **build date**  ^
| mdanalysis        | 2023.03.20      |
| mdanalysis-2.3.0  | 2023.03.20      |

Enable the default version of the "mdanalysis" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate mdanalysis
</code>

Enable the 2.3.0 version of the "mdanalysis" environment:

<code>
module load miniconda3
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate mdanalysis-2.3.0
</code>

Packages in the "mdanalysis" environment (any version):

<code>
conda list
</code>

Disable the "mdanalysis" environment (any version):

<code>
conda deactivate
</code>