Strumenti Utente

Strumenti Sito

Traccia: • CONDA user defined environments
calcoloscientifico:cluster:softwareapplicativo:conda_user

Indice

CONDA user defined environments

Users or groups can build their own environments.

General template

General template that can be used to create personal or group environments:

hpc-make-condaenv-template.sh
#!/bin/bash
 
set -e
 
module load miniconda3
 
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
 
CONDAENV_NAME='template'
CONDAENV_VERSION=$(date '+%Y.%m.%d')
PYTHON_VERSION='3.9'
 
CONDAENV="${CONDAENV_NAME}-${CONDAENV_VERSION}"
 
if [ $UID = 0 ]; then
    CONDAENV_PREFIX="/hpc/share/tools/miniconda3/envs/$CONDAENV"
    CONDAENV_SYMLNK="/hpc/share/tools/miniconda3/envs/$CONDAENV_NAME"
else
    if [ -z "$CONDA_ENVS_PATH" ]; then
        echo 'Error: the CONDA_ENVS_PATH envarionment variable is not set' 1>&2
        exit 1
    fi
    if [ -z "$CONDA_PKGS_DIRS" ]; then
        echo 'Error: the CONDA_PKGS_DIRS envarionment variable is not set' 1>&2
        exit 1
    fi
    echo "CONDA_ENVS_PATH: $CONDA_ENVS_PATH"
    echo "CONDA_PKGS_DIRS: $CONDA_PKGS_DIRS"
    CONDAENV_PREFIX="$CONDA_ENVS_PATH/$CONDAENV"
    CONDAENV_SYMLNK="$CONDA_ENVS_PATH/$CONDAENV_NAME"
fi
 
if [ $UID = 0 ]; then
    conda --version
    conda update conda
    conda --version
    echo
fi
 
echo "CONDAENV_PREFIX: $CONDAENV_PREFIX"
echo "CONDAENV_SYMLNK: $CONDAENV_SYMLNK"
echo
 
if [ -d "$CONDAENV_PREFIX" ]; then
    echo -n "Should we remove the already existing condaenv '$CONDAENV_NAME' (yes/no)? "
 
    read ans
 
    case "$ans" in
        [yY]|[yY][Ee][Ss])
        conda env remove --name "$CONDAENV"
        ;;
        [nN]|[nN][oO])
        ;;
        *)
        echo
        exec "$0"
        exit
        ;;
    esac
fi
 
if [ -d "$CONDAENV_PREFIX" ]; then
    echo -n "Should we proceed with the modification of the condaenv '$CONDAENV_NAME' (yes/no)? "
else
    echo -n "Should we proceed with the creation of the condaenv '$CONDAENV_NAME' (yes/no)? "
fi
 
read ans
 
case "$ans" in
    [yY]|[yY][Ee][Ss])
    ;;
    [nN]|[nN][oO])
    exit 1
    ;;
    *)
    echo
    exec "$0"
    exit
    ;;
esac
 
conda info
 
if [ ! -d "$CONDAENV_PREFIX" ]; then
    conda create --yes --prefix "$CONDAENV_PREFIX" python${PYTHON_VERSION:+=$PYTHON_VERSION}
fi
 
conda activate "$CONDAENV"
 
conda install --yes --name "$CONDAENV" \
    --channel conda-forge \
    xz
 
conda list || true
 
conda deactivate
 
ln -sf "$CONDAENV_PREFIX" "$CONDAENV_SYMLNK"
ls -la "$CONDAENV_SYMLNK"*

This script is located in 

/hpc/share/tools/conda/bin/hpc-make-condaenv-template

Customize your script by editing CONDAENV_NAME, PYTHON_VERSION and the "conda install" command.

The "conda install" command can be repeated multiple times with different options and parameters.

It is also possible to install python packages using the "pip" command.

Example with conda and pip

General template that can be used to create personal or group environments:

hpc-make-condaenv-networkx
#!/bin/bash
 
set -e
 
PYTHON_VERSION='3.12'
CONDAENV_NAME='networkx'
CONDAENV_RELEASE='py3.12'
CONDAENV_DEFAULT='yes'
NETWORKX_VERSION='3.6.1'
 
help() {
    cat <<EOF
Synopsis:
  $(basename $0) [--dry-run]
  $(basename $0) [--pip-dry-run]
  $(basename $0) --help
 
Usage:
  $(basename $0)
  $(basename $0) --dry-run
  $(basename $0) --pip-dry-run
 
Example:
  $(basename $0)
  $(basename $0) --dry-run
  $(basename $0) --pip-dry-run
 
Options:
  -d, --dry-run                 Only display what would have been done by conda
      --pip-dry-run             Only display what would have been done by pip
  -h, --help                    Display a help message and quit
EOF
}
 
die() {
    if [ $# -ge 1 ]; then
        echo "$(basename "$0"): $@" >&2
    fi
    exit 1
}
 
# Command line options
opt=$(getopt \
    --options hd \
    --longoptions help,pip-dry-run,dry-run \
    --name $(basename $0) \
    -- \
    "$@")
 
retval=$?
 
if [ $retval -ne 0 ]; then
    exit $retval
fi
 
eval set -- "$opt"
 
while true; do
    case "$1" in
        -h|--help)
            shift
            help
            exit;;
 
        -d|--dry-run)
            shift
            opt_dry_run=true;;
 
        --pip-dry-run)
            shift
            opt_pip_dry_run=true;;
 
        --)
            shift
            break;;
 
        *)
            die 'options parsing error'
            exit 1;;
    esac
done
 
# CONDA dry run
if [ -n "$opt_dry_run" ]; then
    unset PYTHON_VERSION
fi
 
# PIP dry run
if [ -n "$opt_pip_dry_run" ]; then
    unset NETWORKX_VERSION
fi
 
# CONDA environment name
if [ -n "$NETWORKX_VERSION" ]; then
    CONDAENV_VERSION="$NETWORKX_VERSION"
else
    if [ -n "$PYTHON_VERSION" ]; then
        PY=${PYTHON_VERSION:+$(echo $PYTHON_VERSION | sed -re 's/([0-9]+)\.([0-9]+).*/py\1.\2/')}
    fi
    CONDAENV_VERSION=${PY:+${PY}-}$(date '+%Y.%m.%d')
fi
if [ -n "$CONDAENV_RELEASE" ]; then
    CONDAENV_VERSION="${CONDAENV_VERSION}-${CONDAENV_RELEASE}"
fi
CONDAENV="${CONDAENV_NAME}-${CONDAENV_VERSION}"
 
module load miniconda3
 
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
 
if [ $UID = 0 ]; then
    CONDAENV_PREFIX="/hpc/share/tools/miniconda3/envs/$CONDAENV"
    CONDAENV_SYMLNK="/hpc/share/tools/miniconda3/envs/$CONDAENV_NAME"
else
    if [ -z "$CONDA_ENVS_PATH" ]; then
        echo 'Error: the CONDA_ENVS_PATH envarionment variable is not set' 1>&2
        exit 1
    fi
    if [ -z "$CONDA_PKGS_DIRS" ]; then
        echo 'Error: the CONDA_PKGS_DIRS envarionment variable is not set' 1>&2
        exit 1
    fi
    echo "CONDA_ENVS_PATH: $CONDA_ENVS_PATH"
    echo "CONDA_PKGS_DIRS: $CONDA_PKGS_DIRS"
    CONDAENV_PREFIX="$CONDA_ENVS_PATH/$CONDAENV"
    CONDAENV_SYMLNK="$CONDA_ENVS_PATH/$CONDAENV_NAME"
fi
 
echo "CONDAENV_PREFIX: $CONDAENV_PREFIX"
if echo "$CONDAENV_DEFAULT" | egrep -q '^[yY][Ee][Ss]$'; then
    echo "CONDAENV_SYMLNK: $CONDAENV_SYMLNK"
fi
echo
 
if [ -z "$opt_dry_run" ]; then
    if [ -d "$CONDAENV_PREFIX" ]; then
        echo -n "Should we remove the already existing condaenv '$CONDAENV_NAME' (yes/no)? "
 
        read ans
 
        case "$ans" in
            [yY]|[yY][Ee][Ss])
                conda env remove --name "$CONDAENV"
                ;;
            [nN]|[nN][oO])
                ;;
            *)
                echo
                exec "$0"
                exit
                ;;
        esac
    fi
 
    if [ -d "$CONDAENV_PREFIX" ]; then
        echo -n "Should we proceed with the modification of the condaenv '$CONDAENV_NAME' (yes/no)? "
    else
        echo -n "Should we proceed with the creation of the condaenv '$CONDAENV_NAME' (yes/no)? "
    fi
 
    read ans
 
    case "$ans" in
        [yY]|[yY][Ee][Ss])
            ;;
        [nN]|[nN][oO])
            exit 1
            ;;
        *)
            echo
            exec "$0"
            exit
            ;;
    esac
fi
 
conda info
 
if [ ! -d "$CONDAENV_PREFIX" ]; then
    # CONDA create
    conda create \
        --yes \
        ${opt_dry_run:+--dry-run} \
        --prefix "$CONDAENV_PREFIX" \
        --override-channels \
        --channel conda-forge \
        python${PYTHON_VERSION:+=$PYTHON_VERSION} \
        pip \
        matplotlib
fi
 
if [ -n "$opt_dry_run" ]; then
    conda deactivate
    exit
fi
 
CONDADEF_PIP="$(which pip)"
 
conda activate "$CONDAENV"
 
CONDAENV_PIP="$(which pip)"
 
# PIP check
if [ "$CONDADEF_PIP" = "$CONDAENV_PIP" ]; then
    echo "Error: condaenv pip is conda default pip: '$CONDAENV_PIP'" 1>&2
    exit 1
fi
 
# PIP install or upgrade pip
pip install --upgrade ${opt_pip_dry_run:+--dry-run} pip
 
# PIP install or upgrade
pip install --upgrade ${opt_pip_dry_run:+--dry-run} \
    networkx${NETWORKX_VERSION:+==$NETWORKX_VERSION}
 
conda list || true
 
conda deactivate
 
# PIP dry run
if [ -n "$opt_pip_dry_run" ]; then
    exit
fi
 
# CONDA environment symbolic link
if echo "$CONDAENV_DEFAULT" | egrep -q '^[yY][Ee][Ss]$'; then
    rm -f "$CONDAENV_SYMLNK"
    ln -s "$CONDAENV_PREFIX" "$CONDAENV_SYMLNK"
    symlinks -cs $(dirname "$CONDAENV_PREFIX")
    ls -lard "$CONDAENV_SYMLNK"*
fi

This script is located in 

/hpc/share/tools/conda/bin/hpc-make-condaenv-networkx

Creating a user environment

Create a script from the template and run it.

Creating a group environment

Change group: 

newgrp <GROUP>

Create a script from the template and run it.

Working with conda

With conda, you can create, export, list, remove, and update that have different versions of Python and/or packages installed in them. Switching or moving between environments is called activating the environment.

Interactive work session

If you want to work at group level change group otherwise skip this step: 

newgrp <GROUP>

Typical work session with conda (CPU): 

[<USER>@ui01 ~]$ srun --nodes=1 --ntasks-per-node=2 --partition=cpu --qos=cpu --mem=8G --pty bash

[<USER>@wn80 ~]$ module load miniconda3

[<USER>@wn80 ~]$ source "$CONDA_PREFIX/etc/profile.d/conda.sh"

[<USER>@wn80 ~]$ conda --help

[<USER>@wn80 ~]$ conda --version
conda 25.7.0

[<USER>@wn80 ~]$ conda info

[<USER>@wn80 ~]$ conda activate

(base) [<USER>@wn80 ~]$ which python
/hpc/share/tools/miniconda3/bin/python

(base) [<USER>@wn80 ~]$ python -V
Python 3.9.18

(base) [<USER>@wn80 ~]$ conda create --name my_conda_env [...]
(base) [<USER>@wn80 ~]$ conda activate my_conda_env

(my_conda_env) [<USER>@wn80 ~]$ [...]
(my_conda_env) [<USER>@wn80 ~]$ conda deactivate

(base) [<USER>@wn80 ~]$ conda deactivate

[<USER>@wn80 ~]$ exit

[<USER>@ui01 ~]$

Typical work session with conda (GPU): 

[<USER>@ui01 ~]$ srun --nodes=1 --ntasks-per-node=2 --partition=gpu --qos=gpu --gres=gpu:p100:1 --mem=8G --pty bash

[<USER>@wn41 ~]$ module load miniconda3

[...]

[<USER>@wn41 ~]$ exit

[<USER>@ui01 ~]$
conda search sklearn

Alternate channels

To search for alternate channels that may provide the conda package you're looking for, navigate to anaconda.org and use the search bar at the top of the page.

To add a new channel: 

conda config --add channels conda-forge

Remove unused packages and caches

To remove unused packages and caches 

conda clean --all

Environments list

To see a list of all available environments run: 

conda env list

References

Managing environments

calcoloscientifico/cluster/softwareapplicativo/conda_user.txt · Ultima modifica: da fabio.spataro
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki