Modern Software on Old Cluster – Algorithms and Complexity Group

Modern Software on Old SGE Cluster

Although the cluster is based on an older Linux distribution and uses Sun Grid Engine (SGE),
you can still run modern software stacks (e.g. Python 3.11, current R, irace)
without touching any system-wide packages.

This is achieved via a central Miniforge installation combined with Conda/Mamba environments.

Central installation:

/home1/share/conda/miniforge3

Concept

Miniforge is installed centrally (read-only for users)
Two types of Conda/Mamba environments are supported:
- centrally maintained environments (e.g. for teaching)
- user-managed environments (e.g. for research projects)
Both types are activated in the same way using Mamba
No interference with system Python or system R
Works both interactively and in SGE batch jobs

For teaching and common workloads, centrally maintained environments (e.g. R with irace)
are provided to ensure a consistent and reproducible software setup across all users.

For research and project-specific needs, users are encouraged to create and maintain their own
environments in their home directories.

1. One-time per-user Conda configuration

Run this once to ensure all environments and packages live in your home directory:

mkdir -p ~/.conda/envs ~/.conda/pkgs

cat > ~/.condarc << 'EOF'
envs_dirs:
  - ~/.conda/envs
pkgs_dirs:
  - ~/.conda/pkgs
channels:
  - conda-forge
channel_priority: strict
auto_activate_base: false
EOF

2. Initialize your shell for Mamba (one-time)

This enables mamba activate in your shell.

source /home1/share/conda/miniforge3/bin/activate
mamba shell init --shell bash --root-prefix ~/.local/share/mamba
source ~/.bashrc

From now on, your shell knows the mamba commands.

3. Example A: Centrally Maintained Environment (R + irace)

This environment is centrally maintained and primarily intended for use in teaching and coursework.
Users should not install or modify packages in this environment.

A recent version of R and irace is provided via the following Conda environment:

r-irace

Example SGE job script (`irace_test.sh`)

#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#$ -V
#$ -N irace_test
#$ -l h_vmem=4G

# Load Miniforge and Mamba integration
source /home1/share/conda/miniforge3/bin/activate
eval "$(mamba shell hook --shell bash)"
mamba activate r-irace

Rscript -e 'cat(R.version.string, "\n"); library(irace); print(packageVersion("irace"))'

Submit the job:

qsub irace_test.sh

4. Example B: User-Managed Environment (Modern Python)

User-managed environments allow full flexibility and are recommended for research projects and individual workflows.

Create a Python 3.11 environment

mamba create -n py311 python=3.11
mamba activate py311
python --version

Expected output:

Python 3.11.x

Example SGE job script (`py-hello.sh`)

#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#$ -N py-hello
#$ -o hello.out
#$ -e hello.err

# Load Miniforge and Mamba integration
source /home1/share/conda/miniforge3/bin/activate
eval "$(mamba shell hook --shell bash)"
mamba activate py311

echo "Running on host: $(hostname)"
echo -n "Python version: "
python --version