Running Bulker on HPC Clusters

Bulker works on HPC clusters using Apptainer as the container engine. Two main concerns arise in HPC environments: managing TMPDIR for shimlinks, and integrating with SLURM job schedulers.

The TMPDIR issue

Bulker creates a temporary directory of shimlinks each time you activate a crate. By default, this directory is created in $TMPDIR (which falls back to /tmp). On many HPC systems, compute nodes cannot access shimlinks created on the login node, or /tmp may be restricted or node-local.

The solution is to set TMPDIR to a shared scratch location before activating bulker:

export TMPDIR=/scratch/$USER/tmp
mkdir -p $TMPDIR
bulker activate my-crate

This ensures the shimlink directory is created on a filesystem accessible to both login and compute nodes.

Standalone usage (without a pipeline runner)

When submitting jobs manually (without looper or another pipeline runner), activation must happen on the compute node so that shimlinks are created in that node’s $TMPDIR. Use bulker activate -e (eval mode) inside your job script:

#!/bin/bash
#SBATCH --job-name=my-job
#SBATCH --output=my-job.log

export TMPDIR=/scratch/$USER/tmp/$SLURM_JOB_ID
mkdir -p $TMPDIR

eval "$(bulker init-shell bash)"
bulker activate -e my-crate

# Commands from the crate are now on PATH
samtools view input.bam

The -e flag causes activate to modify the current shell’s PATH via eval, rather than spawning a subshell. This is necessary inside batch scripts where you want subsequent commands in the same script to use the crate.

With looper

The {PRE_COMMAND} hook in looper’s divvy templates is the cleanest way to handle TMPDIR setup and module loading. Set pre_command in the looper config cli.run.compute section:

cli:
  run:
    compute:
      pre_command: "export TMPDIR=/scratch/$USER/tmp && mkdir -p $TMPDIR && module load apptainer"

Then use the built-in bulker_slurm compute package:

looper run --package bulker_slurm --compute bulker_crate=my-crate

The bulker_slurm template handles crate activation on the compute node. The PRE_COMMAND runs before anything else, so TMPDIR and modules are ready before bulker activates. See the divvy CLI reference for more detail on PRE_COMMAND and other template variables.

Module loads

On systems where apptainer is not on PATH by default, use module load apptainer (or whatever the site-specific module name is) in the PRE_COMMAND. This ensures the container engine is available when bulker constructs its exec commands:

export TMPDIR=/scratch/$USER/tmp && mkdir -p $TMPDIR && module load apptainer

Pre-pulling images

Apptainer converts Docker images to .sif format on first use, which can be slow. To avoid this happening mid-job on a compute node, pull images from the login node before submitting jobs:

bulker crate install my-crate -b

The -b flag pulls all container images upfront. Since the SIF cache is stored in a shared location (~/.local/share/apptainer/images by default), compute nodes will find the pre-converted images.

Shared crate installations for teams

If multiple users on a cluster need the same crates, you can set up a shared crate installation to avoid redundant image conversions. See the multi-user guide for details.