Motivation

Container systems like Docker and Apptainer are useful, but there are two use cases that are problematic. Bulker solves both.

How to containerize a workflow

There’s a lot of tooling around individual containers or sets of containers running simultaneously (like Kubernetes or Docker Swarm). But what about containers for workflows that run a series of commands sequentially? In a workflow, we’re only running one container at a time, so we don’t need a container swarm. But going one-by-one is painful when your workflow needs 15 different tools. What’s the best way to containerize a workflow?

One solution is to build a mega-image with all 15 tools. This is common, but it’s inefficient because different workflows using the same software can’t share it. It makes updating individual components hard, bloats images, and goes against the modular philosophy of containers where each tool has its own image.

So, if not mega-images, how should we containerize a workflow that needs many commands? Using individual images sounds nice, but there are challenges:

1. Distribution. It would be nice to distribute a collection of images, but there isn’t a way to do this easily.

2. Portability. Using individual images can reduce portability. If each command requires a different container, the workflow needs to specify the container for each command. Hardcoding docker run ... makes the workflow dependent on a specific container engine. Making it flexible enough for Docker, Apptainer, or no containers at all adds complexity.

How to use one installation across computing environments

I use many different computing environments: desktop at work, remote servers, laptop. I want common tools on all of these systems, but I don’t want to install and maintain software 5 times. This seems like the perfect use case for containers, but the HPC runs Apptainer while I use Docker at home. What I want is to clone a repo on each computer and automatically have pandoc, samtools, latex, and R available without doing anything else — and have that work across container engines.

Eventually I refined my own docker system — shell scripts with a few variables and manually created scripts for each executable stored in a git repo. But the volumes and environment variables differ between computing environments. For example, on the HPC I need to mount a filesystem that doesn’t exist on my laptop.

How bulker solves these problems

Bulker merges three important concepts:

Containerized executables. Bulker creates shimlinks — symlinks to the bulker binary that dispatch to the right container based on the command name (argv[0]). To the user, they behave identically to native tools. Any workflow can use containers without being changed. Bulker handles the details of both the container engine and environment-specific settings like volumes, providing a single user interface.

Decoupling container settings. Some container settings depend on the tool, others on the environment. Bulker decouples them: tool-specific settings go in the manifest, and environment-specific settings (volumes, environment variables, container engine) go in the config file. Bulker combines them at runtime.

Bulker crates. For distribution, bulker introduces the crate — a collection of containerized commands. To build a crate, you write a manifest listing commands and their container images. Users can activate all the software from a manifest with bulker activate namespace/toolset, making every tool available in the PATH. This makes the workflow portable because it’s completely unaware of containers.

Manifests are also useful beyond workflows. I wrote a personal manifest with all the tools I want in my workspace. I configure each environment (HPC, laptop, desktop, cloud) with a local config file. Bulker pairs the manifest with the environment config at runtime. To install a new package across all my systems, I update the manifest and re-activate: bulker activate databio/nsheff.