16 RSE Workbench Manual
Adapted from my RSE Workbench Repo
The RSE workbench was the major inspiration for manuals. As you can see, creating a singularity container for reproducible data science on the cluster is a long, involved process. I was able to line up 12 notebooks of procedures, each with specific code to run, artifacts and files left behind, and checks to make at each stage. The end product is that I can now launch 5 or 6 simultaneous project workspaces on the cluster, each of which is running in an isolated singularity container (think Docker and Kubernetes, but on HPC), specifically configured for the project, including Linux dependencies, geospatial packages, my editor of choice, etc.
Putting all of these together for the first time in a critical project is a lengthy, several day process. Redoing it each and every single time a project counts as critical is Sisyphean. The RSE Workbench Manual allows me to learn the complicated procedure once, then duplicate the pages for a new project, change the paths, insert and remove a few project specific packages, and then recompile a new singularity container for the new project in a matter of minutes. This is a huge time saver, and it was also a great way to learn how tools like Spack, Singularity, and Quarto work together to create a reproducible computing environment.
Research software engineers often inherit or support projects where critical setup assumptions are implicit:
- Where does the project live?
- Which files are project-owned?
- Which data are raw, interim, or processed?
- Which dependencies are system-level versus language-level?
- Which steps must happen before others?
- Which runtime assumptions come from the HPC cluster?
- How do we know the environment is ready?
- How do we hand the project off to another researcher or RSE?
A README can explain setup commands, and a CLI can automate a stable workflow. Both are useful. But many research projects sit between those extremes.
A README works well when setup is short, linear, stable, and mostly copy-pasteable. A CLI works well when the workflow is predictable enough for a developer to encode the right defaults. RSE Workbench targets the harder middle case: setup procedures where order matters, later steps depend on earlier artifacts, users need to edit project-specific variables, files are generated along the way, and success needs to be verified.
The goal of this manual is not to replace documentation, package managers, containers, workflow engines, or tests. The goal is to compose them into a guided, editable project assembly process.
This initial use case is reproducible computing environments for climate and public health data science on shared HPC infrastructure.
The current prototype is designed around:
- project-owned files;
- data, code, and notebook organization;
- R and Python analysis code;
- Quarto notebooks and reports;
- Spack-managed system dependencies;
- language-level package management;
- container and runtime isolation;
- activation and dependency-check scripts;
- Slurm launch support;
- pytest-based project inspections.
Future manuals could target other recurring workflows, such as:
- a lab’s standard data analysis pipeline;
- a machine-learning workflow;
- a geospatial processing workflow;
- an end-to-end data cleaning, modeling, and deployment workflow.