14  Page: Emit A Snakemake Download Workflow

Use quarto-emit to materialize pipeline files without replacing Snakemake

This page sits between the single-job download definition and the full aggregation step. Its job is to show one way a manual can bridge the two: describe the download workflow to the operator, then emit a minimal Snakemake workflow that can run the grouped jobs at scale.

Before emitting a Snakemake workflow, confirm that the manual has already defined:

• the request-grid logic from the orchestration page;
• the single-job parameter contract from the download page;
• the expected output naming pattern for raw ERA5 files.

required_concepts = [
    "request_dicts",
    "download_era5(request_dict, ...)",
    "data/raw/era5land/<country>_<dataset>_<variable>_<year>-<month>.grib",
]
required_concepts

The first emitted artifact can be a small configuration file that records which download jobs Snakemake should expect.

downloads:
  - country: MDG
    dataset: reanalysis-era5-land
    variable: 2m_temperature
    year: 2021
    month: "01"
  - country: NPL
    dataset: reanalysis-era5-land
    variable: total_precipitation
    year: 2021
    month: "01"

This does not replace the operator’s reasoning about which jobs belong in the workflow. It simply materializes that decision into a file Snakemake can read.

The second emitted artifact can be a Snakefile that maps each configured job to the Python download function from the previous page.

# Pseudocode adapted for a Quarto Manual example
configfile: "workflow/downloads.yaml"

DOWNLOADS = config["downloads"]

rule all:
    input:
        expand(
            "data/raw/era5land/{country}_{dataset}_{variable}_{year}-{month}.grib",
            zip,
            country=[job["country"] for job in DOWNLOADS],
            dataset=[job["dataset"] for job in DOWNLOADS],
            variable=[job["variable"] for job in DOWNLOADS],
            year=[job["year"] for job in DOWNLOADS],
            month=[job["month"] for job in DOWNLOADS],
        )

rule download_era5:
    output:
        "data/raw/era5land/{country}_{dataset}_{variable}_{year}-{month}.grib"
    params:
        dataset=lambda wc: wc.dataset,
        variable=lambda wc: wc.variable,
        year=lambda wc: wc.year,
        month=lambda wc: wc.month,
    shell:
        # In a real project this would call the emitted Python download job.
        "python scripts/download_era5.py --country {wildcards.country} "
        "--dataset {params.dataset} --variable {params.variable} "
        "--year {params.year} --month {params.month}"

Finally, we can assert that the emitted files exist and that Snakemake is responsive to the workflow definition.

assert Path("workflow/downloads.yaml").exists()
assert Path("workflow/Snakefile").exists()

import subprocess
result = subprocess.run(["snakemake", "--dry-run"], capture_output=True, text=True)
assert result.returncode == 0, f"Snakemake dry-run failed: {result.stderr}"