12  Page:Orchestrate The ERA5 Pipeline

Declare the request grid and split it into manageable download jobs

Caution

The content on this page contains pseudocode for demonstration purposes.

This page orchestrates the data aggregation pipeline. Its job is to define the overall ERA5 request space and turn it into a set of grouped API calls that later pages can execute.

Before building the request grid, confirm that the project already knows where its staged inputs, stagecoach manifest, and downstream download notebook live.

library(here)

stagecoach_manifest <- here("stagecoach_manifest.yml")
input_dir <- here("data", "inputs")

# In a real manual, these would be concrete prerequisite checks.
stagecoach_manifest
input_dir

Our job defines a large grid of request parameters covering country, dataset, variable, year, month, day, and hour. It then filters out invalid dates and avoids redundant hourly requests for time-invariant variables. End users of a manual could use this space to customize, tinker with, and experiment with the request grid before it is split into download jobs, and are encouraged to exercise whatever methods they prefer to get there, including data transformation libraries, data visualization tools, and other data analysis frameworks.

library(tibble)
library(tidyr)
library(dplyr)
library(lubridate)
library(stringr)

cds_api_config <- expand_grid(
  years = 2010:2025,
  months = str_pad(1:12, 2, pad = "0"),
  days = 1:31,
  hours = paste0(str_pad(0:23, 2, pad = "0"), ":00"),
  countries = c("MDG", "NPL"),
  datasets = c("reanalysis-era5-land", "reanalysis-era5-single-levels"),
  variables = c(
    "2m_temperature",
    "2m_dewpoint_temperature",
    "volumetric_soil_water_layer_1",
    "volumetric_soil_water_layer_2",
    "surface_solar_radiation_downwards",
    "total_precipitation",
    "10m_u_component_of_wind",
    "10m_v_component_of_wind",
    "surface_pressure",
    "soil_temperature_level_1",
    "geopotential"
  ),
  data_format = "grib",
  download_format = "unarchived"
) |>
  filter(
    !(months %in% c(4, 6, 9, 11) & days == 31),
    !(months == 2 & days > 29),
    !(months == 2 & days == 29 & !leap_year(years)),
    !(variables == "geopotential" & hours != "00:00")
  )

The notebook does not send one API call for every row. Instead, it groups the grid by country, dataset, variable, year, and month, then converts each group into a single request dictionary that a downstream job can execute.

library(purrr)

request_groups <- cds_api_config |>
  group_by(countries, datasets, variables, years, months) |>
  group_split()

request_dicts <- map(request_groups, \(x) {
  list(
    country = unique(x$countries),
    dataset = unique(x$datasets),
    variable = unique(x$variables),
    year = unique(x$years),
    month = unique(x$months),
    day = unique(x$days),
    time = unique(x$hours),
    data_format = unique(x$data_format),
    download_format = unique(x$download_format)
  )
})

In some cases, authors may choose not to include a file artifact in a check block, but instead an end-of-page summary that describes what the operator should be able to explain or demonstrate.

# This page is complete when the operator can explain:
# - which inputs are required before downloads begin,
# - which dimensions define the CDS request grid,
# - why invalid dates are filtered out,
# - and why requests are grouped by country, dataset, variable, year, and month.
stopifnot({
  "I can explain the required inputs, the request grid dimensions, the filtering of invalid dates, and the grouping of requests."
})