Data Ingress: Where it All Begins

On the Frontier, we use the term “data ingress,” broadly. Rather than treating ingestion, registration, cataloging, and discovery as separate systems, we view them as parts of a single process whose goal is to transform incoming data into a managed scientific asset. This stage sets the foundation for the entire data lifecycle, ensuring that data is properly captured, organized, and accessible for future use. In this deep dive, we will explore the architecture of the Golden Lab’s data ingress setup, highlight the tools and technologies that enable efficient data collection and management, and discuss the design decisions and trade-offs involved in designing our system.

Some Motivating Thoughts

While it can be tempting to believe that only big analytics companies and tech giants need to worry about the architecture of their data pipelines, the reality is that any organization can benefit from a well-designed data infrastructure, even if the data itself is small. This is because at some point in the future, someone will need to access a recorded paper trail of a dataset, understand the decisions made about it, and potentially reproduce the analysis. Being deliberate about the data ingress process sounds like tedium in the present¹, but it pays off in the long run. This aligns with our 4th data science principle, “Good Habits Compound Over Time,” and speaks to an interesting idea about scaling that academia often misses:

Important

scaling technology in an organization is not necessarily about the number of people involved, the execution of fancy tools, or even the physical size of the data.

Scaling technology in an organization is about tackling digital complexity with deliberate design decisions. In other words, scaling is about doing things on purpose, and reducing the number of things that happen by accident.²

The Golden Lab’s decision to build a data ingress system is an example of this idea in action. Instead of hoping we will remember where a dataset came from, we design a system that encourages³ us to record that information at the moment of data collection with as little friction as possible. Instead of hoping we will be able to find a dataset in the future, we design a system that organizes datasets in a way that makes them easy to find and self-organizing. Instead of hoping we will be able to understand the decisions we made about a dataset, we design a system that requires us to document those decisions in a standardized way as they happen and publishes that knowledge for others to review for accuracy.

Put simply, it’s wishful thinking to hope to recruit the perfect student who remembers every conversation, writes every transformation down, and organizes all of their work perfectly. Instead, we design systems that help us do those things painlessly, and create a culture where that activity is the norm.

The Golden Lab’s Data Ingress Strategy

On the Frontier, we have designed our data ingress system to be flexible and scalable, allowing us to handle a wide variety of data sources and formats while maintaining a consistent and organized structure.

To design this system, we first have to think about what ways in which we receive different types of data related to our work. While every dataset deserves appropriate stewardship, the mechanisms by which data enters the lab vary considerably. To create a concrete list, I recommend thinking about data ingress in 3 broad categories:

1. True Streaming Data: This is data that is generated in real-time and needs to be ingested continuously. Examples include sensor data, satellite data, or any other data source that produces a constant stream of information. Due to their high frequency, high velocity, and automatic nature, these data usually come from other machines. In Climate-Smart Public Health projects, we stream weather data from sensor APIs, such as AirQO.

2. Batch Data: This data is generated and collected in batches, rather than in real-time. In this case, the data collection procedure was time constrained, and so the data generated from it is also time constrained. Examples include surveys and experiments that are conducted at fixed points in time. A data engineer may have automation scripts to deal with this type of data, but importantly there is no need for the engineer to have an “always on” technology, ready to receive data at any moment. Instead, most of the data can safely be staged in some collection area and then processed when it is needed. In the Golden Lab, and in small-to-medium scientific arenas more broadly, this is the most common type of data we deal with — think emails from collaborators, spreadsheets, survey data, etc.

3. Scheduled Batch Data: This is data that is generated in batches but needs to be ingested and processed to some degree on a regular schedule in order to be useful. The definition is certainly more squishy — it’s almost Pseudo-Streaming/Pseudo-Batch Data — but the core idea is that if you were to leave this data sitting in collection for too long, the technical debt of having to deal with a large backlog would become unmanageable. Just think of the last time you turned off your email notifications for a week and then came back to find 500 unread emails. The longer you wait to deal with it, the more overwhelming it becomes. So while almost all of our data is batch data, we make the active decision to treat some of it as pseudo-streaming, pseudo-batch data, and we design our data ingress system to accommodate that. A good example is the South West Cohort data, which is updated on a regular schedule and needs to be processed in a timely manner for its papers to be published. Treating this as streaming data would be over-engineering, as the data only comes in once every few months; whatever we build would largely sit idle. On the other hand, treating it as batch would be unwise as well, because it may take an afternoon to process that data, OR — and this very much happened — we may find that the data wrangling process is more complicated than we expected, and as a result, we missed an important deadline that depended on it. Having a strategy for dealing with this type of data is greatly beneficial, and as we’ll see below, fits into a larger strategy for data ingress that is designed to be flexible and scalable.

On The Frontier, Prospectors are the tools responsible for managing data ingress and preprocessing. Different Prospectors may operate continuously, on schedules, or entirely on-demand, depending on the characteristics of the data source. We’ll learn more about Prospectors in another article.⁴ In sum, data ingress is the process by which new resources cross the Frontier boundary and become known to the Gold Mine, and Prospectors are the actors orchestrating that process.

System Design and Architecture

Our data ingress system is built on a combination of two commercial level tools provided by Harvard:

• Qualtrics, for which Harvard has on-prem licenses for storage of survey data, and which provides a robust API for data access and management.

• Globus, which is a data transfer service that “sits on top of” FASRC’s shared file system, and provides a user-friendly interface for transferring data to and from the cluster, as well as a robust API for automating data transfers.

Regardless of what tools we chose to use, buy, or build, a key design principle of the Frontier is that data should enter the system once. After ingress, all subsequent processing occurs within the Frontier itself. This minimizes duplicated downloads, reduces opportunities for human error, and creates a clear provenance trail showing how data moved from source to scientific result.

We also use Google Sheets for additional visibility of the system. The user story looks something like this:

• When a researcher identifies an interesting dataset, they might want to include it in their project. Because the researcher is a good steward of data and citizen of The Frontier, they will want to make sure that the dataset will be in an accessible location for themselves and for others, and that people know what it is and that it exists. To do this, they need a way to register the dataset for visibility

• The “Send Us Your Data” Qualtrics survey is purposely designed for this use case. Using this survey link, the researcher begins a new data ingress survey. The survey includes sections to understand a few details like:

  • What is the dataset called?
  • Where did it come from?
  • What does it contain?
  • Who should be contacted for questions about it?

• Once the researcher reaches the end of the survey, they will be prompted to upload the dataset using one of two methods:

  • if the dataset is small, they can upload it directly through the survey;
  • if the dataset is large, the survey concludes, recording the user’s email address. After a brief delay, the user receives an email with instructions on how to use Globus upload their data.

• In the background, Qualtrics adds the dataset metadata to our survey tracking Google Sheet, for redundant visibility and record keeping⁵.

• Once the data is uploaded, the Qualtrics Prospector⁶ dashboard checks the tracking sheet each morning for new entries. If it finds any, it transfers the data from the Qualtrics server or Globus endpoint into the Frontier’s gold_mine, specifically the 01_ore layer located at /n/holylabs/LABS/cgolden_lab/Lab/frontier/gold_mine/01_ore.

• Finally, because the dataset has been automatically registered and cataloged during the ingress process, it is immediately discoverable and available for use. Researchers can request access to the dataset using stagecoach by adding the appropriate gold_mine path to their project’s staging manifest. When stagecoach prepares the project environment, it records exactly which datasets were requested and where they originated from, creating a reproducible audit trail of the data’s provenance. In this way, researchers gain the benefits of rigorous data governance and reproducibility without needing to manually maintain records of every file they use.

Tip

A useful rule of thumb is:

Data enters the Frontier once through a Prospector, lives in the Gold Mine, and reaches projects through the Stagecoach.

flowchart TD

    subgraph User
        A[Researcher identifies dataset]
    end

    subgraph Registration
        B[Qualtrics Data Ingress Survey]
    end

    subgraph Transfer
        C[Small Dataset Upload]
        D[Large Dataset Upload via Globus]
    end

    subgraph Visibility
        E[Survey Tracking Google Sheet]
    end

    subgraph Prospectors
        F[Qualtrics Prospector]
    end

    subgraph GoldMine
        G[gold_mine/01_ore]
    end

    subgraph Projects
        H[Stagecoach Manifest]
        I[Research Project]
    end

    A --> B

    B --> C
    B --> D

    B --> E

    C --> F
    D --> F

    E --> F

    F --> G

    G --> H
    H --> I

Closing Thoughts

It can be tempting to rush into data analysis with whatever dataset you can get your hands on, as science can be a messy (and exciting) iterative process. But in order for science to be a cumulative, reliable, truth-seeking endeavour, we must act with the end in mind — preempt the rigorous proof that all bold claims will require. Doing so is deceptively simple — design working environments that enable us to easily and frictionlessly manage the provenance of our experiments and the data they generate. Recall that the real substance of science isn’t the fancy model, or the impressive figures, or even the journal you send your paper to. It’s in the ability to say, without a shadow of a doubt, “I had a hunch about this unexplained phenomenon, and so I designed an experiment to test it. I collected data, analyzed it, and found that my hunch was correct. Here is exactly how I did it, and here is the data that supports my claim.”

Today, I encourage you to make the pledge to not wait any longer; you don’t have to be an expert to start using these systems, and you don’t have to wait until a “worthwhile project” before you implement these practices. Embrace wise data stewardship from the very beginning of our projects, and see how it transforms your work and the work of your collaborators. The best time to establish data provenance is the moment data enters the Frontier. Every step after that becomes easier.

Footnotes

1. After all, YOU are special. YOU will definitely remember each and every one of the steps you took, perfectly, and without error, several years from now, right…? Right….? RIGHT…………..?

2. By thinking of scaling this way, we can better embrace our own capability to scale technology as individuals in our own lives. Ever been frustrated by the fact that you have a hard time finding a certain file on your computer? Maybe it’s time to scale your personal file organization system by using a more deliberate folder structure, naming convention, or search tool. Ever been frustrated by how long it takes you to set up a project? Scale your project setup process by creating a project template that you can reuse. Etc…

3. We say “encourage” rather than “require” because we want to strike a balance between enforcing best practices and allowing for flexibility and creativity in how we work with data. We want to create a culture of good data management habits, but we also want to avoid being overly rigid or bureaucratic. By designing a system that encourages good habits without being too prescriptive, we hope to foster a sense of ownership and responsibility among lab members while still allowing for individual differences in how people approach their work. It’s part of the social contract in principle 4: “Good Habits Compound Over Time.”

4. Prospectors are a family of data ingestion and processing tools on the Frontier. They are responsible for discovering new data sources, validating incoming data, performing transformations, and depositing outputs into the Gold Mine. You may have noticed the growing number of “prospector-something” repositories on GitHub.

5. While Qualtrics serves as the system of record, the Google Sheet acts as a lightweight operational dashboard that makes newly submitted datasets visible to the broader team and to automated Prospectors.

6. Prospectors are a family of data ingestion and processing tools on the Frontier. They are responsible for discovering new data sources, validating incoming data, performing transformations, and depositing outputs into the Gold Mine. You may have noticed the growing number of “prospector-something” repositories on GitHub.