What is data curation? Data curation involves collecting, cleaning, and joining raw data sources to a common format. Most real-world machine learning or data science projects involve data curation to some extent in order to get data ready for analysis.
What are AI coding agents? Agents like Codex, Claude Code, or Opencode wrap an LLM and allow it to execute tool calls: in addition to writing code, the LLM can invoke external programs a like a command-line interface or Python interpreter, giving it the ability to take actions on the user's behalf.
Why did we create this repo? We have been experimenting with using agents for data cleaning and curation, and they are pretty great. It is very quick to write a script that, for example, reads in a raw data file, parses and visualizes its contents, and makes modifications according to a natural language prompt. There are risks, however---because agents can modify files themselves, it may be impossible to trace or reproduce all the changes that they make. So we came up with a framework that adds a layer of reproducibility to data curation with coding agents. It works like this:
The main user-facing process is an interactive loop with an agent and a browser-based graphical user interface (left). We use the agent to both help us perform data modifications as well as to create interactive visualization components that allow us to verify any data changes that were made. The agent does not touch the data directly, however; dataset modifications are mediated through a REST API that separates agents and raw data (center). On the backend, proposed data modifications are recorded in an executable data provenance graph, maintaining the full data version history in a reproducible manner (right). This version history is saved alongside the dataset, allowing any dataset version to be reconstructed from the raw data by playing forward the transformations stored in the graph, and providing a record of all modifications for posterity. Altogether, we call this framework Scrub Data: a scaffold for building data apps backed by a reproducible data graph.
The Scrub Data framework is implemented as a lightweight web application that is designed to be modified by AI agents in response to user requests. This is an important design decision that is unique to agentic workflows—--the software itself is malleable and is meant to be customized by each individual user to suit their needs. You will clone this repository, and ask the agent to help you customize it in real time according to your visualization and curation needs.
Agents can perform data modifications, but are instructed to do so by submitting executable transformations through the API. The agents gather this context upon initialization so you don't have to worry about giving them instructions. Submitted transformations are logged in a directed acyclic graph that represents the full version history of a project and can be easily be used to reproduce any previous version of the dataset.
To begin working with the codebase:
- Clone this repository.
- Put your raw data files in
data/<project>/. - Launch an agent instance from your terminal from inside the codebase, for example by running
codex,claude code, oropencode. - Launch the web application from another terminal window with
python server.py, and direct your browser to the appropriate port onlocalhost(5420 by default). - Ask your agent to extend the starter app in
server.pyandstatic/according to your needs.
See examples/trajectory/ for a reference starter app built for animal movement (GPS) data.
Repository layout:
app/ generic lineage, execution, preview, and HTTP internals
server.py default starter app server
static/ default starter app frontend
examples/trajectory/ richer reference app
data/<project>/ project inputs plus scrubdata state
docs/ minimal contracts for agents
These contracts provide instructions for how the agent should behave and what actions it should take: