Skip to content

feat(workflow): pull and clear as functions for Workflow pipelines - #206

Merged
chitrangpatel merged 1 commit into
mainfrom
feat/workflow-functions
Nov 1, 2026
Merged

chitrangpatel merged 1 commit into
mainfrom
feat/workflow-functions

Conversation

@chitrangpatel

@chitrangpatel chitrangpatel commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Why

Workflow pipelines (e.g. baseband localization on CANFAR, CHIMEFRB/baseband-analysis#661) stage and clear data with datatrail pull / datatrail clear. Run as commands, the runner captures their output into the Work's results in MongoDB, and it doesn't show in the step's logs. Run as functions, the output goes to the session's logs, but the click commands can't be called correctly:

  • pull reports several failures only on the console and exits 0: no valid CADC certificate, minoc down, an unknown scope, no files found at minoc. A step that pulled nothing is marked a success, and the next step runs on missing data. This also affects the CLI used as a command today.
  • ctx.exit(1) is returned, not raised, when a click command is called with standalone_mode=False (as Workflow does), so those failures are successes too.
  • The config check before a pull ([ -f ~/.datatrail/config.yaml ] || datatrail config init --site canfar) can't be expressed in a function step.
  • -v is a counted option, which Workflow can't pass to a click command.

What

dtcli/workflow.py, without changing the CLI:

  • pull_dataset(scope, dataset, cores=1, site=None, verbose=1) runs pull --force, then checks with find_missing_dataset_files that every file is here. It raises if the command returned an exit code, the dataset can't be checked, files are missing, or there are none.
  • clear_dataset(scope, dataset, site=None, clear_parents=False, verbose=1) finds the dataset's path as clear does, runs clear --force, and raises if the path is still there. A dataset with no local copy is left alone and isn't a failure, so it can clean up after a failed pull.
  • Both create the configuration for site when there's none (like config init --site <site>), and raise if it's set up for another site.
  • Both return (results, products, plots), the shape Workflow expects of a function. Results: scope, dataset, plus files/directory (pull) or path/cleared (clear).

Use in a pipeline step:

work:
  function: dtcli.workflow.pull_dataset
  parameters:
    scope: chime.event.baseband.raw
    dataset: "123456789"
    cores: 4
    site: canfar

docs/workflow.md documents both, linked from mkdocs.yml under User Guide.

Tests

tests/test_workflow.py (14 tests): the outcome shape and arguments passed; creating the configuration; refusing another site's; raising for an exit code, an unknown scope, missing files, no files and click errors; clearing and checking; no local copy; files left behind; the dataset's path not found or the server unreachable.

I didn't set up this repo's uv environment locally; I ran them in the baseband-analysis:v1.12.0 image with this branch in place of its installed dtcli (all 14 pass). Pre-commit passes. Please check CI.

🤖 Generated with Claude Code

dtcli.workflow.pull_dataset and clear_dataset run `datatrail pull`/`clear` in
the calling process, so a Workflow step's logs show their output, then check
what they did and raise if they failed: pull reports some failures (no CADC
certificate, minoc down, unknown scope, no files) only on the console and
exits 0, and ctx.exit(1) is returned rather than raised under
standalone_mode=False. Both create the configuration for `site` when there's
none, and return (results, products, plots) as Workflow expects. Clearing a
dataset with no local copy succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.76543% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.48%. Comparing base (fc30032) to head (90a0455).

Files with missing lines Patch % Lines
dtcli/workflow.py 96.61% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #206      +/-   ##
==========================================
- Coverage   87.91%   83.48%   -4.44%     
==========================================
  Files          33       35       +2     
  Lines        4493     4655     +162     
==========================================
- Hits         3950     3886      -64     
- Misses        543      769     +226     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@chitrangpatel
chitrangpatel merged commit de80e6d into main Nov 1, 2026
11 checks passed
@chitrangpatel
chitrangpatel deleted the feat/workflow-functions branch November 1, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants