Skip to content

feat(workflow): pull_dataset can pull to a given directory - #208

Merged
masonng-astro merged 1 commit into
mainfrom
feat/pull-dataset-directory
Nov 1, 2026
Merged

masonng-astro merged 1 commit into
mainfrom
feat/pull-dataset-directory

Conversation

@chitrangpatel

Copy link
Copy Markdown
Contributor

Why

The CANFAR baseband localization reads its raw data from /arc (CephFS). A probe session on 2026-11-01 read 2 GB from /arc at 68 MB/s but from the session's /scratch (node-local ext4, empty at start and gone with the session) at 3.7 GB/s. To use it, the localization will pull the raw data into its own session's /scratch (baseband-analysis PR to follow).

dtcli.workflow.pull_dataset always pulled to the site's root mount from ~/.datatrail/config.yaml. On CANFAR that file is in the shared home, so changing it would redirect every session's pulls.

What

  • pull_dataset(..., directory=None): when it's given, it's created if needed, passed to pull --directory, and used for the check that every file arrived. Files land at <directory>/data/chime/baseband/raw/..., as under the root mount.
  • Fix in pull --directory: click's resolve_path drops the trailing slash, and the check for files already there joins with +, so it looked in /scratchdata/..., found nothing, and downloaded everything again. The directory now ends with a slash.
  • docs/workflow.md: documents directory.

Tests

  • New: test_pull_dataset_pulls_to_a_given_directory and test_pull_checks_a_given_directory_for_existing_files. The second one fails without the fix.
  • tests/test_workflow.py, test_pull_*: 88 passed.
  • Full suite: the same 12 test_cli.py failures as on main (they need the live Datatrail services).

Needs a release (1.2.0) before baseband-analysis can depend on it.

🤖 Generated with Claude Code

pull_dataset always pulled to the site's root mount from the datatrail
config, which on CANFAR is /arc and shared by every session. A
`directory` argument lets a step pull somewhere else, e.g. the
session's own /scratch, which reads much faster than /arc.

Also fixes `pull --directory`: click drops the trailing slash, so the
check for files already there looked in e.g. /scratchdata/... and
every file was downloaded again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.58%. Comparing base (b743aac) to head (ae3bb39).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #208      +/-   ##
==========================================
- Coverage   88.29%   83.58%   -4.71%     
==========================================
  Files          35       35              
  Lines        4655     4679      +24     
==========================================
- Hits         4110     3911     -199     
- Misses        545      768     +223     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@masonng-astro
masonng-astro merged commit 7f2da01 into main Nov 1, 2026
11 checks passed
@masonng-astro
masonng-astro deleted the feat/pull-dataset-directory branch November 1, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants