Skip to content

Read and write the generated data in napari/plugin-api-data - #2

Draft
TimMonko wants to merge 1 commit into
mainfrom
cf/01-data-repo
Draft

TimMonko wants to merge 1 commit into
mainfrom
cf/01-data-repo

Conversation

@TimMonko

Copy link
Copy Markdown
Member

npe2api committed ~40 MB of JSON under public/ to main every two hours;
its history was 62,585 commits, 86% of them data. Here the data lives in
its own repository, napari/plugin-api-data, which keeps every snapshot
since 2022 without weighing down this one. Workflows check it out into
public/, so no script changes: bigquery.py, fetch_manifests.py and
reindex.py still read and write public/ as before.

  • update.yml checks out the latest snapshot into public/ with
    DATA_DEPLOY_KEY (a deploy key with write access to the data repository
    only), validates the result with tests/test_output.py (new: invalid
    data is no longer committed) and commits it there. A concurrency group
    stops two runs committing at once. Pull requests check out the data
    read-only and commit nothing.
  • python-test.yml checks out the data so tests/test_output.py has some.
  • public/ is ignored; CONTRIBUTING.md explains cloning the data with
    --depth 1 for local work.

Stack created with GitHub Stacks CLI • Give Feedback 💬

@TimMonko
TimMonko added this pull request to stack #9 November 20, 2026 06:03
Comment thread .github/workflows/update.yml Fixed
npe2api committed ~40 MB of JSON under public/ to main every two hours;
its history was 62,585 commits, 86% of them data. Here the data lives in
its own repository, napari/plugin-api-data, which keeps every snapshot
since 2022 without weighing down this one. Workflows check it out into
public/, so no script changes: bigquery.py, fetch_manifests.py and
reindex.py still read and write public/ as before.

- update.yml checks out the latest snapshot into public/ with
  DATA_DEPLOY_KEY (a deploy key with write access to the data repository
  only), validates the result with tests/test_output.py (new: invalid
  data is no longer committed) and commits it there. A concurrency group
  stops two runs committing at once. Pull requests check out the data
  read-only and commit nothing.
- python-test.yml checks out the data so tests/test_output.py has some.
- public/ is ignored; CONTRIBUTING.md explains cloning the data with
  --depth 1 for local work.
- The old Next.js build in the Test workflow gets the data checkout too,
  until the next PR removes the app.
- Python Test runs with read-only contents permission.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants