Skip to content

ci: add PanDA Compose integration workflow - #1

Open
veprbl wants to merge 54 commits into
masterfrom
ci/panda-compose-integration
Open

veprbl wants to merge 54 commits into
masterfrom
ci/panda-compose-integration

Conversation

@veprbl

@veprbl veprbl commented Nov 17, 2026

Copy link
Copy Markdown
Member

Brings up PanDAWMS/panda-compose (sibling checkout, compose-file merge, same staging order as BNLNPPS/swf-testbed), builds a snakemake worker image from this PR's plugin code, runs a one-job snakemake workflow through --executor panda, and asserts the task reaches 'done' and the output DID is registered in Rucio on MOCK-POSIX.

Trigger: push/PR to master + workflow_dispatch (panda_compose_repo/ref and snakemake_version overridable).

Hermes Agent added 30 commits November 17, 2026 13:02
Brings up PanDAWMS/panda-compose (sibling checkout, compose-file merge,
same staging order as BNLNPPS/swf-testbed), builds a snakemake worker
image from this PR's plugin code, runs a one-job snakemake workflow
through --executor panda, and asserts the task reaches 'done' and the
output DID is registered in Rucio on MOCK-POSIX.

Trigger: push/PR to master + workflow_dispatch (panda_compose_repo/ref
and snakemake_version overridable).
…ts fetched pull/N/merge, missing .env.example)
…heredoc); write patch script as separate file, COPY before RUN
…WORKSPACE} (compose does not expand env syntax)
…mq, wait stable, then up -d (all services); drop nonexistent idds/ruciodb service names
…patch handles the cast; avoids the str-timeout crash entirely)
…rting the plugin loads RUCIO_CONFIG at import time)
…ible with the 41.x client (KeyError 'prefix' on upload)
Hermes Agent added 23 commits November 17, 2026 22:36
…protocol selection, live UploadClient attempt
…ally) — latest may differ in upload path
…p (rucio server bind-mounts it; posix.put falls into the OSError branch that reads self.rse['prefix'] -> KeyError when the dir is missing)
…missing grid proxy (server runs token_authType=None)
…er.veprbl..., rejected by the server as invalid outDS)
…> (Snakefile re-parses on the worker; uuid4 gave host/worker different names)
…archive on exit (rucio plugin raises NotImplementedError on delete, failing the step after the task completed)
…moves the workflow-sources archive on the rucio provider (delete raises NotImplementedError and failed the step)
…e via snakemake --deploy-sources, so the host must not upload/remove it on rucio (delete raises NotImplementedError and failed the run after 100% done)
…al_files=True, which is the actual gate for source-archive upload/remove)
… no-op the rucio plugin's remove() on host+worker instead — upstream raises NotImplementedError on delete, which failed the run after 100% done; worker-side job failure in 44386961232 was caused by the missing source archive
…ubmitter activity is invisible in the current tail
…ker-submitter lines from harvester container
…rns Python repr otherwise); dump harvester-worker container logs + /tmp/harvester_output
…es a type=collection search that never matches file DIDs)
…io storage plugin patch into committed ci/ files

- remove one-shot debug steps/dumps (workspace layout, RSE protocol dump,
  UploadClient probe that also polluted mock:snakemake-uploads with
  ci-probe-runner.txt, redundant always()-log commands, dead TIMEOUT env,
  no-op RUCIO_CLIENT_VERSION re-export)
- ci/worker.Dockerfile replaces the runtime-heredoc Dockerfile+patch script;
  storage plugin version is now a build-arg pinned from workflow env
- ci/patch_rucio_storage_plugin.py replaces the two drifting copies of the
  timeout int-cast + remove() no-op patches (host and worker now share it);
  idempotent, asserts its anchors, fails loudly on plugin version drift
Comment thread .github/workflows/panda-compose.yml Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant