[ryuwd]

Identifies each CWL process by hash_json of the process as submitted in a cwltool --pack document, with each step's run replaced by its child's identity, so a workflow's identity covers its children's. Nothing is normalised: another cwltool version, an edited doc or a renamed file only costs a duplicate row, and DiracX never rewrites what the user tested. check_self_contained refuses documents that need anything from outside themselves ($ keys other than $graph/$namespaces, a run outside the $graph, File/Directory given by path or location), and subgraph returns the document one transformation runs. A worked example is in docs/dev/explanations/cwl-process-identity.md.

Closes #1106. Part of #1080.

[chrisburr]

I've spent a while going through this and I get the impression it would be better/simpler to do cwltool --pack? In DiracX we would then only need to:

1. Client: run cwltool --pack, take the dirac: hints out into the transformation fields and drop them from the document.
2. Server: split the $graph into components, replace each run: #x with the child's hash and hash each component with hash_json.
3. Both: run one shared check_self_contained() (stdlib only, in diracx-core) so the document never needs anything from outside once it reaches DiracX:
   • no $ keys except $namespaces ($schemas gets fetched)
   • every run is a #id within the $graph
   • requirement/hint classes from an allowlist (cwltool tries to fetch unknown ones)
   • File/Directory literals only with inline contents (pack turns a local path into the submitter's absolute file:/// path and keeps URLs)
   • no dockerLoad/dockerImport/dockerFile

The identity is then the hash of what was submitted rather than of a normalised form. A different cwltool version, a renamed file or an edited doc gives a new id, but that only costs a duplicate row. In exchange we drop the rules that reimplement schema-salad's resolution.

[aldbr]

I haven't had a look at the PR yet, but your suggestion makes sense to me.

1. Client: run cwltool --pack, take the dirac: hints out into the transformation fields and drop them from the document.

I think packing the cwl from the client was the idea we discussed during the workshop, am I wrong?
Note: cwl_utils.pack.pack would work in python (instead of running cwltool --pack).
Though, it's should not be part of this PR, should it?

• no dockerLoad/dockerImport/dockerFile

Note that @natthan-pigoux made it possible to get docker images from CVMFS if I remember correctly. Can you remind us how it works please?

[natthan-pigoux]

Note that @natthan-pigoux made it possible to get docker images from CVMFS if I remember correctly. Can you remind us how it works please?

It was not directly targeting CVMFS but it was for dockerPull: to handle unpacked images (that the way images are stored on cvmfs). So if you target an unpacked image from CVMFS through dockerPull: it works.

[read-the-docs-community]

Documentation build overview

📚 diracx | 🛠️ Build #35647448 | 📁 Comparing f716e6a against latest (833c593)

  🔍 Preview build  

24 files changed · + 2 added · ± 22 modified

+ Added

• dev/explanations/cwl-process-identity/index.html
• dev/explanations/tasks/dummy-compute-backend/index.html

± Modified

• admin/explanations/architecture_diagram/index.html
• admin/explanations/chart-structure/index.html
• admin/explanations/tasks/index.html
• admin/reference/env-variables/index.html
• admin/reference/values/index.html
• admin/tutorials/monitor-tasks/index.html
• dev/explanations/dependency-management/index.html
• dev/explanations/tasks/index.html
• dev/explanations/web-architecture/index.html
• dev/explanations/web-testing/index.html
• and 12 more...

[aldbr]

• requirement/hint classes from an allowlist (cwltool tries to fetch unknown ones)
• no dockerLoad/dockerImport/dockerFile

The following points are listed in #1107 so I assume they are expected to come later.

[aldbr]

Could you just check what would happen with the following examples please?
(this is kind of niche, but it can happen if you would try to submit a CWL process without passing through the diracx client and directly go through the REST API for instance)

• duplicate ids:

{
    "cwlVersion": "v1.2",
    "$graph": [
      {"id": "#main", "class": "Workflow",
       "steps": [{"id": "sim",   "run": "#gauss.cwl"},
                 {"id": "merge", "run": "#merge.cwl"}]},
      {"id": "#gauss.cwl", "class": "Workflow",
       "steps": [{"id": "run", "run": "#lb-run-app.cwl"}]},
      {"id": "#lb-run-app.cwl", "class": "CommandLineTool",
       "baseCommand": ["lb-run", "Gauss/v56r1"]},          // entry A
      {"id": "#merge.cwl", "class": "Workflow",
       "steps": [{"id": "run", "run": "#lb-run-app.cwl"}]},
      {"id": "#lb-run-app.cwl", "class": "CommandLineTool",
       "baseCommand": ["lb-run", "--verbose", "LHCb/v57r2"]} // entry B, same id
    ]  
  }

• missing ids:

{
    "cwlVersion": "v1.2",
    "$graph": [
      {"class": "CommandLineTool", "baseCommand": ["echo", "first"]},
      {"class": "CommandLineTool", "baseCommand": ["echo", "second"]}
    ] 
  }

[aldbr]

Suggested change

	See ``docs/dev/explanations/cwl-process-store.md`` for a worked example.
	See ``docs/dev/explanations/cwl-process-identity.md`` for a worked example.