Repository navigation
225 add a page to begin documenting standards for pipelines shared externally - #238
arielleleon wants to merge 10 commits into
Conversation
…ous topics; add topic pages.
alejoe91
left a comment
alejoe91
left a comment
There was a problem hiding this comment.
Thanks @arielleleon
Added some comments as talking points
|
|
||
| ## Processing pipeline development | ||
|
|
||
| Pipelines are a standardized series of processing steps that take raw data from a single modality and typically produce an NWB file as output. Pipelines are organized in a Nextflow pipeline with individual Code Ocean capsules performing internal steps. Platforms should use established pipelines for processing. |
There was a problem hiding this comment.
| Pipelines are a standardized series of processing steps that take raw data from a single modality and typically produce an NWB file as output. Pipelines are organized in a Nextflow pipeline with individual Code Ocean capsules performing internal steps. Platforms should use established pipelines for processing. | |
| Pipelines are a standardized series of processing steps that take raw data from a single modality and typically produce an NWB file as output. Pipelines are organized in a Nextflow pipeline with individual processes implemented as separate GitHub repositories performing internal steps. Platforms should use established pipelines for processing. |
| aind_processing_pipelines.md | ||
| pipeline_and_process_testing.md | ||
| pipeline_parameters.md | ||
| pipeline_branch_management.md |
There was a problem hiding this comment.
| aind_processing_pipelines.md | |
| pipeline_and_process_testing.md | |
| pipeline_parameters.md | |
| pipeline_branch_management.md | |
| pipeline_parameters.md | |
| pipeline_testing.md | |
| aind_processing_pipelines.md |
Just reordering in my personal favorite order of importance
|
@AllenNeuralDynamics/pipeline-devs I did a first round of edits, mainly based on recent structural changes from the ephys pipeline, where I automated most of the steps. Left some TODO in the docs for discussion and potential centralization @dougollerenshaw tagging you here too since we discussed about this and we'll need to align on metadata too! |
|
We need to think about the case where there is a different container per process. How can container changes made in the remote repository of a process get synced back to the pipeline? |
| Step repositories SHOULD keep a `code/params.json` with defaults, so the step can also run on its own. | ||
|
|
||
| ```python | ||
| parser.add_argument("--params", default=None, help="JSON string or path to a JSON file") |
There was a problem hiding this comment.
alternative in smarstspim pipeline:
nextflow distributes full params.json to processes, whcih are in charge of reading and pull-out the step-specific section
There was a problem hiding this comment.
@camilolaiton implemented here: AllenNeuralDynamics/aind-ephys-pipeline#146
Can you take a look and validate this approach?
Then each process looks for a capsule/params.json as source for parameters
| acquisition_name = LOGGING.get("acquisition_name") | ||
| if acquisition_name is None: | ||
| data_description_json = next(data_folder.glob("**/data_description.json"), None) | ||
| if data_description_json is not None: | ||
| acquisition_name = json.loads(data_description_json.read_text())["name"] |
There was a problem hiding this comment.
it should be the nextflow script to set the acquisition name for the logging section
There was a problem hiding this comment.
done for ephys here: AllenNeuralDynamics/aind-ephys-pipeline#146
acquisition_name and pipeline_name are aded to the logging section of the params by the main.nf script
| |---|---| | ||
| | `Dockerfile_<image>` | One Dockerfile per container image (e.g. a base image, plus images for steps with special dependencies, like GPU spike sorters). | | ||
| | `images.json` | Maps an image key to its Dockerfile and its GHCR image name. CI builds, promotes and tags every image listed here. | | ||
| | `package-versions.txt` | Pinned (`==`) versions of the Python packages installed in the images. The Dockerfiles install from it, CI installs from it to run checks against the same versions, and release notes are checked against it. | |
There was a problem hiding this comment.
how do people feel about using uv instead of pip? I think the software best practices documenation recommends using uv.
I think this would mean we could maybe have a pyproject.toml and then the uv lockfile which would contain the pinned versions being installed. The Dockerfile would have to be updated to use uv to install instead of pip. Personally I find uv to be faster and easier to use, but just wondering what other people think
There was a problem hiding this comment.
Good idea, but this is up to the individual Dockerfiles. This file cannot be a uv.lock because it could happen that not all package versions are not satisfiable in the same environment (see AllenNeuralDynamics/aind-ephys-pipeline#57)
I think we could add something about using uv as a recommended package manager in the Dockerfiles.
Modified
platform_support.mdto edit "Platform requirements", add a "Data flow pipeline" blurb, and edit "Logging". Left everything else the same but we may want to edit some of the processing pipeline documentation to reference pipeline standards documentation.processing_pipeline_standardsis a new heading under "Policies and practices". It contains a sub-contents for:platform_support.md)I left a couple headings within "Policies and practices" in case they do not deserve a full page
Closes #225
📚 Documentation preview 📚: https://scicomp-docs--238.org.readthedocs.build/en/238/