Repository navigation
purl-next: Collect PURLs - 1 - design base framework #656
Description
To create a decentralized, distributed PURL metadata collection using CI/CD jobs we first need to design, create and setup the base framework.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- Status
Activity
[-]purl-next: Collect PURL metadata for one ecosystem - 1[/-][+]purl-next: Collect PURL metadata - 1 - setup base framework [/+][-]purl-next: Collect PURL metadata - 1 - setup base framework [/-][+]purl-next: Collect PURL metadata - 1 - design base framework[/+]JonoYang commentedon Aug 13, 2025
We want to create a CI time job that:
This process is essentially done in the publish to federatedcode pipeline (https://github.com/aboutcode-org/scancode.io/blob/main/scanpipe/pipelines/publish_to_federatedcode.py)
What we want to do is to be able to run a modified version of this pipeline in a CI job, where we run the pipeline using the scanpipe cli (https://scancodeio.readthedocs.io/en/latest/command-line-interface.html#scanpipe-create-project-name)
We would need to keep track of what we indexed such that we do not revisit data we already have and only collect new/modified packages.
Initially, we should just collect the index of the repo and get the package names and versions. We then will have workers in another job go and fetch the metadata for the given package name and versions.
The workers should run forever and expect to be stopped at any moment. We should save state and metadata on a periodic basis as we run. When a worker is interrupted and restarted, it should read the last committed state and start from there.
[-]purl-next: Collect PURL metadata - 1 - design base framework[/-][+]purl-next: Collect PURLs - 1 - design base framework[/+]