purl-next: Collect PURLs - 2 - package collector for deployment #657

Activity

1. pombredanne

   added this to 00-AboutCodePlanneron Jul 24, 2025
2. pombredanne

   added a parent issue on Jul 24, 2025

   • purl-next: Create decentralized, distributed PURL metadata collection #655
3. pombredanne

   assigned

   JonoYang

   on Jul 24, 2025
4. pombredanne

   moved this to Needs prep in 00-AboutCodePlanneron Jul 24, 2025
5. pombredanne

   changed the title [-]purl-next: Collect PURL metadata for one ecosystem - 2[/-] [+]purl-next: Collect PURL metadata - 2 - collect for one small ecosystem[/+] on Aug 5, 2025
6. pombredanne

   changed the title [-]purl-next: Collect PURL metadata - 2 - collect for one small ecosystem[/-] [+]purl-next: Collect PURL metadata - 2 - [/+] on Aug 5, 2025
7. pombredanne

   changed the title [-]purl-next: Collect PURL metadata - 2 - [/-] [+]purl-next: Collect PURL metadata - 2 - package collector[/+] on Aug 5, 2025
8. pombredanne

   changed the title [-]purl-next: Collect PURL metadata - 2 - package collector[/-] [+]purl-next: Collect PURL metadata - 2 - package collector for deployment[/+] on Aug 5, 2025
9. AyanSinhaMahapatra

   moved this from Needs prep to In prep in 00-AboutCodePlanneron Aug 21, 2025
10. AyanSinhaMahapatra

    moved this from In prep to In progress in 00-AboutCodePlanneron Sep 1, 2025
11. pombredanne

    changed the title [-]purl-next: Collect PURL metadata - 2 - package collector for deployment[/-] [+]purl-next: Collect PURLs - 2 - package collector for deployment[/+] on Sep 3, 2025
12. pombredanne

    added the

    Feature

    issue type on Sep 5, 2025
13. AyanSinhaMahapatra

    self-assigned this

    on Sep 11, 2025
14. AyanSinhaMahapatra

    added a commit that references this issue on Sep 16, 2025

    Add packageURL mining in CI with scancode-action

    ...

    77ef307
15. AyanSinhaMahapatra commented on Sep 17, 2025

    

    AyanSinhaMahapatra

    on Sep 17, 2025

    Last edited by AyanSinhaMahapatra

    Member

    More actions

    We are now mining packageURLs with checkpointing functionality which saves/loads from progress in a git repo.

    Pull requests to add this functionality:

    • Add support to mine pypi Package-URLs #662 #677
    • Rename minecode_pipeline to minecode_pipelines #682
    • Rename minecode_pipeline to minecode_pipelines #682
    • Collect purl metadata scancode.io#1868
    • Update federatedcode pipes scancode.io#1831 from @JonoYang
    • Move common functions to minecode_pipelines.pipes #660 #696 from @JonoYang
    • Add support to mine maven Package-URLs #660 #678 from @JonoYang
    • Add support to mine cargo Package-URLs #679 from @ziadhany

    Other types of index mining being implemented:

    • maven like repository Add support to mine maven Package-URLs #660 #678
    • github repo with data Add support to mine cargo Package-URLs #679

    This is now released at: https://pypi.org/project/minecode-pipelines/0.0.1b2/

    We also have this running in the CI using scancode-action:

    • github action file: https://github.com/aboutcode-data/minecode-pipelines-config/blob/main/.github/workflows/mine-pypi-packageurls.yml
    • github action run: https://github.com/aboutcode-data/minecode-pipelines-config/actions/runs/17778018391

    repos with data from pypi packageURLs mined:

    • pypi packageURL data: https://github.com/aboutcode-data/minecode-data-pypi-test
    • pypi mining checkpoints: https://github.com/aboutcode-data/minecode-pipelines-config/tree/main/pypi

    All github actions:

    • https://github.com/aboutcode-data/minecode-pipelines-config/tree/main/.github/workflows
16. AyanSinhaMahapatra

    moved this from In progress to Review ready in 00-AboutCodePlanneron Sep 17, 2025
17. pombredanne commented on Sep 18, 2025

    

    pombredanne

    on Sep 18, 2025

    MemberAuthor

    More actions

    Thanks. Closing.
18. pombredanne

    closed this as completedon Sep 18, 2025
19. github-project-automation

    moved this from Review ready to Done in 00-AboutCodePlanneron Sep 18, 2025
20. pombredanne

    moved this from Done to In validation in 00-AboutCodePlanneron Sep 18, 2025
21. pombredanne

    moved this from In validation to Validated in 00-AboutCodePlanneron Oct 6, 2025