purl-next: Collect PURLs - 3 - collect pilot ecosystem #658

Activity

1. pombredanne

   added this to 00-AboutCodePlanneron Jul 24, 2025
2. pombredanne

   added a parent issue on Jul 24, 2025

   • purl-next: Create decentralized, distributed PURL metadata collection #655
3. pombredanne

   assigned

   JonoYang

   on Jul 24, 2025
4. pombredanne

   moved this to Needs prep in 00-AboutCodePlanneron Jul 24, 2025
5. pombredanne

   changed the title [-]purl-next: Collect PURL metadata for one ecosystem - 3[/-] [+]purl-next: Collect PURL metadata - 3 collect for pilot ecosystem[/+] on Aug 5, 2025
6. pombredanne

   changed the title [-]purl-next: Collect PURL metadata - 3 collect for pilot ecosystem[/-] [+]purl-next: Collect PURL metadata - 3 - collect pilot ecosystem[/+] on Aug 5, 2025
7. AyanSinhaMahapatra

   self-assigned this

   on Aug 21, 2025
8. AyanSinhaMahapatra

   moved this from Needs prep to In prep in 00-AboutCodePlanneron Aug 21, 2025
9. AyanSinhaMahapatra

   moved this from In prep to In progress in 00-AboutCodePlanneron Sep 1, 2025
10. pombredanne

    changed the title [-]purl-next: Collect PURL metadata - 3 - collect pilot ecosystem[/-] [+]purl-next: Collect PURLs - 3 - collect pilot ecosystem[/+] on Sep 3, 2025
11. pombredanne

    added the

    Feature

    issue type on Sep 5, 2025
12. AyanSinhaMahapatra commented on Sep 17, 2025

    

    AyanSinhaMahapatra

    on Sep 17, 2025

    Member

    More actions

    We are now mining packageURLs with checkpointing functionality which saves/loads from progress in a git repo.

    We started with these ecosystems as a pilot:

    • pypi: purl-next: Collect PURLs for PyPI ecosystem #662
    • maven: purl-next: Collect PURLs for Maven ecosystem #660
    • cargo: purl-next: Collect PURLs for Rust ecosystem #666

    Pull requests to add this functionality:

    • Add support to mine pypi Package-URLs #662 #677
    • Rename minecode_pipeline to minecode_pipelines #682
    • Rename minecode_pipeline to minecode_pipelines #682
    • Collect purl metadata scancode.io#1868
    • Update federatedcode pipes scancode.io#1831 from @JonoYang

    Other types of index mining being implemented:

    • maven like repository Add support to mine maven Package-URLs #660 #678
    • github repo with data Add support to mine cargo Package-URLs #679

    This is now released at: https://pypi.org/project/minecode-pipelines/0.0.1b2/

    We also have this running in the CI using scancode-action:

    • github action file: https://github.com/aboutcode-data/minecode-pipelines-config/blob/main/.github/workflows/mine-pypi-packageurls.yml
    • github action run: https://github.com/aboutcode-data/minecode-pipelines-config/actions/runs/17778018391

    repos with data from pypi packageURLs mined:

    • pypi packageURL data: https://github.com/aboutcode-data/minecode-data-pypi-test
    • pypi mining checkpoints: https://github.com/aboutcode-data/minecode-pipelines-config/tree/main/pypi
13. AyanSinhaMahapatra

    moved this from In progress to Review ready in 00-AboutCodePlanneron Sep 17, 2025
14. pombredanne commented on Sep 18, 2025

    

    pombredanne

    on Sep 18, 2025

    MemberAuthor

    More actions

    Thanks!
15. pombredanne

    closed this as completedon Sep 18, 2025
16. github-project-automation

    moved this from Review ready to Done in 00-AboutCodePlanneron Sep 18, 2025
17. pombredanne

    moved this from Done to In validation in 00-AboutCodePlanneron Sep 18, 2025
18. pombredanne

    moved this from In validation to Validated in 00-AboutCodePlanneron Oct 6, 2025