Repository navigation
purl-next: Collect PURLs for PyPI ecosystem #662
Description
Create decentralized, distributed PURL metadata collection using CI/CD jobs - one ecosystem
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- Status
{{ message }}
Activity
[-]purl-next: Collect PURL metadata for one ecosystem - 7[/-][+]purl-next: Collect PURL metadata for ecosystems - 7[/+][-]purl-next: Collect PURL metadata for ecosystems - 7[/-][+]purl-next: Collect PURL metadata for PyPI ecosystems - 7[/+][-]purl-next: Collect PURL metadata for PyPI ecosystems - 7[/-][+]purl-next: Collect PURL metadata for PyPI ecosystems[/+]AyanSinhaMahapatra commentedon Aug 25, 2025
The pypi miner code doesn't work at https://github.com/aboutcode-org/purldb/blob/main/minecode/miners/pypi.py#L53, fails with the following error:
This is because the xmlrpc client and the API is deprecated, they have an API to get a list of all the packages and get package release info from https://pypi.org/simple/ (as mentioned in https://docs.pypi.org/api/index-api/) which supports JSON and HTML.
This is possibly cached for fast downloads and is refreshed atleast once a day.
See also https://peps.python.org/pep-0691/ where this was proposed and accepted. See also https://discuss.python.org/t/pep-691-json-based-simple-api-for-python-package-indexes/15553
The raw JSON file with all package names (and
_last-serial) is around 64 MB, and is quite fast/manageable to get all package names like this.There is also the library https://github.com/jwodder/pypi-simple with minimal dependencies that seems to do the same with a lot of extra added features for getting all available kinds of data from the index, which we can use as a dependency possibly or as a refinement.
The
X-PyPI-Last-Serial:and_last-serialfields seem nice if someone wants to optimize getting only the packages which had a release after the last sync quickly or quickly check for package updates for a specific version. But this is a refinement which doesn't seem to be required to get just the list of packages and their versions.Refactor and fix pypi miners
[-]purl-next: Collect PURL metadata for PyPI ecosystems[/-][+]purl-next: Collect PURLs for PyPI ecosystems[/+]Add support to mine pypi Package-URLs #662 (#677)
[-]purl-next: Collect PURLs for PyPI ecosystems[/-][+]purl-next: Collect PURLs for PyPI ecosystem[/+]Support mining pypi purls with checkpoints
Support mining pypi purls with checkpoints
Support mining pypi purls with checkpoints (#708)
AyanSinhaMahapatra commentedon Sep 17, 2025
We are now mining pypi packageURLs from the pypi simple index at https://pypi.org/simple/ with checkpointing functionality which saves/loads from progress in a git repo.
Pull requests to add this functionality:
This is now released at: https://pypi.org/project/minecode-pipelines/0.0.1b2/
We also have this running in the CI using scancode-action:
repos with data from pypi packageURLs mined:
pombredanne commentedon Sep 18, 2025
Thanks ... looking fine!