Recent NuGet PURLs missing from public.purldb.io despite existing in federated data; NuGet mining jobs repeatedly time out

[marcoberger]

Summary

I am seeing a significant gap in recent NuGet/.NET package data in the public PurlDB instance (public.purldb.io).

Initially, I suspected that recent .NET/NuGet packages were simply no longer being collected. However, after looking at the current AboutCode mining infrastructure, the situation appears to be more specific:

1. Recent NuGet PURLs, including current .NET 10 packages, are already present in the federated aboutcode-data/purls-nuget-* repositories.
2. These packages do not appear to be available as expected in the public PurlDB instance.
3. At the same time, the scheduled NuGet mining workflow repeatedly reaches GitHub Actions' six-hour execution limit and is cancelled.
4. On-demand population of an individual recent NuGet package into PurlDB works successfully.

This suggests that there may be two related issues:

• the NuGet mining process may currently produce an incomplete federated data set because runs do not finish;
• the federated NuGet PURLs that have already been mined may not be imported/defederated into public.purldb.io regularly or completely.

Examples

A useful example is:

pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@10.0.11

NuGet.org currently contains this package/version. It was published in August 2026.

The PURL is also already present in the federated AboutCode data:

Repository:

aboutcode-data/purls-nuget-0000

Path:

nuget-0024/Microsoft.NETCore.App.Runtime.linux-x64/purls.yml

The same file currently contains, among others:

pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@10.0.3
pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@10.0.10
pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@10.0.11

Older versions such as the following are present there as well:

pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@3.1.32
pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@6.0.36
pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@8.0.19
pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@9.0.8

I see the same pattern for another Microsoft package family:

pkg:nuget/Microsoft.AspNetCore.App.Runtime.linux-x64

The corresponding federated PURL file in:

aboutcode-data/purls-nuget-0512

contains recent versions including:

pkg:nuget/Microsoft.AspNetCore.App.Runtime.linux-x64@8.0.19
pkg:nuget/Microsoft.AspNetCore.App.Runtime.linux-x64@10.0.3
pkg:nuget/Microsoft.AspNetCore.App.Runtime.linux-x64@10.0.11

Therefore, recent .NET PURLs are definitely reaching the federated AboutCode data repositories.

However, recent package/version records from this range do not appear to be available as expected in the public PurlDB instance.

A query that can be used to check an individual PURL on the public instance is for example:

GET /api/packages/?purl=pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@10.0.11

On-demand population works

As an additional test, I scanned a very recent NuGet package as a single package in ScanCode.io and then used the populate PURL DB functionality.

The package was:

pkg:nuget/1-000-free-robux-codes-v6042@1.1.8660

After the population step, the package was successfully created in public.purldb.io.

The public API now returns this package successfully.

This seems to demonstrate that:

• current NuGet packages can be downloaded and processed by PurlDB;
• NuGet PURLs themselves are supported;
• the issue is probably not a general incompatibility with recent .NET/NuGet packages;
• on-demand collection and the regular bulk/federated population path behave differently.

This also seems consistent with the PurlDB release history: NuGet on-demand package collection was added in PurlDB v7.0.0, while the NuGet PURL mining pipeline and decentralized PURL data were added as part of the newer PurlDB/Purl Next architecture around v7.1.0.

NuGet mining workflow repeatedly reaches the 6-hour limit

The scheduled NuGet mining workflow is located in:

aboutcode-data/minecode-pipelines-config/.github/workflows/mine-nuget-packageurls.yml

It currently runs:

schedule:
  - cron: '0 */6 * * *'

and invokes:

pipelines: "mine_nuget"

with the federated data destination:

FEDERATEDCODE_GIT_ACCOUNT_URL: https://github.com/aboutcode-data

Recent workflow executions appear to repeatedly run for approximately six hours and then get cancelled.

For example, workflow run #1093, triggered on August 26, 2026, ended with:

Status: Cancelled
Total duration: 6h 0m 19s

The job has exceeded the maximum execution time of 6h0m0s
The operation was canceled.

Several surrounding scheduled runs show essentially the same six-hour duration.

Possible mining issue

The NuGet mining implementation introduced in aboutcode-org/purldb#721 appears to:

1. clone the AboutCode NuGet catalog mirror;
2. initialize a new in-memory package/version mapping;
3. iterate over all catalog page JSON files;
4. collect all package/version combinations;
5. publish the resulting PURLs to the federated Git repositories in batches.

The implementation contains logic equivalent to:

package_versions = {}

for page in catalog_pages.rglob("*.json"):
    ...

I do not see a NuGet-specific incremental checkpoint/cursor in that implementation.

Since PURLs are committed in batches during the publishing step, a job that is killed after six hours can still leave partial updates in the purls-nuget-* repositories.

This may also explain another observation: the federated data is very recent, but it is not fully current.

For example, NuGet.org currently contains:

pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@11.0.0-preview.7.26381.103

while the corresponding federated PURL file currently stops at:

pkg:nuget/Microsoft.NETCore.App.Runtime.linux-x64@11.0.0-preview.5.26302.115

So the miner appears to be processing recent catalog data, but there are also recent gaps.

I realize that the currently deployed mine_nuget implementation may have changed since #721, so it would be useful to confirm whether the production mining workflow still processes the complete catalog on each execution or whether there is now another checkpoint mechanism.

Federated data -> public PurlDB

PurlDB PR aboutcode-org/purldb#790 implemented the decentralized/federated PURL data import.

The implementation appears to consume the aboutcode-data PURL repositories and create PurlDB Package records from the federated PURLs.

This raises the main question for the public instance:

Is public.purldb.io currently importing/defederating the aboutcode-data/purls-nuget-* repositories?

I could not find a public scheduled production job showing when or how this import is executed for public.purldb.io.

Given that recent .NET 10 PURLs are already present in the federated repositories, but do not appear to be available in the public PurlDB as expected, this may be the point where the data becomes stale.

Questions

It might help to work along those questions while digging into the details of that issue. At least it helped me to write them down alongside the research on that issue.

1. Is public.purldb.io expected to regularly import the PURLs from aboutcode-data/purls-nuget-*?

2. If yes, what process currently performs this import/defederation and how frequently does it run?

3. Is that import currently running successfully for NuGet?

4. Is there an expected delay between a NuGet PURL appearing in aboutcode-data/purls-nuget-* and the corresponding Package appearing in public.purldb.io?

5. Are the repeated six-hour cancellations of the mine_nuget workflow known/expected?

6. Does the currently deployed NuGet mining pipeline have an incremental checkpoint, or does every scheduled run process the complete NuGet catalog again?

7. Can an interrupted mining run leave only a partial subset of NuGet PURLs updated until a later run happens to process the remaining packages?

8. Is there currently a planned backfill or synchronization of the federated NuGet data into the public PurlDB?

9. Which PurlDB version/deployment architecture is currently used by public.purldb.io? Could the missing NuGet data be related to the transition to the decentralized PURL data architecture introduced around PurlDB 7.1?

Expected behavior

Ideally, a NuGet PURL that:

• exists on NuGet.org, and
• has already been mined into an aboutcode-data/purls-nuget-* repository

should eventually appear as a corresponding Package in public.purldb.io within a predictable synchronization interval.

The scheduled NuGet miner should also eventually process the complete catalog without leaving persistent gaps caused by the six-hour workflow timeout.

Thank you for any insight into how the NuGet mining and public PurlDB synchronization are currently intended to work.