download_huggingface_dataset sends no token for public+gated repos (private is the wrong predicate)

[vahid-ahmadi]

Summary

download_huggingface_dataset decides whether to authenticate by testing ModelInfo.private. For a repo that is public but gated, private is False, so no token is sent and the gate returns 401 GatedRepoError — even when a valid, gate-approved token is available in the environment.

policyengine_core/tools/hugging_face.py:81-90:

fetched_model_info: ModelInfo = model_info(repo)
is_repo_private: bool = fetched_model_info.private

authentication_token: str = None
if is_repo_private:
    authentication_token: str = get_or_prompt_hf_token()

return hf_hub_download(
    repo_id=repo,
    ...
    token=authentication_token,   # None for public+gated
)

Reproduction

policyengine/policyengine-uk-data-private was flipped from private to public + gated (manual approval) on 31 July 2026:

>>> from huggingface_hub import model_info
>>> i = model_info("policyengine/policyengine-uk-data-private")
>>> i.private, i.gated
(False, 'manual')

Any download_huggingface_dataset call against it now fails:

huggingface_hub.errors.GatedRepoError: 401 Client Error.
Cannot access gated repo for url https://huggingface.co/policyengine/policyengine-uk-data-private/resolve/1.40.3/enhanced_frs_2023_24.h5.
Access to model policyengine/policyengine-uk-data-private is restricted. You must have access to it and be authenticated to access it.

Impact

This took down every dataset-backed CI job in policyengine-uk from 31 July to 11 August (PolicyEngine/policyengine-uk#1816). The failure mode is unusually expensive to diagnose because it is indistinguishable from a bad credential: the error says "you must be authenticated", so the natural response is to rotate the token. That was done on 10 August and changed nothing, because no token was being sent. Validating the replacement token out of band also passed — that check supplied the token explicitly, which is the step this function skips.

Any country package that downloads from a gated HF repo has the same latent bug, and gated-instead-of-private is the posture HF actually recommends for licensed data, since access grants are no-ops on private repos.

Suggested fix

The private test made sense when it was added (#320) — the only restricted repos were private ones. It no longer partitions the space. Two options:

1. Pass the token whenever one is available. Simplest and hard to get wrong; hf_hub_download ignores a token it does not need for a genuinely public repo.
2. Test gated alongside private: if fetched_model_info.private or fetched_model_info.gated:. Note gated is False | 'auto' | 'manual', so truthiness works but the tri-state deserves a comment.

Worth noting that dropping the explicit token entirely would also work, since huggingface_hub falls back to the HF_TOKEN environment variable when token=None (get_token_to_send → get_token) — but only for HF_TOKEN, not PolicyEngine's HUGGING_FACE_TOKEN, so the fallback is silent and name-dependent. Better to pass it explicitly.

A regression test would need a gated fixture repo, or a mock asserting the token reaches hf_hub_download when model_info reports private=False, gated='manual'.

Workaround in the meantime

Export HF_TOKEN alongside HUGGING_FACE_TOKEN so huggingface_hub picks it up implicitly: PolicyEngine/policyengine-uk#1817.

[vahid-ahmadi]

Correction: option 1 as written would introduce an interactive prompt

Reviewing my own suggestion — option 1 ("pass the token whenever one is available") is a trap if implemented literally.

Calling get_or_prompt_hf_token() unconditionally would reach hugging_face.py:109-118, which calls getpass() when HUGGING_FACE_TOKEN is unset and os.isatty(0) is true. That would add an interactive credential prompt to every public, non-gated dataset download on a developer machine — a clear regression for the common case.

If option 1 is taken, it must read the environment without prompting (os.environ.get("HUGGING_FACE_TOKEN"), or huggingface_hub.get_token(), which also picks up HF_TOKEN and the token file).

Option 2 (private or gated) is the safer recommendation and I would now lead with it. Confirming the predicate is sound: hf_api.py:823 types the field Optional[Literal["auto", "manual", False]], defaulting to None (:853), so truthiness is correct in every state. 'auto' means auto-approval on accepting terms — the requester must still be authenticated — so a token is required there too. The one residual risk is that an omitted field yields None and is silently treated as ungated.

For completeness: the same function is reached from policyengine_core/simulations/simulation.py:186 and policyengine_core/data/dataset.py:548, and there is no second implementation with this bug. policyengine-uk-data is unaffected because it passes tokens unconditionally to hf_hub_download.