ioda-data cannot be fetched

[rickgrubin-noaa]

Description

Package ioda-data fails to fetch as a dependency of package ioda / dependency of gdas variant for global-workflow-env

Error:

==> No binary for ioda-data-2.9.0.20260319-mazerpgaxp6zqmzkclphcv7szzkkpofi found: installing from source
Initialized empty Git repository in /tmp/tmpg_la0k80/ioda-data/.git/
==> Error: FetchError: All fetchers failed for spack-stage-ioda-data-2.9.0.20260319-mazerpgaxp6zqmzkclphcv7szzkkpofi
    file:///apps/contrib/spack-stack/source-cache/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz: URLError: <urlopen error [Errno 2] No such file or directory: '/apps/contrib/spack-stack/source-cache/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz'>
  file:///apps/contrib/spack-stack/build-cache/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz: URLError: <urlopen error [Errno 2] No such file or directory: '/apps/contrib/spack-stack/build-cache/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz'>
  https://mirror.spack.io/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz: DetailedHTTPError: GET https://mirror.spack.io/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz returned 404: Not Found
  [git] https://github.com/JCSDA-internal/ioda-data.git at commit 2886c75398b4b3bdd3a4235298af188370727c2e: ProcessError: Command exited with status 128:
  /usr/bin/git -c advice.detachedHead=false -C ioda-data checkout --quiet 2886c75398b4b3bdd3a4235298af188370727c2e
==> Warning: Skipping build of ioda-2.9.0.20260326-tatvbkavhpe4ibs6dr64br6r7ijjt5be since ioda-data-2.9.0.20260319-mazerpgaxp6zqmzkclphcv7szzkkpofi failed

To Reproduce

Create / activate / concretize / install the unified-dev env

Expected behavior

Package ioda-data is successfully fetched

System, compiler, code, ...

NOAA EPIC RDHPDS hosts

• hercules
• orion

irrespective of compiler / version.

Additional context

None

[eap]

@rickgrubin-noaa It looks like your machine's source cache (/apps/contrib/spack-stack/source-cache) doesn't have a tarball for 2886c75 that spack-stack started pinning in July. Spack should have solved this fetching from GitHub but for unclear reasons that isn't working on these platforms.

The following command passes on my machine even without credentials, so the reference is definitely correct and available.

git clone --depth 1  https://github.com/JCSDA-internal/ioda-data.git ioda-data \
  && git -C ioda-data fetch --depth 1 origin 2886c75398b4b3bdd3a4235298af188370727c2e \
  && git -C ioda-data checkout 2886c75398b4b3bdd3a4235298af188370727c2e

I don't know why its failing but there's also a fallback that's failing and we might be able to fix it by changing these lines in the spack-stack ioda-data repo to the following adding get_rull_repo.

  version("2.9.0.20260319", commit="2886c75398b4b3bdd3a4235298af188370727c2e", get_full_repo=True)
  version("2.9.0.20250805", commit="c6f8842648ea473eebc9f66d7c27e2204e5220d6", get_full_repo=True)

Can you try that? If it works on your side, we can update the package.

[rickgrubin-noaa]

@rickgrubin-noaa It looks like your machine's source cache (/apps/contrib/spack-stack/source-cache) doesn't have a tarball for 2886c75 that spack-stack started pinning in July. Spack should have solved this fetching from GitHub but for unclear reasons that isn't working on these platforms.

Indeed the source cache does not have a tarball for ioda-data (from a requirement for ioda) because the global-workflow-env package recipe didn't have a dependency on ioda until ~2 weeks ago.

The following command passes on my machine even without credentials, so the reference is definitely correct and available.

git clone --depth 1  https://github.com/JCSDA-internal/ioda-data.git ioda-data \
  && git -C ioda-data fetch --depth 1 origin 2886c75398b4b3bdd3a4235298af188370727c2e \
  && git -C ioda-data checkout 2886c75398b4b3bdd3a4235298af188370727c2e

This is how I created a repo from which I could then create a valid tarball and place it in a repo's source cache to tests its veracity.

I don't know why its failing but there's also a fallback that's failing and we might be able to fix it by changing these lines in the spack-stack ioda-data repo to the following adding get_rull_repo.

  version("2.9.0.20260319", commit="2886c75398b4b3bdd3a4235298af188370727c2e", get_full_repo=True)
  version("2.9.0.20250805", commit="c6f8842648ea473eebc9f66d7c27e2204e5220d6", get_full_repo=True)

Can you try that? If it works on your side, we can update the package.

$ git diff repos/spack_stack/spack_repo/spack_stack/packages/ioda_data/package.py
diff --git a/repos/spack_stack/spack_repo/spack_stack/packages/ioda_data/package.py b/repos/spack_stack/spack_repo/spack_stack/packages/ioda_data/package.py
index 170c7c5..a82dbdc 100644
--- a/repos/spack_stack/spack_repo/spack_stack/packages/ioda_data/package.py
+++ b/repos/spack_stack/spack_repo/spack_stack/packages/ioda_data/package.py
@@ -17,7 +17,8 @@ class IodaData(CMakePackage):
     maintainers("climbfuji")
 
     version("develop", branch="develop", no_cache=True)
-    version("2.9.0.20260319", commit="2886c75398b4b3bdd3a4235298af188370727c2e")
+    version("2.9.0.20260319", commit="2886c75398b4b3bdd3a4235298af188370727c2e", get_full_repo=True)
     version("2.9.0.20250805", commit="c6f8842648ea473eebc9f66d7c27e2204e5220d6")
 
     generator("make")

fails with the same error noted in the issue description.

It may be that EPIC will have to create a <commit>.tar.gz file for ioda-data and manually insert it into definitive source caches.

[eap]

Is there some sort of URL whitelist or other filtering that could be at play? ioda-data uses git-lfs which fetches the data from secondary urls (also hosted by github, but an aggressive whitelist might not catch them.

Is there a way to get verbose output on this failure so we can see exactly when the failure happens?

spack -d install --verbose ioda-data

[rickgrubin-noaa]

Is there some sort of URL whitelist or other filtering that could be at play? ioda-data uses git-lfs which fetches the data from secondary urls (also hosted by github, but an aggressive whitelist might not catch them.

Is there a way to get verbose output on this failure so we can see exactly when the failure happens?

spack -d install --verbose ioda-data

No URL whitelist of which I'm aware, but the comment about needing git-lfs likely points out the problem on these hosts: for git-lfs to be available, a module must be loaded, and part of the build process is to first unload all modules.

Running a test now to see if this is indeed what's going on. Thanks for the memory kick!

[climbfuji]

Is there some sort of URL whitelist or other filtering that could be at play? ioda-data uses git-lfs which fetches the data from secondary urls (also hosted by github, but an aggressive whitelist might not catch them.
Is there a way to get verbose output on this failure so we can see exactly when the failure happens?

spack -d install --verbose ioda-data

No URL whitelist of which I'm aware, but the comment about needing git-lfs likely points out the problem on these hosts: for git-lfs to be available, a module must be loaded, and part of the build process is to first unload all modules.

Running a test now to see if this is indeed what's going on. Thanks for the memory kick!

I am pretty sure that's it, and I don't know why I didn't remember. I had to add git and git-lfs into the path on some systems.

[mathomp4]

Well crap. Because I removed the ioda-data@2.12.0 updates from my branch I now get on discover:

Initialized empty Git repository in /gpfsm/dnb34/tdirs/login/discover33.161367.mathomp4/tmpherod7j2/ioda/.git/
==> Adding package ioda-data@2.9.0.20260319 to mirror
Initialized empty Git repository in /gpfsm/dnb34/tdirs/login/discover33.161367.mathomp4/tmp54qxvetq/ioda-data/.git/
Initialized empty Git repository in /gpfsm/dnb34/tdirs/login/discover33.161367.mathomp4/tmpsos4hw9t/ioda-data/.git/
Initialized empty Git repository in /gpfsm/dnb34/tdirs/login/discover33.161367.mathomp4/tmpn_o5r22h/ioda-data/.git/
==> Warning: Error while fetching ioda-data@2.9.0.20260319
  All fetchers failed for spack-stage-ioda-data-2.9.0.20260319-f53cv4egfe33rlm4zom7mya5gcuiplnn
        file:///discover/nobackup/projects/gmao/SIteam/spack-stack/source-cache/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz: URLError: <urlopen error [Errno 2] No such file or directory: '/discover/nobackup/projects/gmao/SIteam/spack-stack/source-cache/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz'>
    file:///discover/nobackup/projects/gmao/SIteam/spack-stack/build-cache/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz: URLError: <urlopen error [Errno 2] No such file or directory: '/discover/nobackup/projects/gmao/SIteam/spack-stack/build-cache/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz'>
    https://mirror.spack.io/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz: DetailedHTTPError: GET https://mirror.spack.io/_source-cache/git/JCSDA-internal/ioda-data.git/2886c75398b4b3bdd3a4235298af188370727c2e.tar.gz returned 404: Not Found
    [git] https://github.com/JCSDA-internal/ioda-data.git at commit 2886c75398b4b3bdd3a4235298af188370727c2e: ProcessError: Command exited with status 128:
    /usr/bin/git -c advice.detachedHead=false -C ioda-data checkout --quiet 2886c75398b4b3bdd3a4235298af188370727c2e

@rickgrubin-noaa Did anything you do get this working? I'm going to try to get git-lfs into my scripts PATH to see if that helps.

[mathomp4]

Ahh. Yep. git-lfs was it. When I was trying the clone by hand, it was in my PATH, but my batch_install script didn't have it.

I wonder if we should add comment(s) in the ioda and/or ioda-data packages about this?

[climbfuji]

Ahh. Yep. git-lfs was it. When I was trying the clone by hand, it was in my PATH, but my batch_install script didn't have it.

I wonder if we should add comment(s) in the ioda and/or ioda-data packages about this?

I hope ioda-data, ufo-data, and all other JCSDA -data repos do have a dependency on git-lfs in the package. If not, I will add that separately. That of course won't solve the fetch problem, but at least it's some sort of indicator. I agree that it would be nice to add a note in the package docstring/description for all these packages. And, even better, have a way to obtain this data WITHOUT git-lfs in the future.

[climbfuji]

Ahh. Yep. git-lfs was it. When I was trying the clone by hand, it was in my PATH, but my batch_install script didn't have it.
I wonder if we should add comment(s) in the ioda and/or ioda-data packages about this?

I hope ioda-data, ufo-data, and all other JCSDA -data repos do have a dependency on git-lfs in the package. If not, I will add that separately. That of course won't solve the fetch problem, but at least it's some sort of indicator. I agree that it would be nice to add a note in the package docstring/description for all these packages. And, even better, have a way to obtain this data WITHOUT git-lfs in the future.

But again, no changes needed in your PR. I will take care of this separately when I deal with the Python version question.

[rickgrubin-noaa]

Is there some sort of URL whitelist or other filtering that could be at play? ioda-data uses git-lfs which fetches the data from secondary urls (also hosted by github, but an aggressive whitelist might not catch them.
Is there a way to get verbose output on this failure so we can see exactly when the failure happens?

spack -d install --verbose ioda-data

No URL whitelist of which I'm aware, but the comment about needing git-lfs likely points out the problem on these hosts: for git-lfs to be available, a module must be loaded, and part of the build process is to first unload all modules.
Running a test now to see if this is indeed what's going on. Thanks for the memory kick!

I am pretty sure that's it, and I don't know why I didn't remember. I had to add git and git-lfs into the path on some systems.

==> [2026-09-01-19:55:11.390069] Cloning git repository: https://github.com/JCSDA-internal/ioda-data.git at commit 2886c75398b4b3bdd3a4235298af188370727c2e
==> [2026-09-01-19:55:11.403213] /usr/bin/git --version
==> [2026-09-01-19:55:11.406843] /usr/bin/git -c advice.detachedHead=false clone --no-single-branch --filter=blob:none https://github.com/JCSDA-internal/ioda-data.git ioda-data
Cloning into 'ioda-data'...
git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly
warning: Clone succeeded, but checkout failed.

Loading the git-lfs module prior to env install solves this issue.

Of note re: some prior comments: git-lfs is spec'd in <env>/site/packages.yaml and this seems irrelevant.

Will close this issue.