Skip to content

fix bug in common/io.py when retrieving the fstep in ZarrIO - #2771

Open
iluise wants to merge 6 commits into
developfrom
iluise/fix-forecast-steps-stream-lookup
Open

fix bug in common/io.py when retrieving the fstep in ZarrIO #2771
iluise wants to merge 6 commits into
developfrom
iluise/fix-forecast-steps-stream-lookup

Conversation

@iluise

@iluise iluise commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Description

There is a bug in common/io.py when retrieving the fstep info from ZarrIO.
the code looks for the 0/ERA5_in/0 group even if this stream was used just in input.
Changed it to loop over the streams and actually open only the first available stream in output.

Issue Number

Closes #2770

Is this PR a draft? Mark it as draft.

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

FastEvaluation

  • I have updated the public documentation if necessary

@github-actions github-actions Bot added bug Something isn't working infra Issues related to infrastructure labels Aug 21, 2026
@clessig
clessig requested a review from grassesi August 21, 2026 10:20
@clessig

clessig commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

@grassesi : could you have a look?

@grassesi

Copy link
Copy Markdown
Contributor

I have not verified yet, but i am a bit puzzled by this PR: Usually validation io should filter for input only streams.. Looking at the stacktrace linked in the original Issue, the failing line is the forecast_offset property lookup which this PR does not address at all. For me this looks more like a misconfiguration during inference or a deeper issue in how the zarr store is written. Can you provide me with the resolved config of the inference run and the generated zarr hierarchy? (Or the run id so I can look it up) Was this inference run generated on develop?

@grassesi grassesi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above

@enssow

enssow commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

I just tested and ran into the same error again.
Command: uv run python packages/evaluate/src/weathergen/evaluate/export/export_inference.py \ --run-id "cw6a" \ --stream "ERA5" \ --type prediction \ --format netcdf \ --regrid-type regular_ll \ --regrid-degree 1 \ --output-dir "results/cw6a/" \ --fstep-hours "1" \ --n-processes "16" \ --rank "3,4,5" \ --epoch "0"
and then output: AssertionError: Zarr group: 1/ERA5_in/0 does not exist.

I'll have a dig and see if maybe changing the forecast_offset lookup section fixes it :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working infra Issues related to infrastructure

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Bug in common/io.py for runs with forcing

4 participants