Publish MaxText to PyPI when Airflow E2E tests succeed

[SurbhiJainUSC]

Description

This PR refactors the MaxText release and CI workflows to automate the E2E release lifecycle. The goal is to connect the GitHub-based build process with the Airflow-based E2E test suite, ensuring that MaxText is only published to PyPI after successful verification.

Key Technical Changes

E2E Automation and Gated Publishing

• Workflow Integration: Updated release_pipeline.yml to trigger the maxtext_e2e_tests DAG via the run_e2e_tests.yml workflow after successful building of package and Docker images.
• Verification-Gated Release: Refactored pypi_release.yml to trigger via a repository_dispatch event (airflow-dag-complete). This ensures that PyPI publication and final image promotion only occur once the Airflow-based E2E suite reports success.
• Automatic Image Promotion: Added a tag_docker_image job to promote release-candidate images to PyPI version tag only after the full validation suite passes.

Infrastructure Fixes

• SHA Pinning: Introduced a get_maxtext_sha job at the start of the release pipeline to translate a release tag into a Git SHA. This SHA is then propagated to all downstream build and test jobs.

• Improved Build Logic: Updated build_package.yml and ci_pipeline.yml to support a maxtext_sha input, allowing the caller to define the exact code commit to build.

• Documentation Builds: Updated the documentation analysis logic to always force a build for non-PR events, ensuring docs are verified by the release pipeline.

Tests

• MaxText Release Pipeline: https://github.com/AI-Hypercomputer/maxtext/actions/runs/27991474145 successfully builds package, runs CI tests, builds docker image, and triggers E2E Airflow DAG.
• Next Step (after PR is merged): Manual run of pypi_release.yml to verify that the candidate docker images are tagged correctly.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!