Add end-to-end XLML test scripts and documentation for Qwen3.8-2.4T-A95B

[JamesDeng42]

Description

Follow-up to #5366 to add end-to-end TPU validation scripts, user guide instructions, and documentation updates for Qwen3.8-2.4T-A95B

Changes

• End-to-end test scripts:
  • tests/end_to_end/tpu/qwen/moe/qwen3.8-2.4t-a95b/1_test_qwen3.8_2.4t_a95b.sh: CPU checkpoint conversion script from Hugging Face (bf16) to MaxText Orbax
    format (scanned for training and unscanned for decoding).
  • tests/end_to_end/tpu/qwen/moe/qwen3.8-2.4t-a95b/2_test_qwen3.8_2.4t_a95b.sh: End-to-end TPU validation script (v5p-512) covering forward-pass golden logit
    verification (forward_pass_logit_checker.py), pre-training, fine-tuning, and decoding.
• User guide (tests/end_to_end/tpu/qwen/moe/run_qwen_moe.md):
  • Added Qwen3.8-2.4T-A95B to the supported Qwen MoE models list, checkpoint download/conversion examples, and correctness validation instructions.
• Supported models reference & announcement:
  • docs/reference/models/supported_models_and_architectures.md: Added a Qwen3.8 section (2.4T-A95B MoE with Hybrid Attention) and updated the Qwen source
    reference link.
  • README.md: Added Qwen3.8 2.4T announcement under Latest news and listed Qwen 3.8 MoE (2.4T) under supported Alibaba models.

Tests

Tested end-to-end on TPU (v5p-512) using the scripts added in this PR:

# Step 1: Convert HF checkpoint to scanned and unscanned MaxText checkpoints on CPU
bash tests/end_to_end/tpu/qwen/moe/qwen3.8-2.4t-a95b/1_test_qwen3.8_2.4t_a95b.sh

# Step 2: Run forward-pass logit checker, pre-training, fine-tuning, and decoding on TPU
bash tests/end_to_end/tpu/qwen/moe/qwen3.8-2.4t-a95b/2_test_qwen3.8_2.4t_a95b.sh

Here is my XLML test results on TPU cluster: https://paste.googleplex.com/5849671810023424

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree
  directive) as explained in our documentation.

[gemini-code-assist]

Code Review

This pull request introduces support for the Qwen3.8-2.4T-A95B MoE model, updating the documentation and adding end-to-end TPU test scripts. Feedback on the changes highlights a syntax error in the documentation's bash example where an inline comment follows a line continuation backslash. Additionally, the test scripts should be improved by making the Hugging Face token optional when a local model path is provided, and by validating the token early in the execution to fail fast before running resource-intensive steps.

[github-actions]

🤖 Hi @JamesDeng42, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[github-actions]

📋 Review Summary

Verdict: Valid end-to-end scripts and docs for Qwen3.8-2.4T-A95B, but blocked on parent PR #5366 with minor bash syntax and token check issues.

Top issues:

1. [Spec] PR #5773 has a hard dependency on PR #5366, which introduces model_name=qwen3.8-2.4t-a95b, qwen3.8-2.4t-a95b.yml, and conversion mappings. Merging this PR into main before #5366 will cause both scripts and documentation examples to fail immediately on unrecognized model configs. → Hold merge until PR #5366 lands on main (or rebase onto its branch).
2. [Bug] In run_qwen_moe.md, a trailing backslash followed by an inline comment (\ # ...) breaks bash line continuation, causing subsequent flags to execute as detached commands. → See inline comment on run_qwen_moe.md.
3. [Robustness] In 2_test_qwen3.8_2.4t_a95b.sh, missing early validation for HF_TOKEN risks failing at the final decode step after long multi-host TPU runs across 512 v5p chips. → See inline comment on 2_test_qwen3.8_2.4t_a95b.sh.
4. [Docs] Step 3 in run_qwen_moe.md mislabels the CPU conversion script as running both conversion and validation. → See inline comment on run_qwen_moe.md.

Inferred spec & validation

• Sources: PR #5773 description; parent PR #5366; existing Qwen MoE test scripts (tests/end_to_end/tpu/qwen/moe/).
• Problem: MaxText lacks end-to-end CPU/TPU validation scripts, user guide workflows, and documentation announcements for Qwen3.8-2.4T-A95B following its architecture bring-up.

Spec item	Implementation	Validating test
R1 (stated): CPU conversion script (1_test_...) for scanned/unscanned Orbax checkpoints	✅ Complete	Manual execution; blocked on PR #5366 for model config
R2 (stated): TPU validation script (2_test_...) covering logit checker, pre-training, fine-tuning, and decoding on v5p-512	⚠️ Missing early HF_TOKEN check	Verified by author on v5p-512 cluster
R3 (stated): Update user guide run_qwen_moe.md with download, conversion, and run instructions	⚠️ Bash syntax error in conversion snippet	❌ none (doc snippet verification)
R4 (stated): Announce and document Qwen3.8 in README and supported models reference	✅ Complete	❌ none (markdown docs)
C1 (inferred): Commands in documentation and scripts must be valid bash syntax	❌ Violated in run_qwen_moe.md line 62 (\ #)	❌ none
C2 (inferred): Functional dependency on parent PR #5366	⚠️ Blocked until PR #5366 merges	tests/unit/qwen3_8_conversion_test.py in PR #5366

[shuningjin]

Thank you for updating the docs and scripts!