[None][test] Add ModelExpress post-merge accuracy canaries - #18561
Draft
moraxu wants to merge 7 commits into
Draft
[None][test] Add ModelExpress post-merge accuracy canaries#18561moraxu wants to merge 7 commits into
moraxu wants to merge 7 commits into
Conversation
Add tensorrt_llm/_torch/weight_sharing/weight_manifest.py: a SHA-256 per-tensor manifest of a root module's registered parameters and buffers with layout metadata, storage-alias partitions, skipped-tensor records, a whole-manifest digest, and a format version. It is env-gated by MX_WEIGHT_MANIFEST_DIR and MX_WEIGHT_MANIFEST_ROLE so production loads never hash or write anything. The comparison is byte-for-byte, deliberately stronger than the exact-value equality of torch.testing.assert_close(rtol=0, atol=0, equal_nan=True): it distinguishes signed zeros and NaN payloads. Unit tests pin the contract, including a corruption-injection trio (single bit flip, +0.0 -> -0.0, NaN payload change). Signed-off-by: Michal Guzek <mguzek@nvidia.com>
…aries ModelLoader.load writes the final-state manifest once per rank at its single return point, after every post-load hook and MoE load-balancer finalization and before engine warmup, and records the cost as the weight_manifest_seconds metric. MXCheckpointLoader writes the transfer-boundary manifest at the receiver's full P2P success (now CUDA-synchronized first) and at the donor's publish point, outside the best-effort publish guard so a manifest problem is loud. Both hooks are no-ops unless MX_WEIGHT_MANIFEST_DIR is set. The unit lifecycle harness additionally compares canonical tensor bytes so the staged receiver is held to the same byte-level contract. Signed-off-by: Michal Guzek <mguzek@nvidia.com>
… probe Move the reusable pieces of the ModelExpress E2E test into tests/integration/defs/model_express/mx_harness.py and the stdlib-only mx_evidence.py (transfer-log rules shared with the worker script), and make tests/integration/defs/model_express a package. test_mx_donor_receiver now also collects per-rank weight manifests from all three roles and enforces two tiers: donor-at-publish and receiver-at-receive parameters must be byte-identical, and the final manifests of baseline, donor, and receiver must be pairwise byte-identical (parameters and buffers, skipped sets, alias partitions), with only a per-row, documented exemption mechanism and no numeric tolerance. The behavioral probe grows from 2 prompts x 8 tokens to 8 prompts x 32 greedy tokens with engine limits raised to fit. Payloads, logs, manifests, and timing are archived under --output-dir. Signed-off-by: Michal Guzek <mguzek@nvidia.com>
…n probe Signed-off-by: Michal Guzek <mguzek@nvidia.com>
Extend mx_e2e_worker.py with an opt-in --eval-* mode: the receiver sizes its engine for the task, keeps the tokenizer, self-checks its own MX transfer logs before spending any evaluation time (exit status 3 on a fallback), evaluates the tensorrt_llm.evaluate task with the reference sampling parameters in its own subprocess, and writes the score to JSON. The smoke path is unchanged when the flags are absent. test_model_express_accuracy.py runs one reference-backed task per qualified family (Meta-Llama-3-8B-Instruct MMLU, Qwen2.5-7B-Instruct MMLU, Qwen3-8B GSM8K) on an MX receiver started from a metadata-only snapshot and asserts the accuracy_core hypothesis-testing threshold without constructing an LLM in the pytest process. There is no paired HF-baseline evaluation: the reference value is the baseline. Transfer evidence and donor/receiver weight manifests are checked first, and load/eval times are recorded as junit properties and --output-dir JSON without thresholds. Signed-off-by: Michal Guzek <mguzek@nvidia.com>
Add DGX_H100-2_GPUs-PyTorch-ModelExpress-Post-Merge-1 (Redis + ModelExpress sidecars, post-merge only) and register the accuracy canaries as a stage: post_merge block in l0_model_express.yml. Both land together because every post_merge test row must map to a Post-Merge stage of its test-db file. Signed-off-by: Michal Guzek <mguzek@nvidia.com>
Signed-off-by: Michal Guzek <mguzek@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Post-merge accuracy tier of the MX Model Family Testing Proposal (Levers 2 and 3), stacked on #18560 (weight manifests + widened probe).
mx_e2e_worker.py): an opt-in--eval-*flag group sizes the engine for the task (max_seq_len = input + output,max_num_tokens >= 8192, eval batch size,kv_cache_configfraction), keeps the tokenizer, self-checks its own MX transfer logs right after load and exits with status 3 before spending any eval time on a fallen-back run, runs thetensorrt_llm.evaluatetask with the reference sampling params (greedy,truncate_prompt_tokens), and writes the score to JSON. The receiver stays a fresh subprocess, soCUDA_VISIBLE_DEVICES, offline flags,OMPI_MCA_*, andMX_TRANSFER_LOG_DIRare set before any MPI/UCX import; the smoke path is byte-for-byte unchanged when the flags are absent.test_model_express_accuracy.py):MxAccuracyCase(MxE2ECase)rows +MxEvalSpec.from_task(...)derivenum_samples, dataset path, and input/output lengths from theaccuracy_coretask class, so the sample count always matches the threshold'sn. The pytest process never builds anLLM: it assertsHypothesisTestingParams.assert_passing(score)from the reference YAMLs (no paired HF-baseline eval — the reference value is the baseline), afterassert_transfer_evidenceand a donor↔receiver weight-manifest comparison (transfer + final tiers). Load/eval seconds, score, and threshold are recorded as junit properties and asmodel_express_accuracy/<id>.jsonunder--output-dir; load times are observed, not gated.llama3-8b-instruct-mmlu-tp1(meta-llama/Meta-Llama-3-8B-Instruct, MMLU),qwen2.5-7b-instruct-mmlu-tp1(Qwen/Qwen2.5-7B-Instruct, MMLU),qwen3-8b-gsm8k-tp1(Qwen3/Qwen3-8B, GSM8K). Llama-3.1/3.2 are out of the envelope (rope_type != default/ tied embeddings). No Mistral row yet ([TRTLLM-14881][feat] qualify Mistral dense for MX #18558); adding one is a singleMxAccuracyCaseparam plus a reference entry.DGX_H100-2_GPUs-PyTorch-ModelExpress-Post-Merge-1stage (jenkins/L0_Test.groovy, tuple booleantrue= Redis + ModelExpress sidecars; the-ModelExpress-substring selects the pip installs;Post-Mergeselectsstage=post_merge) and astage: post_mergeblock inl0_model_express.yml— in the same commit, becausetests/unittest/tools/test_test_to_stage_mapping.pyrequires every post_merge row to map to a Post-Merge stage. The stage runs on every main commit and never pre-merge; exact--stage-listnames need no label.test_mx_accuracy_cases_have_references(CPU) guards the reference keys (note the nonstandardQwen3/Qwen3-8Bkey) and round-tripsMxEvalSpec.to_argv()through the worker's argument parser.Test Coverage
pytest -sv tests/integration/defs/model_express/test_model_express_accuracy.py -k cases_have_references(CPU)pytest -sv tests/unittest/tools/test_test_to_stage_mapping.py(CPU; same-commit stage/row invariant)python scripts/test_to_stage_mapping.py --stages "DGX_H100-2_GPUs-PyTorch-ModelExpress-Post-Merge-1"→ the four accuracy test IDs/bot run --stage-list "DGX_H100-2_GPUs-PyTorch-ModelExpress-Post-Merge-1"; expected pass thresholds (α=0.05, σ=50): MMLU 67.74→65.92 and 75.32→73.50, GSM8K 87.11→83.91. To fill in from the stage run: per case donor/receiverload_seconds,eval_seconds, score vs threshold,Matched m/ncounts.PR Checklist
CODING_GUIDELINES.md; NVIDIA header on new files;git commit -sdocs/source/features/model-express.md, "Accuracy Canaries (Post-Merge)")l0_model_express.ymlblock is appended at end-of-file to avoid [TRTLLM-14881][feat] qualify Mistral dense for MX #18558's tail edits)Follow-ups (out of scope): OpenSearch posting of load/eval times, TP2 canary rows in the OnDemand stage, Mistral row, family rotation.