Skip to content

[ET][native] Verify bundled LLM samples through the runner - #24175

Merged
meta-codesync[bot] merged 5 commits into
gh/digantdesai/181/basefrom
gh/digantdesai/181/head
Oct 28, 2026
Merged

[ET][native] Verify bundled LLM samples through the runner#24175
meta-codesync[bot] merged 5 commits into
gh/digantdesai/181/basefrom
gh/digantdesai/181/head

Conversation

@digantdesai

@digantdesai digantdesai commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

run_hf_llm --verify_samples <set> replays a sample set through generate(),
with a fresh context per sample. Each call's logits are compared under
the sample's tolerance, then the sample's next token is forced. Exits 1 on
any mismatch.

Authored with Claude Code.

Differential Revision: D125544389

@pytorch-bot

pytorch-bot Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24175

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 1138eb1 with merge base 4db778e (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Oct 27, 2026
Merged
Merged
Merged
Merged
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@digantdesai
Update
bae6a6a 
[ghstack-poisoned]
digantdesai added a commit that referenced this pull request Oct 27, 2026
@digantdesai
[ET][native] Verify bundled LLM samples through the runner
986efb1 
Pull Request resolved: #24175

`run_hf_llm --verify_samples <set>` replays a sample set through `generate()`,
with a fresh context per sample; every call must run `--method`. Each call's
logits are compared under the sample's tolerance, then the sample's next
token is forced. Exits 1 on any mismatch.

Authored with Claude Code.
ghstack-source-id: 458182947
@exported-using-ghexport

Differential Revision: [D125544389](https://our.internmc.facebook.com/intern/diff/D125544389/)
@digantdesai
Update
f7d0938 
[ghstack-poisoned]
digantdesai added a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Verify bundled LLM samples through the runner
faa68d1 
Pull Request resolved: #24175

`run_hf_llm --verify_samples <set>` replays a sample set through `generate()`,
with a fresh context per sample; every call must run `--method`. Each call's
logits are compared under the sample's tolerance, then the sample's next
token is forced. Exits 1 on any mismatch.

Authored with Claude Code.
ghstack-source-id: 458441424
@exported-using-ghexport

Differential Revision: [D125544389](https://our.internmc.facebook.com/intern/diff/D125544389/)
@digantdesai
Update
2e1a05d 
[ghstack-poisoned]
digantdesai added a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Verify bundled LLM samples through the runner
7700e48 
Pull Request resolved: #24175

`run_hf_llm --verify_samples <set>` replays a sample set through `generate()`,
with a fresh context per sample; every call must run `--method`. Each call's
logits are compared under the sample's tolerance, then the sample's next
token is forced. Exits 1 on any mismatch.

Authored with Claude Code.
ghstack-source-id: 458574976
@exported-using-ghexport

Differential Revision: [D125544389](https://our.internmc.facebook.com/intern/diff/D125544389/)
digantdesai added a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Verify bundled LLM samples through the runner
6efe396 
Pull Request resolved: #24175

`run_hf_llm --verify_samples <set>` replays a sample set through `generate()`,
with a fresh context per sample; every call must run `--method`. Each call's
logits are compared under the sample's tolerance, then the sample's next
token is forced. Exits 1 on any mismatch.

Authored with Claude Code.
ghstack-source-id: 458574976
@exported-using-ghexport

Differential Revision: [D125544389](https://our.internmc.facebook.com/intern/diff/D125544389/)
digantdesai added a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Verify bundled LLM samples through the runner
bddd3a0 
Pull Request resolved: #24175

`run_hf_llm --verify_samples <set>` replays a sample set through `generate()`,
with a fresh context per sample; every call must run `--method`. Each call's
logits are compared under the sample's tolerance, then the sample's next
token is forced. Exits 1 on any mismatch.

Authored with Claude Code.
ghstack-source-id: 458574976
@exported-using-ghexport

Differential Revision: [D125544389](https://our.internmc.facebook.com/intern/diff/D125544389/)
@digantdesai
Update
1138eb1 
[ghstack-poisoned]
digantdesai added a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Verify bundled LLM samples through the runner
0fccde2 
Pull Request resolved: #24175

`run_hf_llm --verify_samples <set>` replays a sample set through `generate()`,
with a fresh context per sample; every call must run `--method`. Each call's
logits are compared under the sample's tolerance, then the sample's next
token is forced. Exits 1 on any mismatch.

Authored with Claude Code.
ghstack-source-id: 458622296
@exported-using-ghexport

Differential Revision: [D125544389](https://our.internmc.facebook.com/intern/diff/D125544389/)
meta-codesync Bot pushed a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Bundle sample method calls in .ptn packages
ec7a7d9 
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom):
* #24175
* #24174
* #24173
* #24172
* __->__ #24171

A `.ptn` can carry recorded method calls under `samples/<method>/<set>/`
as `index.json` plus `data.safetensors`; program loaders ignore them. A
sample is an ordered list of calls from the initial state with no reset
between, so stateful methods such as LLM prefill then teacher-forced decode
can be checked. Tolerance resolves per call, then sample, then set, and
`write_samples` warns above 1e-3.

`write_samples` and `read_samples` live in
`exir/_serialize/_ptn_samples.py`. The spec is in `backends/native/README.md`;
review it first.

Authored with Claude Code.

Differential Revision: D125544384

Pull Request resolved: #24171
meta-codesync Bot pushed a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Make the LLM runner's next-token choice pluggable
24460aa 
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom):
* #24175
* #24174
* #24173
* __->__ #24172
* #24171

Moves the prefill and decode loop of `run_llm` into `generate`, which takes
a `NextToken` callback mapping logits to the next token. `run_llm` passes
`greedy_argmax_last_row`, so behavior is unchanged. The verify mode later
in the stack uses it to force sample tokens.

Authored with Claude Code.

Differential Revision: D125544385

Pull Request resolved: #24172
meta-codesync Bot pushed a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Read bundled samples in C++
a46b1f7 
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom):
* #24175
* #24174
* __->__ #24173
* #24172
* #24171

Adds `read_sample_set`, which reads one `samples/<method>/<set>/` from a
`.ptn` into calls with resolved tolerance. It validates like `read_samples`
and rejects scalar and list values.

Authored with Claude Code.

Differential Revision: D125544386

Pull Request resolved: #24173
meta-codesync Bot pushed a commit that referenced this pull request Oct 28, 2026
@digantdesai
[ET][native] Bundle LLM prefill and decode samples from export_hf_llm
478868b 
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom):
* #24175
* __->__ #24174
* #24173
* #24172
* #24171

`export_hf_llm --samples N` adds a `golden` set to `forward`: per
prompt, a prefill and 4 decode calls, each decode fed the previous eager
argmax. Tolerance is (1e-3, 1e-3) for float and (2.0, 5e-2) for quantized
weights, where engines quantize activations differently from torchao.

Authored with Claude Code.

Differential Revision: D125544388

Pull Request resolved: #24174
@meta-codesync
meta-codesync Bot merged commit 51ec9f3 into gh/digantdesai/181/base Oct 28, 2026
214 of 216 checks passed
@meta-codesync
meta-codesync Bot deleted the gh/digantdesai/181/head branch October 28, 2026 12:59
@meta-codesync
meta-codesync Bot deployed to cherry-pick-bot October 28, 2026 12:59 Active
@pytorchbot pytorchbot mentioned this pull request Oct 28, 2026
Merged
JakeStevens pushed a commit that referenced this pull request Oct 28, 2026
@pytorchbot @digantdesai
[ET][native] Verify bundled LLM samples through the runner (#24234)
f498e0f 
This PR was created by the merge bot to help merge the original PR into
the main branch.
ghstack PR number: #24175 by
@digantdesai
^ Please use this as the source of truth for the PR details, comments,
and reviews
ghstack PR base:
https://github.com/pytorch/executorch/tree/gh/digantdesai/181/base
ghstack PR head:
https://github.com/pytorch/executorch/tree/gh/digantdesai/181/head
Merge bot PR base:
https://github.com/pytorch/executorch/tree/gh/digantdesai/180/orig
Merge bot PR head:
https://github.com/pytorch/executorch/tree/gh/digantdesai/181/orig
Differential Revision:
[D125544389](https://our.internmc.facebook.com/intern/diff/D125544389/)
@diff-train-skip-merge

Co-authored-by: Digant Desai <digantdesai@meta.com>

This branch was successfully deployed

1 active and 1 inactive deployments
cherry-pick-bot — 1138eb1e Deployed Oct 28, 2026 by meta-codesync[bot] via Try to create a PR with ghstack /orig branch #1872
cadence — 1138eb1e Deployed Oct 28, 2026 by digantdesai via hifi-op-test / hifi4 #34994
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SS-JIA SS-JIA SS-JIA approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@digantdesai @SS-JIA