Conversation
Signed-off-by: 0z5a <192209249+0z5a@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
Add an opt-in
ScheduleConfigpreset selectingtiling.gemv.N_tile=128, with complete-generation measurements and reproduction instructions for SmolLM2-135M. The measured schedule reduces tasks per step from 5,286 to 1,716. The default schedule, lowerer, CUDA VM, ABI, oracle and kernel knobs are unchanged.The existing evaluator can consume the JSON preset. The documented complete-generation experiment uses an external adapter to pass that supported configuration into the unmodified lowerer; this PR does not add a generation API or change its default behavior.
Evidence
Fixed upstream
a514bbc20a03bbf698a17443f8f14a27a617fc10; SmolLM2-135M-Instruct revision12fd25f77366fa6b3b4b768ec3050bf629380bac, full 30-layer model, FP32, no quantization. One RTX PRO 4000 Blackwell 24 GB, Torch 2.13.0+cu130, CUDA 13.0, Transformers 5.16.1.Four independent processes ran A/P/P/A, each with independent first compilation, contexts 8/64, 8 new tokens, one warmup and three timed requests per context. All 344 step-level logits comparisons passed both CPU ReferenceVM and eager at the original FP32 tolerance; generated tokens matched exactly. Timing began only after all correctness cases passed.
The metric includes lowering, validation, VM/table construction, weight H2D, KV transfer and sampling from a CPU-resident model. It excludes disk loading, first compilation and correctness instrumentation. These are two process-pair screening results, not a stable speedup, GPU-only gain, GPU-resident decode, service-throughput or cuBLAS/vLLM comparison. Allocator peak was 1,320,648,704 bytes in every process; dynamic shared memory was 38,912 bytes in both arms. Further independent validation is needed before changing defaults.
Original runner, all request timings, per-step comparisons and checkpoint/source hashes. The checked-in JSON retains every recorded request time. A separate prefetch-depth candidate lacked consistent gains and is excluded.
Validation
pytest: 94 passed, 21 skipped, 1 failed. CUDA tests skip on this CPU host. The failure is the unchanged upstreamtest_search_rejects_invalid_configs_without_crashing: the sabotaged default schedule yieldsdefault_score_us=None, while the test asserts it is non-null. The test and search source match upstream HEAD byte-for-byte; this PR modifies no Python source.ruff check .: five existing E702/E741 findings in unchangedinstructions/reference.py; no new Python files or lint findings are introduced.git diff --checkpasses. No GPU tests were rerun for publication.