Skip to content

Add opt-in SmolLM2 tile128 schedule and generation evidence - #5

Open
0z5a wants to merge 1 commit into
RightNow-AI:mainfrom
0z5a:perf/smollm2-tile128-schedule
Open

0z5a wants to merge 1 commit into
RightNow-AI:mainfrom
0z5a:perf/smollm2-tile128-schedule

Conversation

@0z5a

@0z5a 0z5a commented Sep 13, 2026

Copy link
Copy Markdown

Change

Add an opt-in ScheduleConfig preset selecting tiling.gemv.N_tile=128, with complete-generation measurements and reproduction instructions for SmolLM2-135M. The measured schedule reduces tasks per step from 5,286 to 1,716. The default schedule, lowerer, CUDA VM, ABI, oracle and kernel knobs are unchanged.

The existing evaluator can consume the JSON preset. The documented complete-generation experiment uses an external adapter to pass that supported configuration into the unmodified lowerer; this PR does not add a generation API or change its default behavior.

Evidence

Fixed upstream a514bbc20a03bbf698a17443f8f14a27a617fc10; SmolLM2-135M-Instruct revision 12fd25f77366fa6b3b4b768ec3050bf629380bac, full 30-layer model, FP32, no quantization. One RTX PRO 4000 Blackwell 24 GB, Torch 2.13.0+cu130, CUDA 13.0, Transformers 5.16.1.

Four independent processes ran A/P/P/A, each with independent first compilation, contexts 8/64, 8 new tokens, one warmup and three timed requests per context. All 344 step-level logits comparisons passed both CPU ReferenceVM and eager at the original FP32 tolerance; generated tokens matched exactly. Timing began only after all correctness cases passed.

Context Pair Default median seconds/request Tile128 median seconds/request Latency reduction
8 A0/P1 3.486624 1.778946 48.98%
8 A3/P2 3.515794 1.755405 50.07%
64 A0/P1 16.170751 7.911105 51.08%
64 A3/P2 15.953324 7.895794 50.51%

The metric includes lowering, validation, VM/table construction, weight H2D, KV transfer and sampling from a CPU-resident model. It excludes disk loading, first compilation and correctness instrumentation. These are two process-pair screening results, not a stable speedup, GPU-only gain, GPU-resident decode, service-throughput or cuBLAS/vLLM comparison. Allocator peak was 1,320,648,704 bytes in every process; dynamic shared memory was 38,912 bytes in both arms. Further independent validation is needed before changing defaults.

Original runner, all request timings, per-step comparisons and checkpoint/source hashes. The checked-in JSON retains every recorded request time. A separate prefetch-depth candidate lacked consistent gains and is excluded.

Validation

  • Fresh preset checks: 12 structural validations and four complete CPU generation runs pass; default/tile128 match eager tokens for both contexts.
  • Full pytest: 94 passed, 21 skipped, 1 failed. CUDA tests skip on this CPU host. The failure is the unchanged upstream test_search_rejects_invalid_configs_without_crashing: the sabotaged default schedule yields default_score_us=None, while the test asserts it is non-null. The test and search source match upstream HEAD byte-for-byte; this PR modifies no Python source.
  • ruff check .: five existing E702/E741 findings in unchanged instructions/reference.py; no new Python files or lint findings are introduced.
  • Both JSON files parse and git diff --check passes. No GPU tests were rerun for publication.

Signed-off-by: 0z5a <192209249+0z5a@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant