Skip to content

Add static inference planning and Vidur drift gates - #2

Merged
reiase merged 1 commit into
mainfrom
feature/infer
Aug 9, 2026
Merged

Add static inference planning and Vidur drift gates#2
reiase merged 1 commit into
mainfrom
feature/infer

Conversation

@reiase

@reiase reiase commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

What changed

  • [KNOWN] Adds a phase-neutral decoder inference frontend with separate model, mapping, and request inputs.
  • [KNOWN] Lowers prefill and decode phase points through ModelIR -> DistributedTaskIR -> PortablePlanIR, with explicit KV state, workspace capacity, workload facts, collectives, implementation requirements, and pass checkpoints.
  • [KNOWN] Adds analytical inference costing plus a separate InferenceCostProvider.resolve() contract; external baselines expose only InferenceBaseline.lookup() and cannot participate in lowering or costing.
  • [KNOWN] Adds exact-key Vidur raw component-profile alignment, a pinned MIT-licensed Phi-2/A100/TP1 fixture, combined Calculon/Vidur regression contracts, and pull-request CI.
  • [KNOWN] Adds bilingual inference/experiment documentation and an SVG architecture diagram.

Why

[INFERRED] This establishes the first auditable inference-planning slice while preserving Blueprinting's planning/evidence boundaries. It deliberately does not present static decoder-block composition as a serving-system simulator.

Review fixes included

  • [KNOWN] Costing now reads PlanTask.workload and explicit plan buffers instead of a hidden InferenceInvocation copy.
  • [KNOWN] Application outputs are named prefill/decode-step/model-execution time, not TTFT/TPOT/E2E; memory is reported as analytical fit rather than general serving feasibility.
  • [KNOWN] Request workload is separate from execution mapping in InferenceAnalysisDraft.
  • [COMPUTED] Vidur reports now include non-cancelling component MAPE and maximum component error. The current three-case comparable-subtotal MAPE is 23.58%, while component MAPE is 54.08% and maximum component error is 99.50%; the gate is explicitly classified as drift detection, not accuracy certification.
  • [KNOWN] Exact CSV integer keys reject fractional, NaN, and infinite values instead of silently truncating them.
  • [KNOWN] A portable workspace buffer is explicit and verifier-supported.

Validation

  • [COMPUTED] uv run pytest: 89 passed.
  • [COMPUTED] Pinned training/inference regression tests: passed.
  • [COMPUTED] Ruff over every staged Python file: passed.
  • [COMPUTED] Bilingual source check: 28 page pairs passed.
  • [COMPUTED] uv run mkdocs build --strict: passed.
  • [KNOWN] Full-repository Ruff is not enabled by this PR because the existing legacy tree currently has unrelated lint debt.

Known limits and follow-ups

  • [KNOWN] The implemented template is dense MHA plus non-gated MLP; full norm/residual topology, GQA/MQA, gated MLP, embedding, LM head, sampler, and MoE are not represented yet.
  • [KNOWN] Current DistributedTaskIR materializes a local TP block; PP and replica composition remain analytical outside that graph.
  • [KNOWN] Decode contexts are recompiled individually rather than specialized from a parametric plan.
  • [KNOWN] Arrivals, continuous batching, scheduler/KV allocation policy, contention, tail latency, and end-to-end Vidur simulation remain future discrete-event work.
  • [KNOWN] Raw Vidur component-key alignment does not establish full Phi-2 topology equivalence.

@reiase
reiase marked this pull request as ready for review August 9, 2026 05:43
@reiase
reiase merged commit 91b5d7c into main Aug 9, 2026
4 checks passed
@reiase
reiase deleted the feature/infer branch August 9, 2026 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant