Skip to content

examples/qwen3: add SwiGLU MLP component with equivalence tests and microbenchmark#267

Open
chhwang wants to merge 1 commit into
qwen3-q4-attnfrom
qwen3-q5-mlp
Open

examples/qwen3: add SwiGLU MLP component with equivalence tests and microbenchmark#267
chhwang wants to merge 1 commit into
qwen3-q4-attnfrom
qwen3-q5-mlp

Conversation

@chhwang

@chhwang chhwang commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

examples/qwen3: add SwiGLU MLP component with equivalence tests and microbenchmark

Q5 of the Qwen3-on-ARK component build-out. Three new files, no existing
files modified.

ark_mlp.pyark_swiglu_mlp() performs gate/up/down projections via
torch.matmul and the SiLU·gate activation via F.silu(gate) * up.
ARK-native ark_silu_gate (composed ark.sigmoid + ark.mul) is retained
dormant: the upstream composed-graph planner bug that crashed Q4 at 4-D
shapes also crashes at the MLP intermediate shape (2048, 12288). Output
wrapped in ark.copy for .eval() API consistency.

test_mlp.py — 7 pytest cases gated with skipif(not cuda):
test_silu_gate (intermediate activation), test_mlp_small (B=1, S=16),
test_mlp_prefill (B=1, S=128), test_mlp_decode (B=1, S=1),
test_mlp_batch (B=2, S=16), test_mlp_output_shape, and
test_ark_silu_gate_large_xfail (subprocess spike at (2048, 12288)
documenting the upstream bug). Tolerance: atol/rtol=5e-3.

bench_mlp.py — microbenchmark comparing torch eager SwiGLUMLP vs
ark_swiglu_mlp at Qwen3-8B shapes (S=2048 prefill, S=1 decode). Banner
labels output "torch-only (ARK silu·gate deferred to upstream fix / Q10)."

Follows Q4 patterns: torch fallback for crashed ARK ops, small-config tests,
subprocess-isolated xfail for bug documentation.

…ith SiLU·gate, equivalence test vs torch reference, microbenchmark
@chhwang chhwang changed the title ark-dev: examples/qwen3: SwiGLU MLP component — gate/up/down matmul with SiLU·gate, equivalence test vs torch reference, microbenchmark examples/qwen3: add SwiGLU MLP component with equivalence tests and microbenchmark Jun 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant