Skip to content

Add Fused Residual Add and LayerNorm with Saved Statistics challenge (hard #143) - #352

Merged
kunal-mansukhani merged 1 commit into
mainfrom
add-fused-residual-layernorm
Oct 16, 2026
Merged

kunal-mansukhani merged 1 commit into
mainfrom
add-fused-residual-layernorm

Conversation

@kunal-mansukhani

Copy link
Copy Markdown
Contributor

New hard challenge #143: fused residual add + LayerNorm forward that also writes residual_out (x + residual) and per-row mean/rstd for the backward pass.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 152 lines changed.

  • ID 143 is not used by any existing challenge.
  • It is distinct from #83 (residual add + RMSNorm, one output) and #113 (plain LayerNorm): it fuses the residual add with LayerNorm and writes four outputs, including saved mean/rstd. The large-offset tests also force a numerically stable variance.
  • challenge.py imports only ctypes, typing, torch and the challenge base. There is no network, file, subprocess or import-time code.
  • It has all 5 required methods, no init override, and name/atol/rtol/num_gpus/access_tier set. The private helpers _make and _outputs are extras, which is a minor deviation but harmless.
  • reference_impl asserts shapes and dtypes with no device checks, and uses standard PyTorch ops with a float64 stable reference. The hand-computed example matches the code (rstd 1.1547 and 316.2278).
  • All tensor allocations use self.device. Outputs are pre-filled with 123.0 so skipped writes fail.
  • The 12 functional tests cover C=1, constant rows, large eps, zero residual, negative gamma, ragged widths, DC offsets of 500 and -300, and 1600 and 5120 widths. atol/rtol of 1e-3 is sensible and still rejects the one-pass variance on the offset tests.
  • Performance test memory is about 1.07GB (four 16384x4096 float32 tensors), well under the 3.2GB limit.
  • The six starters are empty stubs with only the parameter comment, and the Mojo starter uses max.gpu imports and MutAnyOrigin. challenge.html matches the signature, constraints and example.

@kunal-mansukhani
kunal-mansukhani merged commit fb19153 into main Oct 16, 2026
6 checks passed
@kunal-mansukhani
kunal-mansukhani deleted the add-fused-residual-layernorm branch October 16, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant