Skip to content

[Skills] Fix silently-ignored frontmatter, de-drift kernel skills and CLAUDE.md - #1027

Merged
coderfeli merged 15 commits into
mainfrom
skills/dedrift-and-fix-frontmatter
Aug 24, 2026
Merged

[Skills] Fix silently-ignored frontmatter, de-drift kernel skills and CLAUDE.md#1027
coderfeli merged 15 commits into
mainfrom
skills/dedrift-and-fix-frontmatter

Conversation

@jhinpan

@jhinpan jhinpan commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Audit of all 16 skills in .claude/skills/, verified against the source at
e3380676. No skill was removed — each has a distinct job — but several had
drifted far enough to be actively misleading.

Broken as written

Frontmatter that Claude Code silently ignores. capture-kernel-trace,
kernel-trace-analysis, and lds-optimization declared tools:, which is not
a recognized SKILL.md field — the field is allowed-tools:. Their tool
pre-approval has never applied. kernel-trace-analysis also carried a note:
key holding a real constraint ("analysis is programmatic, do NOT use GUI
tools") that consequently never reached the model. format-code declared no
allowed-tools at all.

Worse, build-rocm-image's description was an unquoted one-line scalar
containing Usage: /build-rocm-image; the embedded : makes the YAML
invalid, and invalid frontmatter is dropped wholesale — taking
allowed-tools: Bash and the declared name/description with it.

All 16 skills now parse as YAML and expose only recognized keys.

A skeleton that cannot run. flydsl-tile-programming Pattern D opened with
from flydsl.expr import buffer_ops. #880 moved that module to
kernels/common/buffer_ops.py, so the skeleton raises ImportError. Replaced
with the make_buffer_tensor + fx.copy view; the raw intrinsics remain
documented as an escape hatch at their real path.

Wrong architecture. The same skill labelled s_wait_loadcnt /
s_wait_storecnt / s_wait_dscnt as "CDNA4 / gfx950". These are gfx12+
(gfx1250 / RDNA4) — in-tree they appear only in gfx1250 and rdna3 kernels, and
kernels/gemm/rdna3_f16_gemm.py:304 says so directly. Following that guidance
on gfx950 does not work.

A TV layout that is not equivalent. Both the wizard and the reference built
a TiledCopy TV layout from a bare raked_product. make_layout_tv also
derives the mapping via composition(right_inverse(layout_mn), ...) — see
examples/02-tiledCopy.py. Both now use make_layout_tv.

Invalid sample. flydsl-kernel-authoring's element-wise template had two
body lines flush-left, so it was not valid Python. Its Abs recipe also dropped
the Vec(...) wrap every sibling recipe uses.

Dead cross-reference. That skill pointed twice at a
flydsl-kernel-code-cleanup skill; the skill is named kernel-code-cleanup.

Self-contradiction

prefetch-data-load documents the gfx942 combined 512-entry arch_vgpr + accum_vgpr model and states ">512 → SPILL", then tells the reader to keep
arch_vgpr + prefetch_vgprs <= 256 "to avoid spills". 256 is the 2-wave
occupancy boundary, not the spill threshold.

More broadly, the authoring/debug/perf skills still taught buffer_ops,
ArithValue, raw rocdl.mfma_*, and copy_atom_call as current, while
kernel-code-cleanup (the newest skill) exists specifically to migrate code
off them. That is now reconciled: copy_atom_callfx.copy at single-atom
sites per cleanup §7b, ArithValue dropped from recommended-type lists, and
the deprecated spellings that remain in transcribed examples are labelled as
such with a pointer to kernel-code-cleanup.

Also: debug-flydsl-kernel used the deprecated rocm-smi (now amd-smi), and
the expr/rocdl module list was missing cdna3, cdna5, rdna3, rdna4.

One developer's environment in shared skills

flydsl-kernel-authoring §12 ran every example through
ssh hjbog-srdc-39.amd.com 'docker exec hungry_dijkstra ...';
kernel-trace-analysis began with cp ~/Documents/input.yaml;
capture-kernel-trace told the agent to read a MEMORY.md that does not exist
in this repo. All three now show the local path first and parameterise or ask
for the remote one.

Smaller items

  • format-code documented only the Python CI gate; pre-checks.yaml also runs
    cpp-style via .github/scripts/check_cpp_style.sh. Documented, including
    that it checks rather than rewrites and that BASE_SHA must be set locally
    or it falls back to HEAD^.
  • gemm-optimization §3/§4 restate lds-optimization and prefetch-data-load
    in GEMM-specific form with no link either way. Cross-references added — the
    split itself is deliberate and left intact.
  • rocprof-trace-decoder was pinned at 0.1.6 in one skill and 0.1.5 in
    another; the version-matching rule is now stated instead.
  • add-target-atom-op's description was 1199 chars, the longest of the 16 and
    always resident in the skill listing. Trimmed to the routing signal; the
    design detail it duplicated is already the skill body (§1.1–1.3). Listing
    cost drops from ~2180 to ~1820 est. tokens.

Deliberately not changed

  • lds-optimization padding math needs an author's eye. It argues that
    padding one fp16 element gives "129 * 2 / 4 = 64.5 → fractional → conflicts
    eliminated". One fp16 is 2 bytes — half a 4-byte bank — so the shift only
    produces a 2-row period rather than eliminating conflicts, and the companion
    formula padding_elements = 32 / element_size_bytes looks inverted. I did
    not rewrite a hardware claim I cannot measure here; flagging for review.
  • fx.thread_idx.x vs gpu.thread_id("x"). flydsl-kernel-authoring
    calls the former legacy, but it is the dominant spelling in kernels/ (38
    uses vs 8). Both are exported; churning the skeletons would make them less
    representative than the tree.
  • capture-kernel-trace / kernel-trace-analysis overlap. The latter's
    "Mode B" duplicates the former's collection steps nearly verbatim, down to
    the att_* values. Consolidating changes how both are invoked, so it is left
    for a separate decision.

Verification

  • All 16 SKILL.md frontmatter blocks parse as YAML with only recognized keys,
    and every name matches its directory.
  • Every python fence in the edited skills re-checked for indentation errors;
    the only real one is fixed (remaining hits are markdown list sub-blocks and a
    multi-line formula).
  • Every repo path, script, doc, and symbol referenced by the edited text was
    confirmed to exist at e3380676.
  • scripts/check_python_style.sh: no Python files changed.
  • Documentation only — no library, kernel, or build code is touched.

Five skills had frontmatter that Claude Code silently ignored:

- capture-kernel-trace, kernel-trace-analysis, lds-optimization used
  `tools:`, which is not a recognized SKILL.md field. The intended tool
  pre-approval never applied, so every invocation re-prompted for
  permissions. The field is `allowed-tools:` (it accepts both the
  space- and comma-separated forms already in use).
- kernel-trace-analysis additionally carried a `note:` key, also not a
  recognized field. Its content ("analysis is programmatic, do not use
  GUI tools") is a real constraint that never reached the model, so it
  moves into the skill body.
- format-code declared no `allowed-tools` at all; add the set it uses.
- build-rocm-image's `description` was an unquoted one-line scalar
  containing "Usage: /build-rocm-image", whose embedded `: ` makes the
  YAML invalid. Invalid frontmatter is dropped wholesale, taking
  `allowed-tools: Bash` with it. Switch to the `>` block form the other
  15 skills use.

All 16 skills now parse as YAML and expose only recognized keys.
The authoring/debug/perf skills had drifted behind kernel-code-cleanup,
so the skill set contradicted itself on which API is current.

Broken as written:

- flydsl-tile-programming's "Pattern D" skeleton opened with
  `from flydsl.expr import buffer_ops`. That module moved to
  `kernels/common/buffer_ops.py` in #880, so the skeleton raises
  ImportError. Replaced with the make_buffer_tensor + fx.copy view, with
  the raw intrinsics kept as a documented escape hatch at their real path.
- flydsl-tile-programming labelled `s_wait_loadcnt` / `s_wait_storecnt` /
  `s_wait_dscnt` as "CDNA4 / gfx950". They are gfx12+ (gfx1250 / RDNA4);
  in-tree they appear only in gfx1250 and rdna3 kernels, and
  rdna3_f16_gemm.py notes the same. Following that text on gfx950 does
  not work. CDNA3/CDNA4 now show the arch-dispatched keyword form.
- Both wizard and reference built a TiledCopy TV layout from a bare
  `raked_product`. That is not equivalent to `make_layout_tv`, which
  additionally derives the mapping via composition(right_inverse(...)) --
  see examples/02-tiledCopy.py. Both now use make_layout_tv.
- flydsl-kernel-authoring's element-wise template had two body lines
  flush-left, so the sample was not valid Python; its Abs recipe dropped
  the `Vec(...)` wrap its sibling recipes all use.
- flydsl-kernel-authoring pointed twice at a
  "flydsl-kernel-code-cleanup" skill, which does not exist under that
  name, and listed the deleted `python/flydsl/expr/buffer_ops.py` among
  the live source paths.

Stale but working:

- copy_atom_call -> fx.copy at single-atom sites, per cleanup 7b
  (one-for-one, behavior-preserving, perf-neutral). The gfx1250 TDM
  calls stay raw: their global operand has no partition form.
- ArithValue dropped from the recommended-types lists in
  debug-flydsl-kernel and prefetch-data-load; it is deprecated.
- debug-flydsl-kernel used rocm-smi, deprecated in favour of amd-smi.
- expr/rocdl module list completed (cdna3, cdna5, rdna3, rdna4 missing).

prefetch-data-load and kernel-trace-analysis keep their transcribed
examples in the raw-intrinsic spelling -- they demonstrate loop structure
and instruction scheduling, not the load API -- but now say so and point
at kernel-code-cleanup instead of reading as current guidance.

prefetch-data-load also contradicted itself on register budget: it
documents the gfx942 combined 512-entry arch+accum model, then told the
reader to keep `arch_vgpr + prefetch_vgprs <= 256` "to avoid spills".
256 is the 2-wave occupancy boundary; spills start above 512.

Index spelling (fx.thread_idx.x vs gpu.thread_id("x")) deliberately left
alone: both are exported and fx.thread_idx.x dominates in kernels/.
Several git-tracked skills instructed the agent to use a specific
machine, container, or local file that only exists on one workstation:

- flydsl-kernel-authoring 12 ran every example through
  `ssh hjbog-srdc-39.amd.com 'docker exec hungry_dijkstra ...'`. Now
  shows the local invocation first and parameterises the remote form on
  $HOST / $CONTAINER / $FLYDSL_ROOT, pointing at build-flydsl for setup.
- kernel-trace-analysis Step 2 began `cp ~/Documents/input.yaml`, a path
  no clone has. The job spec is already given in full immediately below,
  so the copy step is just removed.
- capture-kernel-trace told the agent to "Check MEMORY.md for the user's
  current remote access configuration". There is no MEMORY.md in this
  repo; ask the user instead, and skip the SSH/Docker wrappers on a
  local GPU.
- Example hostnames in the build skills' argument tables genericised.

build-flydsl's "Verified Environments" table is left as-is: it records
which image and ROCm version were actually validated, which is
information rather than an instruction.
- format-code only described the Python gate, but pre-checks.yaml also
  runs a `cpp-style` job via .github/scripts/check_cpp_style.sh. Document
  it, including that it checks rather than rewrites, and that BASE_SHA
  must be set locally or it falls back to HEAD^ instead of the branch.
- gemm-optimization 3 (XOR swizzle) and 4 (prefetch pipeline) restate,
  in GEMM-specific form, what lds-optimization and prefetch-data-load
  cover generally, with no link in either direction. Add the pointers so
  the two levels stay reconciled.
- kernel-trace-analysis pinned rocprof-trace-decoder 0.1.6 while
  build-rocm-image selects by ROCm version and pins 0.1.5. State the
  version-matching rule and cross-reference, rather than fixing a number
  that will drift again.
- add-target-atom-op's description was 1199 chars -- the longest of the
  16, and always resident in the skill listing. Trimmed to the routing
  signal; the design detail it repeated is already the skill body (1.1-1.3).

Skill listing drops from ~2180 to ~1820 est. tokens.
Copilot AI lite review requested due to automatic review settings August 18, 2026 20:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Self-review of the three preceding commits. Two edits were worse than
what they replaced and are reverted; the rest are errors introduced or
carried by this branch.

Reverted -- my rewrites broke working text:

- flydsl-tile-programming Pattern D. The make_buffer_tensor template I
  substituted was not runnable: it dropped `bid`, so every workgroup
  reads the same tile, and `fx.slice(tA, (None, tid))` on an (M,N) view
  yields an M-element strided column feeding a 4-element rmem tensor,
  with a "# after partitioning tA" comment for a step the snippet never
  performs. Restored the original working snippet and fixed only its
  real defect, the dead `flydsl.expr` import path.
- lds-optimization write-read-distance example. Converting the loads to
  `fx.copy` introduced six undefined names (tNext/rNext/tScale/rScale/
  copy_atom/scale_atom), stranded `next_offsets` as dead, and dropped
  the `next_data`/`scale_factor` bindings the prose refers to. Restored
  and labelled schematic, matching prefetch-data-load's treatment.

Wrong as written:

- `s_waitcnt(vmcnt=0, lgkmcnt=0)` was commented "everything". An unset
  counter maps to its maximum, which the hardware reads as already
  satisfied (utils.normalize_s_waitcnt_field), so expcnt was not waited
  on -- weaker than the `s_waitcnt(0)` it replaced. Now names all three.
- The keyword s_waitcnt form is not universal: universal.py dispatches
  gfx942/gfx950/gfx11xx/gfx120x and raises ValueError otherwise, so it
  fails on gfx1250. Restored the positional bitfield as the documented
  escape hatch, which the previous commit had deleted.
- `amd-smi reset -g <idx>` is not a reset: `-g` selects the device and a
  reset argument is mandatory. Corrected to `-G -g <idx>`.
- Claimed build-rocm-image selects the decoder release automatically. It
  hardcodes RTD_VERSION="0.1.5" and only echoes the detected ROCm
  version. Reworded, and the install block now derives its filenames
  from one RTD_VERSION instead of repeating the literal three times.
- `fx.Vector` is `class Vector(ArithValue)`, so telling the reader to use
  fx.Vector "not the deprecated ArithValue" was self-contradictory.
  Reworded to be about the constructor, in both skills carrying it.
- `dtype=T.i32` had no `T` in scope. `fx.T` is exported, so the snippet
  now spells it `fx.T.i32` -- uncalled, because Types members are
  properties. The same audit found `fx.T.f32()` in Pattern D, a
  pre-existing property-call bug now fixed.

Carried defects in text this branch touched:

- kernel-trace-analysis shipped hotspot_analyzer.py and pmc_l2_analyzer.py
  in its scripts/ dir but told the agent to hand-write the first into
  /tmp and ran the second from a repo-root scripts/ containing neither.
  Both now use ${CLAUDE_SKILL_DIR}. Also fixed a Step 4/Step 5 misref.
- format-code's C++ block inherited the false-pass the Python half warns
  about: the wrapper diffs committed revisions, so uncommitted edits exit
  0 locally then fail CI. Documented, and the "Check only" entry point
  now names both jobs.
- flydsl-kernel-authoring listed a `kernels/mma/` package that does not
  exist (it is kernels/common/mma/) and omitted kernels/mega_moe/.
Fixing only the skills left the contradiction half-resolved: CLAUDE.md is
loaded every turn, skills load on demand, so the higher-priority surface
was still prescribing what the skills had just migrated away from.

- "Prefer ... `fx.copy_atom_call`" now reads `fx.copy` / `fx.gemm`, and
  notes the raw buffer intrinsics moved to kernels/common in #880.
- Every path in the Helper-placement rule predated the kernels/
  subpackage reorg and resolved to nothing: kernels_common.py,
  moe_common.py, layout_utils.py, fp8_gemm_utils.py, dpp_utils.py,
  mfma_epilogues.py, mfma_preshuffle_pipeline.py. The rule says to
  search for and reuse an existing helper rather than duplicate one, so
  unresolvable paths actively cause the duplication it forbids.
  pipeline_utils.py no longer exists anywhere and is dropped.
- Same reorg: kernels/rdna3_f16_gemm.py -> kernels/gemm/,
  kernels/pa_decode_fp8.py -> kernels/attention/, custom_all_reduce.py
  qualified as kernels/comm/.
- `buffer_ops` was listed as lazy-loaded from expr/__init__.py, but
  _BACKEND_MODULES holds only rocdl and tdm_ops since #880.
- The expr/rocdl module list omitted cdna3, rdna3, rdna4, utils, enum.

Every `kernels/`, `python/`, `docs/`, `tests/`, `lib/` path in CLAUDE.md
now resolves.

Also drops the internal host and container name from build-flydsl's
Verified Environments table, keeping the image and ROCm version, which
are the part that records what was actually validated.

docs/ still teaches copy_atom_call and the bare raked_product TV layout
(quickstart.rst, kernel_authoring_guide.md, cute_layout_algebra_guide.md,
architecture_guide.md) and publishes to the public site. Left for a
separate PR to keep this one reviewable.
@jhinpan jhinpan changed the title [Skills] Fix silently-ignored frontmatter and de-drift kernel skills [Skills] Fix silently-ignored frontmatter, de-drift kernel skills and CLAUDE.md Aug 18, 2026
@jhinpan

jhinpan commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

Update: self-review pass + CLAUDE.md

Ran a full review over this branch before asking for eyes. It found 15 issues, all
verified against the source
— including two places where my own edits were worse
than the text they replaced. Fixed in 77a7dae5, plus a scope extension in b5fdcda0.

Reverted — my rewrites broke working text

  • flydsl-tile-programming Pattern D. The make_buffer_tensor template I
    substituted was not runnable: it dropped bid (so every workgroup reads the same
    tile), and fx.slice(tA, (None, tid)) on an (M,N) view yields an M-element strided
    column feeding a 4-element rmem tensor, under a # after partitioning tA comment for
    a step the snippet never performs. Restored the original working snippet; fixed only
    its real defect, the dead flydsl.expr import path.
  • lds-optimization write-read-distance example. Converting the loads to fx.copy
    introduced six undefined names, stranded next_offsets as dead, and dropped the
    bindings the surrounding prose refers to. Restored and labelled schematic.

Wrong as written

  • s_waitcnt(vmcnt=0, lgkmcnt=0) was commented "everything". An unset counter maps to
    its maximum, which the hardware reads as already satisfied, so expcnt was never
    waited on — weaker than the s_waitcnt(0) it replaced.
  • The keyword s_waitcnt form is not universal: universal.py dispatches
    gfx942/gfx950/gfx11xx/gfx120x and raises ValueError otherwise, so it fails on
    gfx1250 — whose escape hatch the previous commit had deleted. Restored.
  • amd-smi reset -g <idx> is not a reset; -g only selects the device. Now -G -g.
  • I claimed build-rocm-image selects the decoder release automatically. It hardcodes
    RTD_VERSION="0.1.5".
  • fx.Vector is class Vector(ArithValue), so "use fx.Vector, not the deprecated
    ArithValue" was self-contradictory.
  • Chasing a dtype=T.i32 finding surfaced a pre-existing bug: Types members are
    @property, so Pattern D's fx.T.f32() was calling an ir.Type.

Carried defects in text this branch touched

  • kernel-trace-analysis ships hotspot_analyzer.py and pmc_l2_analyzer.py in its
    scripts/ dir, but told the agent to hand-write the first into /tmp and ran the
    second from a repo-root scripts/ containing neither. Both now use
    ${CLAUDE_SKILL_DIR}.
  • format-code's C++ block inherited the false-pass its Python half warns about: the
    wrapper diffs committed revisions, so uncommitted edits exit 0 locally then fail CI.
  • flydsl-kernel-authoring listed a kernels/mma/ package that does not exist.

Scope extension: CLAUDE.md (b5fdcda0)

The sharpest finding was that de-drifting only the skills left the job half done —
CLAUDE.md is loaded every turn while skills load on demand, and it still said
"Prefer ... fx.copy_atom_call", recreating the exact contradiction this PR set out to
remove. Worse, all 10 paths in its Helper-placement rule resolved to nothing after
the kernels/ subpackage reorg — and that rule tells you to search for and reuse an
existing helper rather than duplicate one, so unresolvable paths actively cause the
duplication it forbids. Every kernels/, python/, docs/, tests/, lib/ path in
CLAUDE.md now resolves.

Still open

docs/ (quickstart.rst, kernel_authoring_guide.md, cute_layout_algebra_guide.md,
architecture_guide.md) still teaches copy_atom_call and the bare raked_product TV
layout, and publishes to the public site. Left for a separate PR to keep this one
reviewable. The lds-optimization padding question from the description is also still
open for someone who can measure it.

Worth noting: nothing in tests/, scripts/, or .github/workflows/ lints doc or
skill prose against the Python API (check_typed_arithmetic_usage.py scans .py only),
so this whole drift class recurs undetected.

A second review over the branch. Two classes of defect: API names that
never existed, and places where an earlier commit in this branch fixed
one occurrence of something and left its twin.

Phantom symbols (verified against the source tree, not the installed
wheel -- /opt/venv ships a stale build that still has expr/vector.py and
expr/buffer_ops.py, which invalidated an earlier runtime check):

- All three names in flydsl-kernel-authoring's MFMA reference were
  wrong: mfma_f32_16x16x16_f16 -> mfma_f32_16x16x16f16,
  mfma_f32_16x16x32_fp8 -> mfma_f32_16x16x32_fp8_fp8,
  mfma_i32_16x16x32i8 -> mfma_i32_16x16x32_i8. The signature is
  (result_type, operands), not (a, b, acc). The atom form now leads.
  prefetch-data-load repeated the phantom name four times, under an
  "API note" this branch added that called the examples current --
  converted to fx.gemm. gemm-optimization's table had the i8 name too.
- fx.rocdl.advance_tdm_atom does not exist. cdna5.py documents the
  K-loop bump as the imm_offset atom-state kwarg, which only fx.copy
  forwards (primitive.py:1109 set_value; copy_atom_call takes no
  **kwargs), so the TDM example needed both changes to be coherent.
- CLAUDE.md's target-neutral rule listed a `vector` module that does not
  exist and omitted `enum`, which does and is star-imported.

Bad advice this branch introduced:

- The gfx1250 fallback I added told readers to use a raw positional
  s_waitcnt bitfield. universal.py passes it through unvalidated, but
  the layout is arch-specific (cdna3 packs lgkmcnt<<8 | expcnt<<4,
  rdna3 packs vmcnt<<10 | lgkmcnt<<4 | expcnt), so a literal does not
  mean the same wait everywhere -- and the dispatcher refuses gfx1250
  precisely because its encoding is unknown. The block also contradicted
  itself three lines later. Now points at the split counters, which is
  what every in-tree gfx1250 kernel actually uses.
- That same comment called the split counters "gfx1250 / RDNA4".
  "gfx1250".startswith("gfx120") is False and the two dispatch
  differently; no longer conflated.
- "PYTHONPATH=./ python3 my_kernel.py" only works under pip install -e.
  Restored the build-fly/python_packages + LD_LIBRARY_PATH form from
  CLAUDE.md:102.

Half-applied edits from earlier commits on this branch:

- kernel-trace-analysis: I replaced "Write hotspot_analyzer.py (see
  above)" in Mode A but left the identical line in Step 4.
- format-code: I retitled "Check only" to cover the C++ job but left
  only the Python command inside it.
- CLAUDE.md: I corrected the expr/rocdl module list at the prose rule
  but not in the layout tree 147 lines above; the two now match, and
  flydsl-kernel-authoring is aligned with both.
- The register-budget cross-reference wrapped so the hyphen led a line,
  rendering as "the register -budget rules".

Also: the Abs recipe was "fixed" earlier by adding a Vec() wrap, but
fx.memref_load_vec already returns a Vector and fx.absf / abs() exist
(math.py:150, utils/arith.py:422), so the whole negate/compare/select
workaround is obsolete.

Flagged, not resolved: gemm-optimization 10.2 teaches a separate
256+256 VGPR model and says the files "do not compete", while
prefetch-data-load and lds-optimization teach a combined 512-entry
budget and call the separate model gfx908/CDNA1-only. They disagree on
wave counts and on whether a prefetch buffer is safe. Marked inline as
unresolved rather than picking a winner on a hardware claim that cannot
be measured from here.
@jhinpan

jhinpan commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

Second review pass (6bff1e67)

A second review over the whole branch found 15 more issues, all verified against the
source tree
. Roughly two-thirds were defects this branch introduced, including in the
commit that fixed the first review — so the earlier "fixed" state was not clean.

Methodology correction first: an earlier runtime check in this branch was run against
the installed wheel in /opt/venv, which is stale — it still ships expr/vector.py and
expr/buffer_ops.py, both long gone from the source tree. Everything in this pass is
verified against python/flydsl/ directly.

Symbols that never existed

  • All three names in flydsl-kernel-authoring's MFMA reference were wrong:
    mfma_f32_16x16x16_f16mfma_f32_16x16x16f16, mfma_f32_16x16x32_fp8
    mfma_f32_16x16x32_fp8_fp8, mfma_i32_16x16x32i8mfma_i32_16x16x32_i8. The
    signature is (result_type, operands), not (a, b, acc). The atom form now leads.
    The phantom name appeared four more times in prefetch-data-load — underneath an "API
    note" this branch added that described the examples as current
    , which blessed
    unrunnable code. Converted to fx.gemm.
  • fx.rocdl.advance_tdm_atom exists nowhere. cdna5.py documents the K-loop bump as the
    imm_offset atom-state kwarg — and only fx.copy forwards kwargs
    (primitive.py:1109 set_value; copy_atom_call takes none), so the TDM example
    needed both fixes to be coherent.
  • CLAUDE.md's target-neutral rule listed a vector module that does not exist and omitted
    enum, which does and is star-imported.

Bad advice this branch introduced

The gfx1250 fallback added in 77a7dae5 told readers to use a raw positional s_waitcnt
bitfield. universal.py passes it through unvalidated, but the layout is arch-specific —
cdna3 packs lgkmcnt<<8 | expcnt<<4, rdna3 packs vmcnt<<10 | lgkmcnt<<4 | expcnt — so
the same literal is not the same wait, and the dispatcher refuses gfx1250 precisely
because its encoding is unknown there. The block also contradicted itself three lines
later. It now points at the split counters, which is what every in-tree gfx1250 kernel
uses. The same comment conflated gfx1250 with RDNA4; "gfx1250".startswith("gfx120") is
False and they dispatch differently.

Also PYTHONPATH=./ python3 my_kernel.py only works under pip install -e .; restored
the build-fly/python_packages + LD_LIBRARY_PATH form from CLAUDE.md:102.

Half-applied edits

Four places where an earlier commit here fixed one occurrence and left its twin:
kernel-trace-analysis Step 4 kept "Write the script (see above)"; format-code's "Check
only" was retitled for C++ but held only the Python command; CLAUDE.md's layout tree kept
the old expr/rocdl list corrected 147 lines below it; and a wrapped hyphen rendered as
"the register -budget rules".

To stop relying on eyeballs for this, the branch is now checked mechanically: all 86
fx.* symbols and every rocdl.* symbol in the skills resolve against
python/flydsl/expr/, all 16 frontmatter blocks parse with only recognized keys, and
every repo path in CLAUDE.md resolves.

Flagged, deliberately not resolved

gemm-optimization §10.2 teaches a separate 256+256 VGPR model and says the files "do not
compete"; prefetch-data-load and lds-optimization teach a combined 512-entry budget
and call the separate model gfx908/CDNA1-only. They give opposite answers on wave count
and on whether a prefetch buffer is safe. Marked inline as unresolved rather than picking
a winner on a hardware claim that cannot be measured from here — same treatment as the
lds-optimization padding question. Both need an owner with hardware access.

Both were previously flagged as unresolvable from this checkout. They are
not: this node is 8x MI355X (gfx950), the architecture the claims concern.
Measured on HIP device 3 (amd-smi 1, 0000:15:00.0), held under
flock /tmp/gpu-3.lock, selected by pick-idle-gpu.py --samples 10 (0% peak
util, 0.3 GiB used).

1. VGPR occupancy: gemm-optimization 10.2 was wrong.

It taught two separate 256-entry files that "do not compete", contradicting
prefetch-data-load and lds-optimization, which teach one combined 512-entry
budget. The combined model is correct for gfx90a+.

LLVM encodes the rule directly --
llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp getTotalNumVGPRs(has90AInsts,
NumAGPR, NumVGPR) returns alignTo(NumVGPR, 4) + NumAGPR when the target has
GFX90AInsts (gfx90a, gfx942, gfx950 all do) and max(NumVGPR, NumAGPR)
otherwise, with a 512 total; occupancy is 512 / alignTo(total, granule).

Confirmed on this gfx950 with hipcc -Rpass-analysis=kernel-resource-usage
over 13 MFMA kernels of increasing accumulator count. The combined model
reproduces the compiler's reported occupancy 13/13; the separate-file model
misses 2 and under-predicts both: arch=36/accum=32 reports 7 waves (combined
7, separate 6) and arch=84/accum=80 reports 3 (combined 3, separate 2).
Spilling first appears at arch=256/accum=256 -- combined 512 exhausted -- not
when either file alone passes 256 (arch=228/accum=224 does not spill).

10.2 now carries the combined table plus the provenance.

2. LDS padding: the skill's advice was right, my doubt was wrong, but its
   formula is harmful.

The PR description questioned "pad 1 fp16 element eliminates conflicts" on
the grounds that 2 bytes is half a bank. That reasoning was wrong: the bank
index is (t * row_stride_bytes / 4) % NBANKS, so an odd number of 2-byte
elements gives a fractional-dword stride that walks all 64 banks. The
conclusion the skill drew was correct even though the "64.5 -> fractional"
wording was loose; it is now derived properly via
gcd(row_stride_dwords, NBANKS).

What is wrong is the sizing formula. `padding_elements = bank_count /
element_size_bytes` yields +32 elements for fp16 on gfx950, an 80-dword
stride, gcd(80,64)=16 -- a 16-way conflict. Measured (one wavefront, strided
column read of a [64][64+PAD] fp16 tile, best of 9 x 50 launches):

  PAD  0 -> 128 B (32.0 dw), 32-way -> 7.11 ms  (+59%)
  PAD  1 -> 130 B (32.5 dw),  1-way -> 4.61 ms  (+3%)
  PAD  2 -> 132 B (33.0 dw),  1-way -> 4.48 ms  (best)
  PAD  4 -> 136 B (34.0 dw),  2-way -> 4.62 ms  (+3%)
  PAD 32 -> 192 B (48.0 dw), 16-way -> 5.81 ms  (+30%)

So the formula the skill recommended is 26% slower than the +1 it also
recommended. Replaced with the gcd rule, the measured table, and an explicit
"more padding is not safer" warning. The benchmark's limits are stated: it is
latency-bound on one wavefront, so ratios are compressed and it cannot
separate 1-way from 2-way.

All three skills now agree on the VGPR model; the inline "unresolved" flag
added earlier is removed.
@jhinpan

jhinpan commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

Both hardware questions are now settled by measurement (865fcd1f)

I had flagged two claims as needing "an owner with hardware access". That was wrong — this
node is 8× MI355X (gfx950), the exact architecture both claims concern. Measured on HIP
device 3 (amd-smi 1, 0000:15:00.0), held under flock /tmp/gpu-3.lock, selected by
pick-idle-gpu.py --samples 10 (0% peak util, 0.3 GiB used).

1. VGPR occupancy — gemm-optimization §10.2 was wrong

It taught two separate 256-entry files that "do not compete", contradicting
prefetch-data-load and lds-optimization. The combined-512 model is correct for
gfx90a+.

LLVM encodes the rule directly — getTotalNumVGPRs(has90AInsts, NumAGPR, NumVGPR) in
llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp returns alignTo(NumVGPR, 4) + NumAGPR
when the target has GFX90AInsts (gfx90a, gfx942, gfx950 all do), and
max(NumVGPR, NumAGPR) otherwise.

Confirmed on this gfx950 with hipcc -Rpass-analysis=kernel-resource-usage across 13 MFMA
kernels of increasing accumulator count. The combined model reproduces the compiler's own
reported occupancy 13/13; the separate-file model misses two and under-predicts both:

arch accum reported occ combined separate
36 32 7 7 ✅ 6 ❌
84 80 3 3 ✅ 2 ❌

Spilling first appears at arch=256/accum=256 — combined 512 exhausted — not when either
file alone passes 256 (arch=228/accum=224 does not spill).

2. LDS padding — the skill's advice was right, my doubt was wrong, its formula is harmful

The PR description questioned "pad 1 fp16 element eliminates conflicts" because 2 bytes is
half a bank. That reasoning was wrong. The bank index is
(t * row_stride_bytes / 4) % NBANKS, so an odd number of 2-byte elements gives a
fractional-dword stride that walks all 64 banks. The skill's conclusion was correct; only
its wording was loose. Now derived properly via gcd(row_stride_dwords, NBANKS).

What is wrong is the sizing formula. padding_elements = bank_count / element_size_bytes
gives +32 elements for fp16 on gfx950 → 80-dword stride → gcd(80,64)=1616-way
conflict
. Measured (one wavefront, strided column read of a [64][64+PAD] fp16 tile,
best of 9 × 50 launches):

PAD row stride predicted measured vs best
0 128 B (32.0 dw) 32-way 7.11 ms +59%
1 130 B (32.5 dw) 1-way 4.61 ms +3%
2 132 B (33.0 dw) 1-way 4.48 ms best
4 136 B (34.0 dw) 2-way 4.62 ms +3%
32 192 B (48.0 dw) 16-way 5.81 ms +30%

The formula the skill recommended is 26% slower than the +1 it also recommended.
Replaced with the gcd rule, the measured table, and an explicit "more padding is not safer"
warning — pad 4 already costs a 2-way conflict.

Benchmark limits stated in the skill: latency-bound on a single wavefront, so absolute
ratios are compressed relative to the conflict factor, and it cannot separate 1-way from
2-way. It establishes the ordering, not an expected speedup.

All three skills now agree on the VGPR model, and the inline "unresolved" flag is removed.
Nothing in this PR is left flagged for someone else to measure.

Everything this branch fixes drifted silently for months because nothing
in CI reads CLAUDE.md or .claude/skills/ -- check_typed_arithmetic_usage.py
scans .py only. These files are executed by agents, so a stale symbol or a
path that no longer resolves is a defect with the blast radius of broken
code, but the only thing catching them was someone reading carefully.

scripts/check_docs_api.py checks four mechanical properties, statically,
without importing FlyDSL:

  1. fx.<name> / rocdl.<name> symbols resolve
  2. repo-relative paths in backticks exist
  3. skill frontmatter parses as YAML, uses only recognized keys, and its
     name matches the directory
  4. **<name>** skill cross-references point at a skill that exists

The symbol universe is what python/flydsl/expr/ defines UNION what
kernels/, tests/ and examples/ actually call. The second half is load
bearing: fx.rocdl re-exports the upstream MLIR ROCDL dialect with
import *, so ops like s_wait_dscnt have no in-tree definition and are
only recognizable because real kernels call them. Two upstream ops that
nothing in-tree calls yet are allowlisted with their provenance.

Only Python fences and prose are scanned; MLIR, TableGen, C++ and shell
blocks legitimately spell ops as `rocdl.mfma.f32...`. A genuine
placeholder opts out with a trailing `<!-- api-check: ignore -->`, used
twice: the deliberately hypothetical CDNA5 subtarget in
add-target-atom-op, and python/flydsl/_mlir/, a build artifact absent
until you build.

Run against origin/main it reports 25 problems, 23 of them defects this
branch had to find by hand across three review rounds: all three `tools:`
keys and the `note:` key, build-rocm-image's invalid YAML, all four
phantom mfma_* spellings, advance_tdm_atom, both dead
flydsl-kernel-code-cleanup references, the deleted expr/buffer_ops.py,
the stale CLAUDE.md helper paths, and the unresolvable analyzer script
paths. The remaining two are the annotated exceptions above.

It is deliberately mechanical. It cannot catch a wrong architecture
attribution, a contradictory occupancy model, or a harmful sizing
formula -- the other three classes this branch fixed still need review.
docs/ is out of scope until its own drift is cleared; --include-docs
enables it.
@jhinpan

jhinpan commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

Enforcement: scripts/check_docs_api.py (9137f26c)

Everything this PR fixes drifted silently for months because nothing in CI reads
CLAUDE.md or .claude/skills/
check_typed_arithmetic_usage.py scans .py only.
These files are executed by agents, so a stale symbol or an unresolvable path is a defect
with the blast radius of broken code, yet the only thing catching them was careful reading.

The new gate checks four mechanical properties, statically, without importing FlyDSL:

  1. fx.<name> / rocdl.<name> symbols resolve
  2. repo-relative paths in backticks exist
  3. skill frontmatter parses as YAML, uses only recognized keys, and name matches its directory
  4. **<name>** skill cross-references point at a skill that exists

The symbol universe is what python/flydsl/expr/ defines ∪ what kernels/, tests/ and
examples/ actually call
. The second half is load-bearing: fx.rocdl re-exports the
upstream MLIR ROCDL dialect with import *, so ops like s_wait_dscnt have no in-tree
definition and are recognizable only because real kernels call them. Two upstream ops
nothing calls yet are allowlisted with provenance.

Only Python fences and prose are scanned — MLIR, TableGen, C++ and shell blocks legitimately
spell ops as rocdl.mfma.f32.... Genuine placeholders opt out with a trailing
<!-- api-check: ignore -->, used exactly twice (the deliberately hypothetical CDNA5
subtarget, and python/flydsl/_mlir/, absent until you build).

It has teeth

Run against origin/main it reports 25 problems — 23 of them defects this PR had to find
by hand across three review rounds
:

CLAUDE.md:196: path does not exist: kernels/kernels_common.py
build-rocm-image/SKILL.md: frontmatter is not valid YAML (mapping values are not allowed here)
capture-kernel-trace/SKILL.md: unrecognized frontmatter key `tools` (silently ignored at load)
kernel-trace-analysis/SKILL.md: unrecognized frontmatter key `note` (silently ignored at load)
flydsl-kernel-authoring/SKILL.md:673: unknown symbol `mfma_f32_16x16x16_f16`
flydsl-kernel-authoring/SKILL.md:619: unknown symbol `advance_tdm_atom`
flydsl-kernel-authoring/SKILL.md:235: no such skill: flydsl-kernel-code-cleanup
flydsl-kernel-authoring/SKILL.md:57:  path does not exist: python/flydsl/expr/buffer_ops.py
kernel-trace-analysis/SKILL.md:31:   path does not exist: scripts/hotspot_analyzer.py
prefetch-data-load/SKILL.md:91:      unknown symbol `mfma_f32_16x16x16_f16`
...

The remaining two are the annotated exceptions.

What it deliberately cannot do

It is mechanical. It would not have caught the other three classes this PR fixes: the
wrong s_wait_* architecture attribution, the contradictory VGPR occupancy model, or the
harmful LDS padding formula. Those still need review and, as it turned out, measurement.

docs/ is out of scope until its own drift is cleared (the remaining open item);
--include-docs enables it, and should be turned on in the PR that fixes those files.

A third review pass, run as a 7-angle adversarial workflow weighted at the
commits nothing had reviewed yet (865fcd1 and 9137f26). It found defects
in both, including in the measurements 865fcd1 introduced.

gfx950 has 32 LDS banks, not 64.

This was wrong throughout lds-optimization -- 10 occurrences predate this
branch, and 865fcd1's new gcd derivation was built on the same false
premise. LLVM defines only FeatureLDSBankCount16 and FeatureLDSBankCount32
(llvm/lib/Target/AMDGPU/AMDGPU.td:397); FeatureISAVersion9_4_Common carries
LDSBankCount32 and gfx950 inherits it.

Confirmed on this MI355X with a discriminating probe (HIP device 3 under
flock /tmp/gpu-3.lock): a 64-lane wavefront reading one dword per lane takes
12.70 ms at a 32-dword row stride and 12.75 ms at 64 dwords -- 0.4% apart,
both ~1.7x a conflict-free stride-1 baseline of 7.41 ms. A 64-bank part
would make the 64-dword stride about twice as slow as the 32-dword one.

Corrected: the bank count and index formula, the throughput and latency
figures derived from it, the gfx942-vs-gfx950 comparison table (the bank
count is unchanged between them, so swizzle masks carry over), the
conflicting-stride worked example, the XOR mask formula, and the gcd table.
The measured padding table is unaffected -- its predictions recompute
identically under 32 banks -- but now states which row width it used, since
it differs from the worked example above it.

Occupancy thresholds were miscomputed.

865fcd1's table said `<= 170 -> 3 waves`; the granule is 8, so 170 rounds
to 176 and yields 2. It also capped the table at 4 waves when the per-SIMD
maximum is 8. Both constants are in LLVM for GFX90AInsts targets:
getMaxWavesPerEU returns 8, getVGPRAllocGranule returns 8. The table now
runs from <= 64 -> 8 and corrects 170 to 168, and states the formula as
min(8, 512 / alignTo(total, 8)).

prefetch-data-load claimed spills start once arch_vgpr + accum_vgpr passes
512. Each file saturates at 256, so a report can never show a combined
figure above 512; demand beyond it surfaces as arch=256, accum=256 with a
non-zero spill count. Reworded to describe what is actually observable.

build-flydsl hardcoded /FlyDSL in nine places, including the PYTHONPATH and
LD_LIBRARY_PATH exports, which contradicts both CLAUDE.md's ${PWD}-relative
block and the environment de-drift in cf8b47c. Parameterised on
$FLYDSL_ROOT, defaulting to $PWD, with the container's /FlyDSL noted.

The gate added in 9137f26 had six confirmed bugs of its own -- three
false positives that would block valid PRs and three misses:

- _split_frontmatter split on the "---" substring rather than a delimiter
  line, so a value containing --- made a valid file report as invalid YAML
- fx./rocdl. matched inside URLs, paths and filenames, turning file
  extensions into missing symbols
- a `path.py:419` citation was reported as a nonexistent path
- a ```python fence with attributes, or tagged python3, was skipped
  wholesale, hiding any drift inside it
- ./-prefixed repo paths skipped the existence check
- --include-docs globbed only the top level of docs/, reaching 14 of 23 files

tests/unit/test_docs_api_guard.py pins all six: 8 of its 14 cases fail
against the pre-fix script and all 14 pass now.

Still outstanding, and pre-existing rather than introduced here: several
API errors in flydsl-kernel-authoring that the gate cannot catch because it
only checks that a name exists, not how it is called -- gpu.ShuffleOp,
T.i4, arith.cmpf argument order, make_tile taking a list, and the claim
that fx.rocdl.MFMA is arch-dispatched.
33fdb03 rewrote lds-optimization to say gfx950 has 32 LDS banks. That was
wrong and it damaged text that had been correct since before this branch.
The AMD CDNA4 ISA documentation states 64 banks and 256 bytes/clock LDS read
bandwidth, against 32 banks and 128 bytes/clock for CDNA3. This reverts the
file to its 865fcd1 state, keeping only the row-width annotation.

Both premises behind the change were bad.

LLVM's LDSBankCount is not a statement about the physical bank count. There
is no FeatureLDSBankCount64 -- only 16 and 32 exist -- the value defaults to
32 when unset (GCNSubtarget.cpp:173), and the only two consumers,
AMDGPUISelDAGToDAG.cpp:3207 and AMDGPUInstructionSelector.cpp:1097, merely
test `!= 16` to pick a lowering for ancient 16-bank parts. Reading gfx950's
inherited FeatureLDSBankCount32 as "this chip has 32 banks" was invalid.

The supporting measurement did not measure anything. It compared a 32-dword
against a 64-dword row stride and found them 0.4% apart, but its
conflict-free baseline was only 1.7x faster than the fully-conflicting case,
so the kernel was latency-bound rather than LDS-bound and could not resolve
a 2x conflict difference. Two further probes were also invalid: one had a
dependent accumulate chain, and one used loop-invariant addresses that were
hoisted out of the timed loop entirely, which is why its conflict-free case
came out slowest of all. No valid bank-count measurement was produced, so no
measured claim about bank count is made here.

Both tables in the restored file are self-consistent under 64 banks: the
worked example (128-element fp16 rows) gives 64/1/1/2/16-way and the
measured padding table (64-element rows) gives 32/1/1/2/16-way, each
matching 64 / gcd(row_stride_dwords, 64). The two use different row widths,
which is what made them look contradictory; that is now stated inline.

The rest of 33fdb03 is unaffected and stands: the occupancy threshold
correction, the spill wording, the build-flydsl path parameterisation, and
the six fixes to scripts/check_docs_api.py.
sjfeng1999
sjfeng1999 previously approved these changes Aug 19, 2026
Comment thread .github/workflows/pre-checks.yaml Outdated
scripts/check_docs_api.py only proves a name exists, not that the call
around it is valid, so these survived it. All four were surfaced by the
third review pass and re-verified here against python/flydsl directly.
They predate this branch but sit in files it already rewrites.

- flydsl-kernel-authoring's warp reduction called gpu.ShuffleOp. That is
  the raw MLIR op; expr/gpu.py binds the dialect as the local name `gpu`
  and exports wrappers instead, so `gpu.ShuffleOp` on flydsl.expr.gpu is
  an AttributeError. Uses gpu.shuffle_xor(value, offset, width) (gpu.py:89).
- The i4 WMMA example passed T.i4. class Types defines i8/i16/i32/i64/i128
  and no i4 (typing.py:289-418). The DSL type is fx.Int4 (numeric.py:814,
  re-exported through typing.py), matching how the fp8 line one row above
  already spells fx.Float8E4M3FN.
- The Arith Ops table documented arith.cmpf(a, b, pred). The predicate is
  the first parameter: def cmpf(predicate, lhs, rhs, **kwargs)
  (arith.py:81), same as cmpi at :66.
- flydsl-tile-programming's Pattern B built its tile with
  fx.make_tile([layout, layout]). make_tile is varargs and its _resolve
  accepts int, None, tuple or Layout, raising ValueError on a list
  (primitive.py:1385). Unpacked to positional arguments.

Also tightened wording I introduced in 6bff1e6 and that
kernel-code-cleanup carried: calling the atom "arch-dispatched (MFMA on
CDNA3/4, WMMA on gfx11/gfx1250)" reads as though fx.rocdl.MFMA selects by
target. It does not -- universal.py:155 unconditionally returns
MmaOpCDNA3_MFMAType. The caller picks the family; MFMA covers CDNA3 and
CDNA4, cdna4.MFMA_Scale covers CDNA4 scaled, WMMA covers gfx11/gfx1250.

This closes the last category the third review pass left open. The gap it
exposes is structural: a signature-aware check would need to parse each
`def` and compare argument counts, order and keyword names against the
call as written in the prose. That is a larger change and is not attempted
here.
jhinpan and others added 2 commits August 19, 2026 19:25
Addresses @sjfeng1999's review note that the workflow should not keep
accumulating checks. It should not, and the step this PR added made it
worse in two further ways it did not point out.

scripts/check_repo.py is now the single entry point. The python-style job
drops from two ad-hoc check steps to one, and stays at one however many
checks are registered; base/head resolution is written once instead of
copy-pasted per step; and `python3 scripts/check_repo.py` reproduces all
of them locally, which previously required knowing each script by name.
`--list` enumerates them, `--only <name>` runs one, and every check runs
even after an earlier failure so one invocation surfaces everything.

The black/ruff and clang-format gates stay on their own wrappers: those
are wired to reviewdog for inline PR annotations, which does not belong
behind a dispatcher.

Two problems with the step this PR added, both real:

- It scanned the whole tree while both of its neighbours are diff-scoped
  via BASE_SHA/HEAD_SHA. Any pre-existing problem anywhere in the repo
  would have blocked every unrelated PR. check_docs_api.py now takes
  --base/--head and honours the same env vars.

  Scoping a cross-reference check needs one exception: renaming a symbol
  under python/flydsl/ can invalidate any document, not only the ones the
  commit touched. So a change under python/flydsl/ widens the scan back to
  every target, and only a docs-only change narrows it. The run prints
  which scope it used, so a narrowed run is never mistaken for a clean
  whole-tree one.

- It sat in a job named "Check Python Code Style" while checking neither
  Python nor style. The job name is left alone -- renaming a check that
  branch protection may require is not worth it here -- but the step is
  now "Repository pre-checks", which describes what it does.

CLAUDE.md documents the dispatcher as the thing to run and to register
new checks in.
@jhinpan

jhinpan commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-ci

@coderfeli
coderfeli merged commit d3f3ce1 into main Aug 24, 2026
12 checks passed
@coderfeli
coderfeli deleted the skills/dedrift-and-fix-frontmatter branch August 24, 2026 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants