Conversation
llama.cpp defines five GGML IQ formats at one and two bits; we ship two. This adds IQ2_XXS, at 2.0625 bits per weight between IQ1_S and IQ2_XS, and is the first of three changes that close the gap. On a real mixed-precision checkpoint (unsloth/Qwen3.8-27B-GGUF) IQ2_XXS alone covers 59 tensors and 2.84B parameters, 10.6% of the file, which a reader limited to IQ1_S and IQ2_XS cannot consume. The encoder follows the existing single-pass grid search at a fixed anchored super-block scale, and the CUDA kernel follows the existing per-block structure. IQ2_XXS reuses IQ2_XS's even-parity sign rule but packs a 4-bit sub-block scale into the same 32-bit word as four 7-bit sign indices, and its 256-entry grid needs no high index bits. Two pieces of groundwork come with it, both of which the next two formats reuse. Export spelled the IQ family as a two-element tuple at nine sites; those become an IQ_FORMATS frozenset plus per-format packer and block-geometry tables, so a format is a row rather than a sweep through the exporters. And the per-format test files, which had drifted apart, become one parametrized module per layer, so every format is held to the same contract. The decoder is validated against llama.cpp's own output over 11,100,160 blocks from that checkpoint, all bit-identical to dequantize_row_iq2_xxs, and the new codebook matches the ggml-common.h table entry for entry. Blocks lifted from the checkpoint ship as conformance vectors so CI keeps checking bytes we did not produce. Measured on a 5632x2048 weight, the CUDA encoder runs at 1047.9 M elem/s against the torch search's 10.7. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Second of three changes completing the GGML IQ formats at one and two bits, after IQ2_XXS. IQ2_S is the widest of the family at 2.5625 bits per weight, and covers 9 tensors and 0.6B parameters of the mixed-precision checkpoint the first change measured. It is the one format llama.cpp's own tooling gives no head start on, so both the search and the kernel are written against the GGML layout directly. The interesting difference from IQ2_XS and IQ2_XXS is the sign handling: IQ2_S stores a full eight-bit sign mask per group rather than a seven-bit parity-coded index, so the encoder takes the input signs as they are instead of flipping the weakest element to fix parity, and the search compares magnitudes directly. Its 1024-entry codebook is twice IQ2_XS's, which makes it the most expensive search of the five and pushes the grid past the static shared memory limit, so the kernel keeps the codebook and its norms in dynamic shared memory. On a 5632x2048 weight that is 725.7 M elem/s against the torch search's 0.8 -- without the kernel, a 27B model would take about ten hours to pack. The decoder is validated against llama.cpp's own output over 2,355,200 blocks from the same checkpoint, all bit-identical to dequantize_row_iq2_s, and the new codebook matches the ggml-common.h table entry for entry. The format joins the shared parametrized test batteries and the IQ_FORMATS registry introduced with IQ2_XXS, so it inherits the whole contract rather than bringing its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Contributor
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: NVIDIA/Model-Optimizer/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (28)
Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review. 📝 WalkthroughWalkthroughThe change adds IQ2_XXS and IQ2_S GGML quantization with Python and CUDA packers. It integrates both formats into fake quantization, Hugging Face and Megatron export, and PTQ recipes. Tests cover format conformance, CUDA parity, and recipe integration. ChangesGGML IQ2 Format Support
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~60 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant TensorQuantizer
participant ggml_fake_quant
participant quantize_iq2_s
participant GGML_CUDA_extension
participant iq2_s_pack_cuda
TensorQuantizer->>ggml_fake_quant: dispatches iq2_s fake quantization
ggml_fake_quant->>quantize_iq2_s: calls format quantizer
quantize_iq2_s->>GGML_CUDA_extension: sends blocks, grid, and scales when CUDA extension is available
GGML_CUDA_extension->>iq2_s_pack_cuda: dispatches IQ2_S packing
iq2_s_pack_cuda-->>quantize_iq2_s: returns packed payload
Merge Risk: ⚪ Minimal · up to No material merge risk remains in the supplied review context; the new formats are wired through export and have conformance and integration coverage. 🚥 Pre-merge checks | ✅ 5 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 64.47% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 76 functions across 19 files. (9 skipped: 9 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
This was referenced Sep 22, 2026
Contributor
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2512 +/- ##
==========================================
+ Coverage 71.20% 78.48% +7.27%
==========================================
Files 603 605 +2
Lines 66796 67061 +265
==========================================
+ Hits 47564 52634 +5070
+ Misses 19232 14427 -4805
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This was referenced Sep 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Type of change: new feature
Adds IQ2_S at 2.5625 bits per weight — the widest of the GGML IQ family at one and two bits. Second of three, stacked on #2511 (IQ2_XXS).
On the mixed-precision checkpoint #2511 measured (
unsloth/Qwen3.8-27B-GGUF), IQ2_S covers 9 tensors and 0.6 B parameters.What's distinctive about it
IQ2_S is the one format llama.cpp's own tooling gives no head start on, so both the search and the kernel are written against the GGML layout directly.
The interesting difference from IQ2_XS and IQ2_XXS is sign handling: IQ2_S stores a full 8-bit sign mask per group rather than a 7-bit parity-coded index. So the encoder takes the input signs as they are, instead of flipping the weakest element to fix parity, and the search compares magnitudes directly — simpler than its siblings.
Its 1024-entry codebook is twice IQ2_XS's, which makes it the most expensive search of the five and pushes the grid past the static shared-memory limit, so the kernel keeps the codebook and its norms in dynamic shared memory.
That cost is why the kernel matters more here than anywhere else:
Without the kernel this format would not be usable on the models these formats exist for.
Usage
Testing
The decoder is validated against llama.cpp's own output, not just round-tripped:
The new 1024-entry codebook matches the
ggml-common.htable entry for entry. Blocks lifted from that checkpoint ship as conformance vectors so CI keeps checking bytes we did not produce.The format joins the
IQ_FORMATSregistry and the shared parametrized test batteries introduced in #2511, so it inherits the whole contract rather than bringing its own set.tests/unit/torch/quantization/ -k 'ggml or iq1 or iq2 or iq_'— 131 passedtests/gpu/torch/quantization/test_iq_formats_cuda.py— 28 passed (7 checks × 4 formats)tests/unit/recipe/test_presets.py— passing;general/ptqnow holds 30 recipes,ptq.mdupdatedPre-existing failures in
test_autoquant.pyare atorchvisioncircular import in my environment — 71 failed / 39 passed identically with and without this change.Before your PR is "Ready for review"
CONTRIBUTING.md: ✅ — the new codebook is a GGML table, carried incodebooks.pywith the source revision recorded. No new dependencies.Additional Information
Second of three: #2511 (IQ2_XXS) → this → IQ1_M. Replaces #2505, which carried all three at once.
🤖 Generated with Claude Code
Summary by CodeRabbit