Skip to content

Reject short output in LZ4DecompressorWithLength safe paths - #126

Merged
yawkat merged 2 commits into
mainfrom
fix/issue-105
Sep 25, 2026
Merged

yawkat merged 2 commits into
mainfrom
fix/issue-105

Conversation

@yawkat

@yawkat yawkat commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Fixes #105.

The safe-decompressor overloads of LZ4DecompressorWithLength passed the declared length as maxDestLen but never checked that that many bytes were produced. As a result, a length prefix larger than the actual data had two effects:

  • The allocating overload silently returned a truncated array.
  • The non-allocating overloads returned a short count, which a caller relying on getDecompressedLength() could miss.

All safe paths now throw LZ4Exception when the decompressed size doesn't match the prefix. The fast paths already required an exact length.

  • The allocating overload now allocates destLen itself and checks the count.
  • decompress(ByteBuffer, ByteBuffer) checks before it updates any positions.

Javadoc is updated to match.

Behaviour change: only malformed input, where the prefix doesn't match the data, is affected. LZ4CompressorWithLength never produces such input.

Test: testDecompressorWithLengthRejectsShortOutput covers all seven safe overloads with heap and direct buffers. It also checks that buffer positions are unchanged after a failure. It fails without the fix. LZ4Test and OutOfBoundsTest pass.

🤖 Generated with Claude Code

The safe-decompressor overloads passed the declared length as the
maximum destination length but accepted any shorter output, returning a
truncated array or a short written count. Throw LZ4Exception when the
decompressed size differs from the length prefix, matching the fast
paths. decompress(ByteBuffer, ByteBuffer) checks before moving the
buffer positions.

Fixes #105

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yawkat

yawkat commented Sep 25, 2026

Copy link
Copy Markdown
Owner Author

potentially breaking?

@yawkat yawkat modified the milestones: 1.11.4, 1.12.0 Sep 25, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yawkat
yawkat enabled auto-merge (squash) September 25, 2026 19:03
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.81%. Comparing base (f6e3304) to head (d19a787).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff              @@
##               main     #126      +/-   ##
============================================
+ Coverage     79.76%   79.81%   +0.05%     
- Complexity      562      564       +2     
============================================
  Files            42       42              
  Lines          1838     1843       +5     
  Branches        247      247              
============================================
+ Hits           1466     1471       +5     
+ Misses          239      238       -1     
- Partials        133      134       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@yawkat
yawkat merged commit 681f503 into main Sep 25, 2026
3 checks passed
@yawkat
yawkat deleted the fix/issue-105 branch September 25, 2026 19:09
dongjoon-hyun added a commit to apache/spark that referenced this pull request Sep 28, 2026
### What changes were proposed in this pull request?

This PR aims to upgrade `at.yawk.lz4:lz4-java` to 1.12.0.

### Why are the changes needed?

To bring the latest security fixes of `v1.11.4` and the stricter input validation of `v1.12.0`. The upstream recommends `1.12.0` over `1.11.4`.

- https://github.com/yawkat/lz4-java/releases/tag/v1.12.0 (2026-09-25)
  - [Throw IOException for invalid or unsupported frame descriptors](yawkat/lz4-java#133)
  - [Reject short output in LZ4DecompressorWithLength safe paths](yawkat/lz4-java#126)
  - [Make stream failures sticky in LZ4FrameInputStream and LZ4BlockInputStream](yawkat/lz4-java#145)
- https://github.com/yawkat/lz4-java/releases/tag/v1.11.4 (2026-09-25)
  - [LZ4FrameInputStream reallocates block buffers for every frame, allowing CPU and GC amplification from small inputs](GHSA-gm45-99xc-r7wv)
  - [LZ4BlockInputStream with stopOnEmptyBlock=false recurses once per empty block, causing StackOverflowError](GHSA-343h-94h5-c4wr)
  - [Native library extraction to a shared temporary directory is vulnerable to file replacement by another local user](GHSA-mcr4-qmvw-px4g)

Note that Apache Spark's `LZ4CompressionCodec` reads streams via `LZ4BlockInputStream` with `withStopOnEmptyBlock(false)`, which is the code path fixed by `GHSA-343h-94h5-c4wr`.

**Full Changelog**: yawkat/lz4-java@v1.11.3...v1.12.0

### Does this PR introduce _any_ user-facing change?

No. There is no behavior change for valid LZ4 streams.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

Closes #59093 from dongjoon-hyun/SPARK-59820.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LZ4DecompressorWithLength safe paths accept output shorter than the declared length

2 participants