Skip to content

Batch no-run container reads during decoding - #571

Open
perfloop-agent wants to merge 2 commits into
RoaringBitmap:masterfrom
perfloop:perfloop-pr-open-58myy4vkgz
Open

perfloop-agent wants to merge 2 commits into
RoaringBitmap:masterfrom
perfloop:perfloop-pr-open-58myy4vkgz

Conversation

@perfloop-agent

Copy link
Copy Markdown
Contributor

Description

Batch no-run container payload reads during decoding from ordinary io.Reader values. The decoder now coalesces exact whole-container payloads without consuming bytes after the bitmap. It also keeps bitmap payloads aligned when an array payload precedes it in a shared read.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • Performance improvement
  • Test improvements

Changes Made

What was changed?

  • Batch no-run payloads in roaringArray.readFrom using cardinalities from the descriptive header, with a bounded group size and exact container slices.
  • Keep direct ByteInput, run-container, and copy-on-write paths on their existing behavior.
  • End a group before a bitmap when its current byte offset is not 8-byte aligned, and add regression coverage for an array container followed by a bitmap container.
  • Add counting-reader benchmark coverage for sparse files and short reads.

Why was it changed?

  • Avoid one small reader operation per sparse no-run container for ordinary reader inputs.
  • Preserve the caller's stream boundary and the alignment required by bitmap containers backed by []uint64.

How was it changed?

  • Calculate each payload length from the key/cardinality header and read groups only at serialized container boundaries.
  • Install bounded sub-slices for each container and start bitmap payloads in aligned groups.

Testing

  • go test ./...
  • go test -tags appengine
  • go tool unconvert ./...
  • GOARCH=386 go build
  • GOARCH=arm go build
  • GOARCH=arm64 go build

Checks: 8 passed.

Formatting

  • go fmt ./...

Fuzzing

  • go test -tags=gofuzz -run=TestGenerateSmatCorpus
  • go test -run='^$' -fuzz=FuzzSmat -fuzztime=300s -timeout=60s

Performance Impact

The sparse-reader benchmark covers unbuffered file and short-read reader cases. The existing BenchmarkUnserializeReadFrom row remains a guard for the ordinary bytes.Reader decode path.

Workload: Bitmap.ReadFrom of a no-run sparse bitmap with 1024 singleton containers from an unbuffered file

Metric Before After Change
read-calls/op 1028 5 99.5% lower

Workload: Bitmap.ReadFrom of a no-run sparse bitmap with 4096 singleton containers from an unbuffered file

Metric Before After Change
read-calls/op 4100 5 99.9% lower

Workload: Bitmap.ReadFrom of a no-run sparse bitmap with 16384 singleton containers from an unbuffered file

Metric Before After Change
read-calls/op 16388 5 100% lower

Workload: Bitmap.ReadFrom of a no-run sparse bitmap with 1024 singleton containers through a 257-byte short-read file reader

Metric Before After Change
read-calls/op 1058 42 96% lower

Workload: Bitmap.ReadFrom of a no-run sparse bitmap with 4096 singleton containers through a 257-byte short-read file reader

Metric Before After Change
read-calls/op 4226 162 96.2% lower

Workload: Bitmap.ReadFrom of a no-run sparse bitmap with 16384 singleton containers through a 257-byte short-read file reader

Metric Before After Change
read-calls/op 16898 642 96.2% lower

Breaking Changes

None.

Related Issues

None.

Additional Notes

No public API changes.


Generated by Perfloop. Human sponsor: Tomás Senart. Measurements and checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant