runcontainer: count or test before allocating in run-bitmap intersections

[tamirms]

Description

Three run-container paths build an 8 KiB bitmap container before they know whether the result is empty, an array, or a bitmap. This counts or tests first, then allocates only what is returned.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Test improvements
• Build/CI changes

Changes Made

What was changed?

• runContainer16.andBitmapContainer counts the intersection with the existing andBitmapContainerCardinality, then builds an empty array container, an exact-size array, or a bitmap, whichever the count calls for.
• bitmapContainer.iandRun16 counts the same way, returns a small result as an array built from the run's intervals, and otherwise clears the gaps between intervals in place.
• runContainer16.intersects walks the run's intervals against the other container and returns at the first shared value, rather than building a container to answer a boolean.
• Added clearBitmapGaps and containerFromWords for the shared pieces, and runbitmap_and_test.go with tests and BenchmarkRunAndBitmap.

Why was it changed?

• andBitmapContainer converted the run to a bitmap container and ANDed, so an empty or array-sized result paid for a bitmap it never returned. iandRun16 did the same conversion on the argument side.
• An empty result is the common case when a run meets a sparse bitmap.
• intersects computed rc.and(a) and tested the result for emptiness, allocating a container to answer a boolean. Every mixed pairing with a run reaches it through the array and bitmap dispatchers, so this sits on the path of every Intersects call that touches a run.

How was it changed?

• The result is sized by its cardinality before it is built, the same shape as bitmapContainer.andBitmap, which already does this.
• A run with more than 64 intervals is ORed into a scratch bitmap on the stack, counted there, and extracted in one pass. Past that count iandRun16 also builds the run in stack scratch and ANDs in one flat pass, since clearing gap by gap costs a call per interval.
• intersects uses a masked word scan for bitmaps, a galloping merge for arrays or a probe per value when the array is much smaller than the interval list, and an interval overlap test for runs.
• No API change.

Testing

runbitmap_and_test.go checks each path against its old definition, newBitmapContainerFromRun(rc).andBitmap(bc) and !rc.and(c).isEmpty(), over random container pairs and the edges: empty run, full run, intervals on word boundaries, results at the array and bitmap threshold, runs with hundreds of intervals, and a few values against many intervals. It also asserts that the empty and boolean paths allocate no container.

go test ./...
go test -tags appengine ./...
GOOS=linux GOARCH=386 go build ./...    # also arm, arm64, ppc64

All pass on linux/amd64 and darwin/arm64.

Formatting

gofmt -s -l . reports nothing. go vet ./... reports only the pre-existing ReadFrom signature complaint on master.

Fuzzing

The step above cannot be run as written: there is no FuzzSmat target on master. smat.go refers to a smat_fuzz_test.go that is not in the repository, so go test -fuzz=FuzzSmat matches nothing and simply runs the ordinary tests. What does run is the smat corpus, which passes:

go test -tags=gofuzz -run 'TestGenerateSmatCorpus|TestSmatHits'
ok  github.com/RoaringBitmap/roaring/v2

Performance Impact

BenchmarkRunAndBitmap is added: And, in-place And and Intersects between a run-optimized bitmap and a dense one over a single key. Xeon 8375C.

                                        before                          after
  And, empty result                     3.0 µs   8328 B/op  4 allocs    594 ns   104 B/op  2 allocs
  And, array result                     8.3 µs  14504 B/op  8 allocs    4.6 µs  6280 B/op  6 allocs
  And, bitmap result                    4.6 µs  16560 B/op  8 allocs    3.6 µs  8336 B/op  6 allocs
  And, 1024-interval run, array        11.9 µs  12456 B/op  8 allocs   10.0 µs  4232 B/op  6 allocs
  And, 1024-interval run, bitmap        9.5 µs  16560 B/op  8 allocs    7.6 µs  8336 B/op  6 allocs
  in-place And, bitmap result           2.4 µs   8224 B/op  2 allocs    290 ns     0 B/op  0 allocs
  in-place And, 1024-interval run       7.3 µs   8224 B/op  2 allocs    4.6 µs     0 B/op  0 allocs
  Intersects, disjoint                  3.0 µs   8248 B/op  3 allocs    121 ns     0 B/op  0 allocs
  Intersects, overlapping               8.3 µs  14392 B/op  4 allocs      8 ns     0 B/op  0 allocs
  Intersects, few values, 1024 intervals 1.3 µs    32 B/op  2 allocs     62 ns     0 B/op  0 allocs

Performance Analysis

• The empty result, the common case when a run meets a sparse bitmap, drops from 3.0 µs and an 8 KiB allocation to 594 ns and 104 bytes.
• The boolean paths no longer allocate at all.
• Memory falls on every row, since nothing larger than the returned container is built.
• BenchmarkIntersection*, BenchmarkUnion* and BenchmarkAndAny/* do not reach these paths and are unchanged.

Breaking Changes

None. No API change.

Related Issues

None.

Additional Notes

bitmapContainer.iandArray has the same conversion for arrays and is left for a separate change.

🤖 Generated with Claude Code