fastaggregation: count before allocating in FastAnd

[tamirms]

Description

FastAnd with three or more inputs intersects pairwise and materializes an intermediate per input per key. This counts each key's result before allocating it.

This depends on #566, and that is why the first of the two commits here is #566's. This change needs clearBitmapGaps, containerFromWords and the allocation-free run intersects that #566 introduces, and GitHub cannot base a pull request on a branch that lives in a fork. Only the second commit, fastaggregation: count before allocating in FastAnd, is under review here. The diff narrows to that commit alone as soon as #566 merges.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Test improvements
• Build/CI changes

Changes Made

What was changed?

• With three or more inputs, FastAnd drives from the input with the fewest containers and walks its keys once, in order, against every other input with advanceUntil. A key every input has is probed with a container intersects test before its containers are intersected. Two inputs keep the existing And path.
• andK intersects one key's containers. Bitmaps are ANDed into stack scratch with the fused andCardSlice, stopping at the first empty prefix, over the words the runs leave. The result is copied out as a bitmap, appended as an exact-size array, or dropped.
• A key with an array container, or of run containers only, goes through the package's own kernels instead: a full run is the identity and drops out, the smallest array leads and the rest keep the caller's order, the first pair into a fresh container and the rest in place.
• bitmapContainer.andArray reloaded three slice headers per element; hoisting them makes it a quarter faster on both amd64 and arm64, which And(bitmap, array) gets as well.
• Added BenchmarkFastAndShapes and BenchmarkRealDataFastAnd, the latter next to BenchmarkRealDataFastOr.

Why was it changed?

• A key whose result is empty still allocated up to 8 KiB per step, and a run container among bitmaps fell back to the same pairwise path.
• For dense inputs whose intersection is empty, the old path does all of the work and all of the allocation for nothing. The same applies to every input after the first two on a non-empty key.

How was it changed?

• A key with nothing in common costs no allocation, and the call allocates nothing but the answer.
• A run narrows the word span to its extent, and only a run with gaps is folded into a mask afterwards, so a range built with AddRange costs no intersection and a result the bitmaps already rule out never walks an interval.
• Nothing bigger than the first intersection result is ever built on the array and run paths.

Testing

fastand_kway_test.go compares FastAnd with pairwise And over 400 random cases of 3 to 6 inputs mixing array, bitmap and run containers with keys that interleave and go missing. It checks that results share no storage with the inputs, covers the run-mask edges (whole key, short and word-straddling intervals, a thousand short intervals, runs folding to a narrower mask, one bitmap under two runs, two ranges that overlap only where the bitmap is empty, runs only, disjoint key sets), and asserts that an empty intersection allocates nothing but the answer.

Results are checked with Validate and Equals, and the array and bitmap sizing rule holds for every container.

go test ./...
go test -tags appengine ./...
GOOS=linux GOARCH=386 go build ./...    # also arm, arm64, ppc64

All pass on linux/amd64 and darwin/arm64.

Formatting

gofmt -s -l . reports nothing. go vet ./... reports only the pre-existing ReadFrom signature complaint on master.

Fuzzing

The step above cannot be run as written: there is no FuzzSmat target on master. smat.go refers to a smat_fuzz_test.go that is not in the repository, so go test -fuzz=FuzzSmat matches nothing and simply runs the ordinary tests. What does run is the smat corpus, which passes:

go test -tags=gofuzz -run 'TestGenerateSmatCorpus|TestSmatHits'
ok  github.com/RoaringBitmap/roaring/v2

Performance Impact

Xeon 8375C.

BenchmarkFastAndShapes is added: each kernel path against the pairwise chain on the same inputs, one key unless noted.

                                pairwise                       FastAnd
  array30-bitmap       231 ns    200 B   6 allocs     269 ns    200 B   6 allocs
  array4000-bitmap     6.6 µs   8328 B   6 allocs     6.1 µs   8328 B   6 allocs
  array30-runs         938 ns    288 B   8 allocs     928 ns    200 B   6 allocs
  array4000-runs       5.7 µs  16544 B   8 allocs     3.7 µs   8328 B   6 allocs
  bitmap-bitmap        2.5 µs   8336 B   6 allocs     2.6 µs   8336 B   6 allocs
  runs-bitmap          5.8 µs   6304 B   8 allocs     5.0 µs   4232 B   6 allocs
  runs-runs            8.3 µs  10400 B   8 allocs     7.5 µs   8328 B   6 allocs
  jointly-empty       12.5 µs  33200 B  18 allocs     1.9 µs     80 B   1 allocs
  eight-keys           1.4 µs   1056 B  24 allocs     1.5 µs   1056 B  24 allocs

BenchmarkRealDataFastAnd is added, before and after:

  census-income            1.3 µs    336 B  12 allocs  ->    604 ns   1872 B   2 allocs
  census-income_srt        2.3 µs   1400 B  20 allocs  ->    670 ns   1872 B   2 allocs
  census1881               595 ns     80 B   1 allocs  ->    572 ns   1872 B   2 allocs
  census1881_srt           599 ns     80 B   1 allocs  ->    625 ns   1872 B   2 allocs
  dimension_003           50.3 µs     80 B   1 allocs  ->   46.2 µs 131152 B   2 allocs
  dimension_008           15.2 µs    520 B  32 allocs  ->   13.7 µs  49232 B   2 allocs
  dimension_033           24.8 µs   2237 B  93 allocs  ->    549 ns   1488 B   2 allocs
  uscensus2000             599 ns     80 B   1 allocs  ->    616 ns   1872 B   2 allocs
  weather_sept_85         15.9 µs  15632 B  43 allocs  ->    522 ns   1872 B   2 allocs
  weather_sept_85_srt      2.7 µs   1024 B  44 allocs  ->    698 ns   1872 B   2 allocs
  wikileaks-noquotes       683 ns    128 B   3 allocs  ->    658 ns   1872 B   2 allocs
  wikileaks-noquotes_srt   659 ns    112 B   3 allocs  ->    633 ns   1872 B   2 allocs

The intersection of a whole dataset is empty, so that table measures the key narrowing and the early exit on real container shapes.

Performance Analysis

• Where the smallest input is a tiny array, the probe is a fixed cost of about 25 ns per key and input. On a key whose whole intersection takes 250 ns it shows as a 1.2x slower call.
• Two dense bitmaps under a whole-key range are within 6% of the pairwise chain, because count-first clears its scratch and copies the result out where the chain clones the first input and ANDs in place.
• A wider sweep, not included here, over container kind, size, interval count, key count and input count, 297 cells and three to six inputs, gave the same picture on the Xeon and on an M1. The only cells the chain wins are those tiny-array ones. FastAnd is faster by 18% at the median and by more than 20% in 145 cells.
• On arm64 the jointly-empty gain is smaller, 11.7 µs to 5.1 µs on an M1, because andCardSlice has no NEON version and runs as the Go loop there.
• BenchmarkIntersection*, BenchmarkUnion* and BenchmarkAndAny/* are unchanged.

Breaking Changes

None. The result is the same as the pairwise And chain.

Related Issues

Related to #566.

Additional Notes

The bitmapContainer.andArray hoist is not specific to FastAnd: And(bitmap, array) and Intersects get the same improvement.

🤖 Generated with Claude Code