From 38b932b909a738daf62a9c000ee8272be23407c8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 18:02:44 +0800 Subject: [PATCH 01/14] [core] Add standalone manifest sidecar format and utilities --- docs/docs/concepts/spec/manifest.md | 230 ++++ .../paimon/manifest/ManifestSidecar.java | 1055 +++++++++++++++ .../manifest/ManifestBlockIndexTest.java | 803 ++++++++++++ .../paimon/manifest/ManifestSidecarTest.java | 1145 +++++++++++++++++ .../src/test/resources/manifest-sidecar.txt | 23 + 5 files changed, 3256 insertions(+) create mode 100644 paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java create mode 100644 paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java create mode 100644 paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java create mode 100644 paimon-core/src/test/resources/manifest-sidecar.txt diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 6ad271bb5a89..97895145462b 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -63,6 +63,236 @@ skip manifests before opening them. Each extra file belongs exclusively to one manifest. It is retained and cleaned up together with that manifest during snapshot, tag, or changelog deletion. +### Manifest Sidecar + +`ManifestSidecar` provides a binary sidecar for selecting complete Avro manifest blocks +using independent partition, row-ID and bucket coverage. A sidecar uses the +`.avro.sidecar` naming convention. Readers find it through an explicit +`.avro.sidecar` reference in the manifest metadata's `_EXTRA_FILES`, without probing a +derived file name. The Avro schemas and `_VERSION` identifiers remain unchanged. + +The utility includes construction, validation, block selection and optional caching. Table +writers and scans do not yet invoke it automatically. Callers are responsible for publishing +sidecar references, managing file ownership, applying entry filters and reconciling ADD/DELETE +entries after block selection. `build` reads the completed physical manifest and returns sidecar +bytes; it does not write or publish another file. + +`Settings` takes a byte budget and separate booleans for partition, row-ID and bucket payload +generation. A disabled dimension uses encoding 0 and has no length or payload bytes. The +partition dictionary is empty when partition generation is disabled. These generation settings +do not prevent readers from using payloads already present in a sidecar. + +`read` returns null for an absent sidecar reference or an `IOException`, allowing the caller +to fall back to the manifest. If the thread is interrupted, the I/O failure is propagated as +`UncheckedIOException`. Other exceptions and errors propagate unchanged. `select` validates +supplied bytes directly and reports invalid containers with `IOException`. + +Version 1 uses the following layout. Container integers and payload integers +are fixed-width big endian. Encoding IDs are unsigned bytes with separate namespaces. + +```text +magic : 8 bytes // ASCII PAIMSCAR +formatVersion : int // 1 +manifestNameHash : 32 bytes // SHA-256 of the UTF-8 basename +manifestLength : long +manifestEntryCount : long // ADD + DELETE +avroHeaderLength : int +avroHeader : bytes // original schema, codec and sync marker +partitionCount : int +partitionDictionary[] + partitionByteLength : int + partitionBytes : bytes // existing manifest BinaryRow serialization +blockCount : int +blocks[] // original physical order + offset : long + length : long // complete encoded block, including sync marker + recordCount : long + partitionEncoding : byte + if partitionEncoding != 0: + partitionPayloadLength : int + partitionPayload : bytes + rowIdEncoding : byte + if rowIdEncoding != 0: + rowIdPayloadLength : int + rowIdPayload : bytes + bucketEncoding : byte + if bucketEncoding != 0: + bucketPayloadLength : int + bucketPayload : bytes +checksum : 32 bytes // SHA-256 of all preceding bytes +``` + +The block ID is its position. Its first entry ordinal is the sum of preceding record +counts and is not stored. Each complete partition tuple appears once in the dictionary, +including all its fields and nulls. The scan's partition type interprets the existing +serialized tuple. Partition predicates are evaluated once per dictionary entry. + +| Dimension | Encoding | Payload | +| --- | --- | --- | +| Any | `0` | Unavailable; only the encoding byte is present. | +| Partition | `1` | Positive `partitionIdCount: int` followed by sorted unique dictionary IDs (`int`). | +| Row ID | `1` | Positive `rangeCount: int` followed by sorted disjoint inclusive `(start: long, end: long)` pairs. Coverage may conservatively include gaps. | +| Bucket | `1` | Positive `pairCount: int` followed by sorted unique `(bucket: int, totalBuckets: int)` pairs. | +| Any | Other nonzero ID | Skip exactly the bounded payload length; treat only this dimension as unavailable. | + +Only nonzero encodings are followed by a length and payload. Payload lengths exclude +the encoding and length fields, but include the count at the start of the payload. +For all three encoding-1 payloads below, `int` is a signed 4-byte integer and `long` is +a signed 8-byte integer, both big endian. Elements have no padding, per-element length +prefixes, or Avro variable-length integer encoding. Counts must be positive; encoding 0 +represents unavailable coverage, rather than encoding 1 with a zero count. + +#### Partition Payload + +When `partitionEncoding == 1`, the block stores the IDs of all distinct partition tuples +represented by its entries: + +```text +partitionPayload + partitionIdCount : int // N > 0 + partitionIds[N] : int // N consecutive 4-byte dictionary IDs + +partitionPayloadLength = 4 + 4 * N +``` + +An ID is the zero-based position of a complete tuple in the sidecar's shared +`partitionDictionary`, not an individual partition field or an entry ordinal. Valid IDs +satisfy `0 <= id < partitionCount` and are strictly increasing, with no duplicates. +The tuple bytes appear only in the dictionary; they are not repeated in each block's payload. +For example, IDs `[0, 3]` are stored as the three integers `[2, 0, 3]`, occupying 12 payload +bytes, or 17 bytes including `partitionEncoding` and `partitionPayloadLength`. + +With a partition filter, the block matches if any referenced dictionary tuple matches. +A tuple containing a null partition value can still have a valid dictionary ID. If any +entry's partition tuple is unavailable, or partition coverage cannot fit its budget, the +block uses encoding 0 so that missing dictionary coverage cannot exclude it. + +#### Row-ID Payload + +When `rowIdEncoding == 1`, the block stores inclusive row-ID intervals: + +```text +rowIdPayload + rangeCount : int // N > 0 + ranges[N] + start : long // inclusive first row ID + end : long // inclusive last row ID + +rowIdPayloadLength = 4 + 16 * N +``` + +Each pair satisfies `0 <= start <= end <= Long.MAX_VALUE`. Pairs are ordered by `start` +and do not overlap: each `start` is greater than the preceding `end`. The writer merges +overlapping and adjacent intervals contributed by the entries. An entry contributes +`[firstRowId, firstRowId + rowCount - 1]`; these are table row IDs, not manifest entry +ordinals. `rangeCount` counts intervals, not entries or individual row IDs. + +There are no separate block min/max fields in this payload. The reader obtains the block +minimum from the first pair's `start` and the maximum from the last pair's `end`. It tests +this envelope first, then checks individual intervals if necessary. For example, +`[(10, 19), (30, 39)]` is stored as `rangeCount = 2` followed by four longs. Its payload +length is 36 bytes, or 41 bytes including the encoding and length fields. Its envelope +is `[10, 39]`, but a query for row ID 25 does not match either interval. + +If the exact union exceeds its available budget, the writer can store one conservative +`[min,max]` pair using the same encoding. That payload has `rangeCount = 1` and length +20 bytes; it can include gaps. There is no separate flag distinguishing a coarsened pair +from an exact interval, so entry filtering remains necessary. Unknown or invalid row-ID +metadata makes coverage unavailable for the block; further byte-budget degradation can +also drop the payload entirely. + +#### Bucket Payload + +When `bucketEncoding == 1`, the block stores distinct bucket/count pairs: + +```text +bucketPayload + pairCount : int // N > 0 + pairs[N] + bucket : int // entry's bucket number + totalBuckets : int // entry's recorded total bucket count + +bucketPayloadLength = 4 + 8 * N +``` + +Each pair satisfies `0 <= bucket < totalBuckets`. Pairs are sorted by `bucket`, then +`totalBuckets`, and deduplicated. `totalBuckets` comes from the entry's `_TOTAL_BUCKETS`; +it is not the number of buckets represented by this block or the table's current bucket +setting. The same bucket number can therefore occur with different totals after rescaling. +For example, `[(1, 4), (1, 8), (3, 4)]` is stored as the seven integers +`[3, 1, 4, 1, 8, 3, 4]`, occupying 28 payload bytes, or 33 bytes including the encoding +and length fields. These pairs have no partition IDs or separate bucket min/max fields. + +Missing, invalid, negative/synthetic or over-budget bucket metadata makes that block's +bucket coverage unavailable (encoding 0, no length or payload). Partition and row-ID +coverage remain independently usable; no mutual-exclusion restriction is imposed. + +A caller can supply a predicate on `(bucket, totalBuckets)` to test this payload. The +predicate must conservatively retain every potentially matching pair. Filters requiring an +entry's partition belong at the entry-filtering stage; omit the bucket predicate when no +safe partition-independent check is available. An unavailable bucket payload cannot exclude +a block. Malformed payload lengths or pair counts invalidate the container. Invalid ordering +or values encountered while matching also invalidate it; elements after the first match are skipped. + +#### Validation and Coverage + +Invalid lengths, known-payload framing, checksum mismatches or inconsistent physical +coverage invalidate the container. Invalid dictionary references or interval ordering +encountered in decoded payload contents also invalidate it. Byte spans must cover the +entire original manifest after its header; record counts must sum to the manifest entry +count. Readers validate the checksum, payload framing (including known count/length +consistency), and the complete block directory even when a block is rejected. Block +payload contents are decoded and validated only for dimensions still needed by the filters, +and only until that dimension matches. A row-ID min/max rejection skips individual +intervals; a match skips the remaining elements of that payload. Skipped payload contents +are not individually validated. + +All entries contribute, including ADD, DELETE and every file format/column group. +Row-ID ranges are never expanded into individual values. If an exact union exceeds its +available byte budget, it becomes the inclusive `[min,max]` envelope with encoding 1. Processing +continues through the end of the block to extend those bounds and detect unknown row IDs. +An unknown or invalid row-ID range makes only that block's row-ID payload unavailable. +Partition budget exhaustion independently makes that block's partition payload unavailable. +The dictionary can consequently be incomplete for the manifest: a dictionary miss never +excludes a block with unavailable partition coverage. Later blocks can still use existing IDs. + +`Settings.maxBytes` bounds the whole serialized container, including the partition +dictionary and all three payload types. The caller supplies the byte budget. It is capped +at 2147483646 bytes to fit the in-memory byte-array representation. A budget too small for a +complete sidecar causes `build` or `read` to return null; callers can keep using the manifest. +The Avro header and block directory share this byte budget without separate size or count limits. +Writers discard optional row-ID payloads, bucket payloads, then partition payloads/dictionary if necessary, +to fit the complete directory. If the directory itself cannot fit, no sidecar is published. +No emitted sidecar omits block descriptors. These are encoded-size bounds; Avro header parsing +and sidecar construction also incur object/buffer overhead. Query concurrency multiplies per-reader costs. + +For conjunctive filters a block is retained only if each dimension is either unavailable +or matches. Within each block, matching tests row ID, partition, then bucket coverage. +It skips absent filters and short-circuits after a dimension rejects a block, skipping +the contents of later payloads. Within each payload, matching stops at the first hit. Matches in different dimensions can come +from different entries in the block, so entry filtering and deletion merging remain +necessary. Block min/max is derived from the first/last interval before testing the +individual intervals. + +Readers still consume and validate the whole bounded sidecar. A partition-only query +therefore reads row-ID payload bytes too; payload lengths save decoding work for unused +payload contents and unknown encodings, not storage I/O. Selected compressed blocks are read by byte range with adjacent +spans coalesced. Building a sidecar does not modify the original manifest. + +`read` and `openManifest` accept an optional caller-supplied `SegmentsCache`. Complete +sidecar bytes are keyed by their explicit `Path`, subject to the cache memory budget and single-file threshold. +Only successful reads and selections populate the cache. Each query creates independent +views and reapplies its filters and byte budget; query-specific selections are not cached. + +Selected Avro blocks also share this cache. Each entry contains one complete compressed +block, keyed by the manifest's full path, original offset and encoded length, separately +from whole-manifest and sidecar entries. Different selections can reuse the same blocks. +Only successful complete reads populate the cache; oversized blocks stream through the +bounded read buffer. Adjacent uncached blocks are read together when they fit the read +buffer, then cached individually. Fully cached selections do not open the manifest file. +Block entries follow the existing memory budget, entry-size limit, expiration and eviction +settings. The Avro decoder and entry filters still run on cached bytes. + ## Manifest Data manifests record **ADD** (`0`) and **DELETE** (`1`) entries. Readers reconcile these entries diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java new file mode 100644 index 000000000000..b2807fa0796f --- /dev/null +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -0,0 +1,1055 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.manifest; + +import org.apache.paimon.annotation.VisibleForTesting; +import org.apache.paimon.data.BinaryRow; +import org.apache.paimon.data.Segments; +import org.apache.paimon.fs.FileIO; +import org.apache.paimon.fs.Path; +import org.apache.paimon.fs.SeekableInputStream; +import org.apache.paimon.partition.PartitionPredicate; +import org.apache.paimon.types.DataField; +import org.apache.paimon.types.RowType; +import org.apache.paimon.utils.RowRangeIndex; +import org.apache.paimon.utils.SegmentsCache; +import org.apache.paimon.utils.SerializationUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.annotation.Nullable; + +import java.io.ByteArrayOutputStream; +import java.io.DataOutputStream; +import java.io.EOFException; +import java.io.IOException; +import java.io.InputStream; +import java.io.UncheckedIOException; +import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.TreeMap; +import java.util.TreeSet; +import java.util.function.BiPredicate; + +/** Independently usable partition, row-id and bucket coverage for each manifest block. */ +public final class ManifestSidecar { + public static final String SUFFIX = ".avro.sidecar"; + private static final Logger LOG = LoggerFactory.getLogger(ManifestSidecar.class); + private static final long MAGIC = 0x5041494d53434152L; + private static final int FORMAT_VERSION = 1; + private static final int HEADER_BYTES = 60; + private static final int BLOCK_BYTES = 27; + private static final byte[] EMPTY = new byte[0]; + private static final int DIGEST_BYTES = 32; + private static final int READ_BUFFER_BYTES = 1024 * 1024; + private static final ProjectedManifestEntry.Projection BLOCK_INDEX_PROJECTION = + createBlockIndexProjection(); + + private ManifestSidecar() {} + + public static Path path(Path manifest) { + return new Path(manifest.toString() + SUFFIX); + } + + @Nullable + public static String fileName(ManifestFileMeta manifest) { + if (manifest.extraFiles() != null) { + for (String extraFile : manifest.extraFiles()) { + if (extraFile.endsWith(SUFFIX)) { + return extraFile; + } + } + } + return null; + } + + /** Construction bounds and independently enabled payloads, supplied by the caller. */ + public static final class Settings { + public final int maxBytes; + public final boolean partitionEnabled; + public final boolean rowIdEnabled; + public final boolean bucketEnabled; + + public Settings( + long maxBytes, + boolean partitionEnabled, + boolean rowIdEnabled, + boolean bucketEnabled) { + if (maxBytes < 0) { + throw new IllegalArgumentException( + "Manifest sidecar byte budget must be nonnegative"); + } + // A sidecar fits in one byte array; reserve one byte for the overflow probe. + this.maxBytes = (int) Math.min(maxBytes, Integer.MAX_VALUE - 1L); + this.partitionEnabled = partitionEnabled; + this.rowIdEnabled = rowIdEnabled; + this.bucketEnabled = bucketEnabled; + } + } + + /** Original file offset/length and zero-based manifest entry ordinal, not table row id. */ + public static final class Block { + public final long offset; + public final long length; + public final long firstRecord; + public final long recordCount; + + public Block(long offset, long length, long firstRecord, long recordCount) { + this.offset = offset; + this.length = length; + this.firstRecord = firstRecord; + this.recordCount = recordCount; + } + } + + /** Selected blocks in original file order. Empty means the manifest can be excluded. */ + public static final class Selection { + private final byte[] header; + private final List blocks; + + private Selection(byte[] header, List blocks) { + this.header = header; + this.blocks = Collections.unmodifiableList(blocks); + } + + public List blocks() { + return blocks; + } + } + + /** + * Builds complete block descriptors even when either optional dimension becomes unavailable. + */ + public static final class Builder { + private final Settings settings; + private final byte[] header; + private final TreeMap ranges = new TreeMap<>(); + private final Map dictionary = new LinkedHashMap<>(); + private final TreeSet partitionIds = new TreeSet<>(); + private final TreeSet bucketPairs = new TreeSet<>(); + private final List blocks = new ArrayList<>(); + private boolean complete; + private long nextOffset; + private long nextRecord; + private Block current; + private long entriesInBlock; + private boolean rowAvailable; + private boolean partitionAvailable; + private boolean bucketAvailable; + private boolean coarse; + private long min; + private long max; + private int dictionaryBytes; + private int optionalBytes; + + public Builder(Settings settings, @Nullable byte[] header) { + this.settings = settings; + this.header = header; + complete = + header != null + && HEADER_BYTES + DIGEST_BYTES + 12L + header.length + <= settings.maxBytes; + nextOffset = header == null ? 0 : header.length; + } + + public boolean complete() { + return complete; + } + + public void beginBlock(long offset, long length, long records) throws IOException { + if (!complete) { + return; + } + require(current == null && offset == nextOffset && length > 0 && records > 0); + // Optional payloads can be discarded later, but descriptors must never be truncated. + if (HEADER_BYTES + + DIGEST_BYTES + + 12L + + header.length + + (blocks.size() + 1L) * BLOCK_BYTES + > settings.maxBytes) { + complete = false; + blocks.clear(); + dictionary.clear(); + return; + } + current = new Block(offset, length, nextRecord, records); + entriesInBlock = 0; + rowAvailable = settings.rowIdEnabled; + partitionAvailable = settings.partitionEnabled; + bucketAvailable = settings.bucketEnabled; + coarse = false; + min = Long.MAX_VALUE; + max = -1; + ranges.clear(); + partitionIds.clear(); + bucketPairs.clear(); + } + + @VisibleForTesting + public void add(@Nullable Long first, long count) { + add(first, count, null); + } + + @VisibleForTesting + public void add(@Nullable Long first, long count, @Nullable byte[] partition) { + add(first, count, partition, null, null); + } + + public void add( + @Nullable Long first, + long count, + @Nullable byte[] partition, + @Nullable Integer bucket, + @Nullable Integer totalBuckets) { + if (!complete) { + return; + } + if (current == null) { + throw new IllegalStateException("No current Avro block"); + } + entriesInBlock++; + addPartition(partition); + addBucket(bucket, totalBuckets); + if (!rowAvailable) { + return; + } + if (first == null || first < 0 || count <= 0 || count - 1 > Long.MAX_VALUE - first) { + rowAvailable = false; + ranges.clear(); + return; + } + long end = first + (count - 1); + min = Math.min(min, first); + max = Math.max(max, end); + // Keep checking subsequent entries, including missing row IDs, after coarsening. + if (coarse) { + return; + } + long start = first; + Map.Entry before = ranges.floorEntry(start); + if (before != null && before.getValue() >= start - 1) { + start = before.getKey(); + end = Math.max(end, before.getValue()); + ranges.remove(before.getKey()); + } + Map.Entry next; + while ((next = ranges.ceilingEntry(start)) != null + && (next.getKey() <= end || next.getKey() - end == 1)) { + end = Math.max(end, next.getValue()); + ranges.remove(next.getKey()); + } + if (8L + 16L * (ranges.size() + 1L) > settings.maxBytes - optionalBytes) { + coarse = true; + ranges.clear(); + } else { + ranges.put(start, end); + } + } + + private void addBucket(@Nullable Integer bucket, @Nullable Integer totalBuckets) { + if (!bucketAvailable) { + return; + } + if (bucket == null + || totalBuckets == null + || bucket < 0 + || totalBuckets <= 0 + || bucket >= totalBuckets) { + bucketAvailable = false; + bucketPairs.clear(); + return; + } + long pair = ((long) bucket << 32) | totalBuckets; + if (!bucketPairs.contains(pair) + && 8L + 8L * (bucketPairs.size() + 1L) > settings.maxBytes - optionalBytes) { + bucketAvailable = false; + bucketPairs.clear(); + } else { + bucketPairs.add(pair); + } + } + + private void addPartition(@Nullable byte[] bytes) { + if (!partitionAvailable) { + return; + } + if (bytes == null) { + partitionAvailable = false; + partitionIds.clear(); + return; + } + Integer id = dictionary.get(ByteBuffer.wrap(bytes)); + if (id == null) { + if (bytes.length + 4L > settings.maxBytes - dictionaryBytes) { + partitionAvailable = false; + partitionIds.clear(); + return; + } + id = dictionary.size(); + dictionary.put(ByteBuffer.wrap(bytes.clone()), id); + dictionaryBytes += 4 + bytes.length; + } + partitionIds.add(id); + } + + public void endBlock() throws IOException { + if (!complete) { + return; + } + require(current != null && entriesInBlock == current.recordCount); + byte[] rowPayload = EMPTY; + byte[] partitionPayload = EMPTY; + if (rowAvailable) { + if (coarse || 8L + 16L * ranges.size() > settings.maxBytes - optionalBytes) { + ranges.clear(); + ranges.put(min, max); + } + if (8L + 16L * ranges.size() <= settings.maxBytes - optionalBytes) { + ByteBuffer out = ByteBuffer.allocate(4 + 16 * ranges.size()); + out.putInt(ranges.size()); + for (Map.Entry range : ranges.entrySet()) { + out.putLong(range.getKey()).putLong(range.getValue()); + } + rowPayload = out.array(); + optionalBytes += payloadSize(rowPayload); + } + } + if (partitionAvailable + && 8L + 4L * partitionIds.size() <= settings.maxBytes - optionalBytes) { + ByteBuffer out = ByteBuffer.allocate(4 + 4 * partitionIds.size()); + out.putInt(partitionIds.size()); + for (int id : partitionIds) { + out.putInt(id); + } + partitionPayload = out.array(); + optionalBytes += payloadSize(partitionPayload); + } + byte[] bucketPayload = EMPTY; + if (bucketAvailable + && 8L + 8L * bucketPairs.size() <= settings.maxBytes - optionalBytes) { + ByteBuffer out = ByteBuffer.allocate(4 + 8 * bucketPairs.size()); + out.putInt(bucketPairs.size()); + for (long pair : bucketPairs) { + out.putInt((int) (pair >>> 32)).putInt((int) pair); + } + bucketPayload = out.array(); + optionalBytes += payloadSize(bucketPayload); + } + blocks.add(new IndexedBlock(current, partitionPayload, rowPayload, bucketPayload)); + nextOffset = Math.addExact(current.offset, current.length); + nextRecord = Math.addExact(current.firstRecord, current.recordCount); + ranges.clear(); + partitionIds.clear(); + bucketPairs.clear(); + current = null; + } + + @Nullable + public byte[] serialize(String name, long fileSize, long entryCount) throws IOException { + if (!complete) { + return null; + } + require(current == null && nextOffset == fileSize && nextRecord == entryCount); + long size = + HEADER_BYTES + + DIGEST_BYTES + + 12L + + header.length + + dictionaryBytes + + blocks.size() * (long) BLOCK_BYTES + + optionalBytes; + // Give directory growth priority over optional coverage. Never remove a descriptor. + for (IndexedBlock block : blocks) { + if (size <= settings.maxBytes) { + break; + } + size -= payloadSize(block.rowIds); + optionalBytes -= payloadSize(block.rowIds); + block.rowIds = EMPTY; + } + for (IndexedBlock block : blocks) { + if (size <= settings.maxBytes) { + break; + } + size -= payloadSize(block.buckets); + optionalBytes -= payloadSize(block.buckets); + block.buckets = EMPTY; + } + if (size > settings.maxBytes) { + size -= dictionaryBytes; + dictionaryBytes = 0; + dictionary.clear(); + for (IndexedBlock block : blocks) { + size -= payloadSize(block.partitions); + optionalBytes -= payloadSize(block.partitions); + block.partitions = EMPTY; + } + } + require(size <= settings.maxBytes); + ByteArrayOutputStream buffer = new ByteArrayOutputStream((int) size); + DataOutputStream out = new DataOutputStream(buffer); + out.writeLong(MAGIC); + out.writeInt(FORMAT_VERSION); + out.write(digest(name.getBytes(StandardCharsets.UTF_8))); + out.writeLong(fileSize); + out.writeLong(entryCount); + out.writeInt(header.length); + out.write(header); + out.writeInt(dictionary.size()); + for (ByteBuffer bytes : dictionary.keySet()) { + out.writeInt(bytes.remaining()); + out.write(bytes.array()); + } + out.writeInt(blocks.size()); + for (IndexedBlock block : blocks) { + out.writeLong(block.block.offset); + out.writeLong(block.block.length); + out.writeLong(block.block.recordCount); + writePayload(out, block.partitions); + writePayload(out, block.rowIds); + writePayload(out, block.buckets); + } + out.write(digest(buffer.toByteArray())); + return buffer.toByteArray(); + } + + private static int payloadSize(byte[] payload) { + return payload.length == 0 ? 0 : Integer.BYTES + payload.length; + } + + private static void writePayload(DataOutputStream out, byte[] payload) throws IOException { + out.writeByte(payload.length == 0 ? 0 : 1); + if (payload.length > 0) { + out.writeInt(payload.length); + out.write(payload); + } + } + } + + private static final class IndexedBlock { + private final Block block; + private byte[] partitions; + private byte[] rowIds; + private byte[] buckets; + + private IndexedBlock(Block block, byte[] partitions, byte[] rowIds, byte[] buckets) { + this.block = block; + this.partitions = partitions; + this.rowIds = rowIds; + this.buckets = buckets; + } + } + + private static ProjectedManifestEntry.Projection createBlockIndexProjection() { + List fields = + new ArrayList<>( + ProjectedManifestEntry.ROW_RANGE_PROJECTION.projectedType().getFields()); + fields.add(ManifestEntry.MANIFEST_ROW_TYPE.getField(ManifestEntry.BUCKET)); + fields.add(ManifestEntry.MANIFEST_ROW_TYPE.getField(ManifestEntry.TOTAL_BUCKETS)); + return ProjectedManifestEntry.Projection.create(new RowType(false, fields)); + } + + /** Rebuild from the final physical blocks, including raw-copy and encoded rewrite paths. */ + @Nullable + public static byte[] build(FileIO io, Path path, long size, long records, Settings settings) + throws IOException { + if (settings.maxBytes < 128) { + return null; + } + try (ManifestAvroReader reader = new ManifestAvroReader(io.newInputStream(path))) { + Builder builder = new Builder(settings, reader.headerBytes()); + ProjectedManifestEntry.Projection projection = BLOCK_INDEX_PROJECTION; + ProjectedManifestEntry entry = projection.createEntry(); + while (builder.complete() && reader.hasNext()) { + ManifestAvroReader.RawBlock block = reader.next(); + builder.beginBlock(reader.blockOffset(), reader.blockLength(), block.recordCount()); + ManifestAvroReader.RowIterator rows = block.toRows(projection.projectedType()); + while (builder.complete() && rows.hasNext()) { + entry.replace(rows.next()); + builder.add( + settings.rowIdEnabled ? entry.file().firstRowId() : null, + settings.rowIdEnabled ? entry.file().rowCount() : 0, + settings.partitionEnabled ? entry.partitionBytes() : null, + settings.bucketEnabled ? entry.bucket() : null, + settings.bucketEnabled ? entry.totalBuckets() : null); + } + builder.endBlock(); + } + return builder.serialize(path.getName(), size, records); + } + } + + /** Selects blocks using row-ID coverage. A null query retains every block after validation. */ + public static Selection select( + byte[] data, + ManifestFileMeta manifest, + @Nullable RowRangeIndex query, + Settings settings) + throws IOException { + return select(data, manifest, query, null, null, settings); + } + + /** Validates framing and tests row ID, partition, then bucket coverage. */ + public static Selection select( + byte[] data, + ManifestFileMeta manifest, + @Nullable RowRangeIndex query, + @Nullable PartitionPredicate partitionFilter, + @Nullable RowType partitionType, + Settings settings) + throws IOException { + return select(data, manifest, query, partitionFilter, partitionType, null, settings); + } + + /** + * Selects blocks using independent filters. The bucket predicate must conservatively test only + * the bucket and recorded total bucket count; omit it if filtering requires an entry partition. + */ + public static Selection select( + byte[] data, + ManifestFileMeta manifest, + @Nullable RowRangeIndex query, + @Nullable PartitionPredicate partitionFilter, + @Nullable RowType partitionType, + @Nullable BiPredicate bucketFilter, + Settings settings) + throws IOException { + require(data.length >= 128 && data.length <= settings.maxBytes); + int limit = data.length - DIGEST_BYTES; + require( + MessageDigest.isEqual( + digest(data, limit), Arrays.copyOfRange(data, limit, data.length))); + ByteBuffer in = ByteBuffer.wrap(data, 0, limit).slice(); + require(in.getLong() == MAGIC); + require(in.getInt() == FORMAT_VERSION); + byte[] hash = new byte[DIGEST_BYTES]; + in.get(hash); + require( + MessageDigest.isEqual( + hash, digest(manifest.fileName().getBytes(StandardCharsets.UTF_8)))); + require(in.getLong() == manifest.fileSize()); + long entries = Math.addExact(manifest.numAddedFiles(), manifest.numDeletedFiles()); + require(in.getLong() == entries); + int headerLength = in.getInt(); + require(headerLength >= 21 && headerLength <= in.remaining() - 8); + byte[] header = new byte[headerLength]; + in.get(header); + require(header[0] == 'O' && header[1] == 'b' && header[2] == 'j' && header[3] == 1); + int partitions = in.getInt(); + require(partitions >= 0 && partitions <= in.remaining() / 16); + boolean[] matches = partitionFilter == null ? null : new boolean[partitions]; + Set unique = new java.util.HashSet<>(); + for (int id = 0; id < partitions; id++) { + require(in.remaining() >= 4); + int length = in.getInt(); + require(length >= 12 && length <= in.remaining()); + ByteBuffer encoded = in.slice(); + encoded.limit(length); + int arity = encoded.getInt(0); + require(arity >= 0 && 4L + ((arity + 71L) / 64) * 8 + arity * 8L <= length); + require(partitionType == null || arity == partitionType.getFieldCount()); + require(unique.add(encoded.asReadOnlyBuffer())); + if (partitionFilter != null) { + byte[] bytes = new byte[length]; + encoded.get(bytes); + BinaryRow partition = SerializationUtils.deserializeBinaryRow(bytes); + matches[id] = partitionFilter.test(partition); + } + in.position(in.position() + length); + } + require(in.remaining() >= 4); + int count = in.getInt(); + require(count >= 0 && count <= in.remaining() / BLOCK_BYTES); + long nextOffset = headerLength; + long firstRecord = 0; + List selected = new ArrayList<>(); + for (int i = 0; i < count; i++) { + require(in.remaining() >= BLOCK_BYTES); + long offset = in.getLong(); + long length = in.getLong(); + long records = in.getLong(); + require(offset == nextOffset && length > 0 && length <= manifest.fileSize() - offset); + require(records > 0 && records <= entries - firstRecord); + ByteBuffer partitionPayload = payload(in, Integer.BYTES); + ByteBuffer rowPayload = payload(in, 2 * Long.BYTES); + ByteBuffer bucketPayload = payload(in, 2 * Integer.BYTES); + long blockFirstRecord = firstRecord; + nextOffset = offset + length; + firstRecord += records; + + if (query != null && rowPayload != null) { + boolean singleRange = rowPayload.remaining() == 2 * Long.BYTES; + long min = rowPayload.getLong(); + long firstEnd = rowPayload.getLong(); + long max = + singleRange + ? firstEnd + : rowPayload.getLong(rowPayload.limit() - Long.BYTES); + require(min >= 0 && firstEnd >= min && max >= firstEnd); + if (!query.intersects(min, max)) { + continue; + } + boolean rowHit = singleRange || query.intersects(min, firstEnd); + long previous = firstEnd; + while (!rowHit && rowPayload.hasRemaining()) { + long rangeStart = rowPayload.getLong(); + long rangeEnd = rowPayload.getLong(); + require(rangeStart >= 0 && rangeEnd >= rangeStart && rangeStart > previous); + previous = rangeEnd; + rowHit = query.intersects(rangeStart, rangeEnd); + } + if (!rowHit) { + continue; + } + } + + if (partitionFilter != null && partitionPayload != null) { + boolean partitionHit = false; + int previous = -1; + while (!partitionHit && partitionPayload.hasRemaining()) { + int id = partitionPayload.getInt(); + require(id > previous && id < partitions); + previous = id; + partitionHit = matches[id]; + } + if (!partitionHit) { + continue; + } + } + + if (bucketFilter != null && bucketPayload != null) { + boolean bucketHit = false; + long previous = -1; + while (!bucketHit && bucketPayload.hasRemaining()) { + int bucket = bucketPayload.getInt(); + int totalBuckets = bucketPayload.getInt(); + require(bucket >= 0 && totalBuckets > bucket); + long pair = ((long) bucket << 32) | totalBuckets; + require(pair > previous); + previous = pair; + bucketHit = bucketFilter.test(bucket, totalBuckets); + } + if (!bucketHit) { + continue; + } + } + selected.add(new Block(offset, length, blockFirstRecord, records)); + } + require(!in.hasRemaining() && nextOffset == manifest.fileSize() && firstRecord == entries); + return new Selection(header, selected); + } + + /** Reads framing and exposes known payload elements without decoding their contents. */ + @Nullable + private static ByteBuffer payload(ByteBuffer in, int elementBytes) throws IOException { + require(in.hasRemaining()); + int encoding = Byte.toUnsignedInt(in.get()); + if (encoding == 0) { + return null; + } + require(in.remaining() >= Integer.BYTES); + int length = in.getInt(); + require(length >= 0 && length <= in.remaining()); + ByteBuffer result = in.slice(); + result.limit(length); + in.position(in.position() + length); + if (encoding != 1) { + return null; + } + require(result.remaining() >= Integer.BYTES); + int count = result.getInt(); + require(count > 0 && result.remaining() == (long) elementBytes * count); + return result; + } + + /** Bounded, bulk sidecar reads. Null means read the original manifest. */ + @Nullable + public static Selection read( + FileIO io, + Path path, + ManifestFileMeta manifest, + @Nullable RowRangeIndex query, + Settings settings) { + return read(io, path, manifest, query, null, null, settings); + } + + @Nullable + public static Selection read( + FileIO io, + Path path, + ManifestFileMeta manifest, + @Nullable RowRangeIndex query, + @Nullable PartitionPredicate partitionFilter, + @Nullable RowType partitionType, + Settings settings) { + return read( + io, path, manifest, query, partitionFilter, partitionType, null, settings, null); + } + + @Nullable + public static Selection read( + FileIO io, + Path path, + ManifestFileMeta manifest, + @Nullable RowRangeIndex query, + @Nullable PartitionPredicate partitionFilter, + @Nullable RowType partitionType, + @Nullable BiPredicate bucketFilter, + Settings settings, + @Nullable SegmentsCache cache) { + String sidecarFileName = fileName(manifest); + if (sidecarFileName == null || settings.maxBytes < 128) { + return null; + } + try { + Path sidecarPath = new Path(path.getParent(), sidecarFileName); + Segments cached = cache == null ? null : cache.getIfPresents(sidecarPath); + boolean cacheHit = cached instanceof ManifestSidecarSegment; + byte[] data = + cacheHit + ? ((ManifestSidecarSegment) cached).bytes() + : readBytes(io, sidecarPath, settings.maxBytes); + Selection selection = + select( + data, + manifest, + query, + partitionFilter, + partitionType, + bucketFilter, + settings); + if (cache != null && !cacheHit && data.length <= cache.maxElementSize()) { + cache.put(sidecarPath, new ManifestSidecarSegment(data)); + } + return selection; + } catch (IOException failure) { + if (Thread.currentThread().isInterrupted()) { + throw new UncheckedIOException(failure); + } + LOG.debug("Cannot use manifest sidecar for {}; reading manifest", path, failure); + return null; + } + } + + /** Complete sidecar bytes stored in the shared manifest cache. */ + static final class ManifestSidecarSegment implements Segments { + private final byte[] bytes; + + public ManifestSidecarSegment(byte[] bytes) { + this.bytes = bytes; + } + + public byte[] bytes() { + return bytes; + } + + @Override + public long totalMemorySize() { + return bytes.length; + } + } + + private static byte[] readBytes(FileIO io, Path path, int maxBytes) throws IOException { + try (InputStream in = io.newInputStream(path)) { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + byte[] buffer = new byte[Math.min(READ_BUFFER_BYTES, maxBytes + 1)]; + int n; + while ((n = in.read(buffer, 0, Math.min(buffer.length, maxBytes + 1 - out.size()))) + != -1) { + require(n <= maxBytes - out.size()); + out.write(buffer, 0, n); + } + return out.toByteArray(); + } + } + + static InputStream openManifest(FileIO io, Path path, @Nullable Selection selected) + throws IOException { + return openManifest(io, path, selected, null); + } + + static InputStream openManifest( + FileIO io, + Path path, + @Nullable Selection selected, + @Nullable SegmentsCache cache) + throws IOException { + return selected == null + ? io.newInputStream(path) + : new SelectedBlockInput(io, path, selected, cache); + } + + /** Separates physical byte ranges from whole-file cache keys. */ + static final class BlockCacheKey { + private final Path path; + private final long offset; + private final long length; + + BlockCacheKey(Path path, long offset, long length) { + this.path = Objects.requireNonNull(path); + this.offset = offset; + this.length = length; + } + + @Override + public boolean equals(Object other) { + if (!(other instanceof BlockCacheKey)) { + return false; + } + BlockCacheKey that = (BlockCacheKey) other; + return path.equals(that.path) && offset == that.offset && length == that.length; + } + + @Override + public int hashCode() { + return Objects.hash(path, offset, length); + } + } + + /** Complete encoded Avro blocks, distinct from cached manifest entries and sidecar bytes. */ + private static final class ManifestBlockSegment implements Segments { + private final byte[] bytes; + + private ManifestBlockSegment(byte[] bytes) { + this.bytes = bytes; + } + + @Override + public long totalMemorySize() { + return bytes.length; + } + } + + /** An OCF stream comprising the original header and selected complete compressed blocks. */ + private static final class SelectedBlockInput extends InputStream { + private final FileIO io; + private final Path path; + private final Selection selected; + @Nullable private final SegmentsCache cache; + @Nullable private SeekableInputStream input; + private boolean closed; + private int headerPosition; + private int blockPosition; + private long remaining; + private byte[] buffer; + private int bufferPosition; + private int bufferLimit; + + private SelectedBlockInput( + FileIO io, Path path, Selection selected, @Nullable SegmentsCache cache) { + this.io = io; + this.path = path; + this.selected = selected; + this.cache = cache; + } + + @Override + public int read() throws IOException { + ensureOpen(); + if (headerPosition < selected.header.length) { + return selected.header[headerPosition++] & 255; + } + return fillBuffer() ? buffer[bufferPosition++] & 255 : -1; + } + + @Override + public int read(byte[] bytes, int offset, int length) throws IOException { + ensureOpen(); + if (length == 0) { + return 0; + } + if (headerPosition < selected.header.length) { + int n = Math.min(length, selected.header.length - headerPosition); + System.arraycopy(selected.header, headerPosition, bytes, offset, n); + headerPosition += n; + return n; + } + if (!fillBuffer()) { + return -1; + } + int copied = Math.min(length, bufferLimit - bufferPosition); + System.arraycopy(buffer, bufferPosition, bytes, offset, copied); + bufferPosition += copied; + return copied; + } + + private boolean fillBuffer() throws IOException { + if (bufferPosition < bufferLimit) { + return true; + } + if (remaining == 0) { + if (blockPosition == selected.blocks.size()) { + return false; + } + Block next = selected.blocks.get(blockPosition); + if (cache != null && next.length <= cache.maxElementSize()) { + readCachedBlocks(next); + return true; + } + Block block = selected.blocks.get(blockPosition++); + long end = block.offset + block.length; + while (blockPosition < selected.blocks.size() + && selected.blocks.get(blockPosition).offset == end + && (cache == null + || selected.blocks.get(blockPosition).length + > cache.maxElementSize())) { + end += selected.blocks.get(blockPosition++).length; + } + seekInput(block.offset); + remaining = end - block.offset; + } + int requested = (int) Math.min(READ_BUFFER_BYTES, remaining); + // A previous buffer may be shared with other readers through the block cache. + if (cache != null || buffer == null || buffer.length < requested) { + buffer = new byte[requested]; + } + bufferPosition = 0; + bufferLimit = 0; + readFully(buffer, requested); + bufferLimit = requested; + remaining -= requested; + return true; + } + + private void readCachedBlocks(Block first) throws IOException { + byte[] cached = cachedBlock(first); + if (cached != null) { + blockPosition++; + buffer = cached; + } else { + int firstPosition = blockPosition++; + long end = first.offset + first.length; + while (blockPosition < selected.blocks.size()) { + Block next = selected.blocks.get(blockPosition); + if (next.offset != end + || next.length > cache.maxElementSize() + || end - first.offset + next.length > READ_BUFFER_BYTES + || cachedBlock(next) != null) { + break; + } + end += next.length; + blockPosition++; + } + + byte[] bytes = new byte[(int) (end - first.offset)]; + seekInput(first.offset); + readFully(bytes, bytes.length); + // Publish only complete reads, and use individual block keys so overlapping + // selections can share data even when their coalesced read spans differ. + int offset = 0; + for (int i = firstPosition; i < blockPosition; i++) { + Block block = selected.blocks.get(i); + int length = (int) block.length; + byte[] blockBytes = + length == bytes.length + ? bytes + : Arrays.copyOfRange(bytes, offset, offset + length); + cache.put( + new BlockCacheKey(path, block.offset, block.length), + new ManifestBlockSegment(blockBytes)); + offset += length; + } + buffer = bytes; + } + bufferPosition = 0; + bufferLimit = buffer.length; + } + + @Nullable + private byte[] cachedBlock(Block block) { + Segments cached = + cache.getIfPresents(new BlockCacheKey(path, block.offset, block.length)); + if (cached instanceof ManifestBlockSegment) { + byte[] bytes = ((ManifestBlockSegment) cached).bytes; + if (bytes.length == block.length) { + return bytes; + } + } + return null; + } + + private void seekInput(long offset) throws IOException { + if (input == null) { + input = io.newInputStream(path); + } + input.seek(offset); + } + + private void readFully(byte[] bytes, int length) throws IOException { + int position = 0; + while (position < length) { + int count = + input.read(bytes, position, Math.min(READ_BUFFER_BYTES, length - position)); + if (count < 0) { + throw new EOFException("Truncated manifest block"); + } + position += count; + } + } + + private void ensureOpen() throws IOException { + if (closed) { + throw new IOException("Manifest stream is closed"); + } + } + + @Override + public void close() throws IOException { + if (!closed) { + closed = true; + if (input != null) { + input.close(); + } + } + } + } + + private static void require(boolean valid) throws IOException { + if (!valid) { + throw new IOException( + "Invalid, unsupported, mismatched or over-budget manifest sidecar"); + } + } + + private static byte[] digest(byte[] bytes) { + return digest(bytes, bytes.length); + } + + private static byte[] digest(byte[] bytes, int length) { + try { + MessageDigest digest = MessageDigest.getInstance("SHA-256"); + digest.update(bytes, 0, length); + return digest.digest(); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException(e); + } + } +} diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java new file mode 100644 index 000000000000..ba1debc9f984 --- /dev/null +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java @@ -0,0 +1,803 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.manifest; + +import org.apache.paimon.data.BinaryRow; +import org.apache.paimon.data.BinaryRowWriter; +import org.apache.paimon.data.BinaryString; +import org.apache.paimon.options.MemorySize; +import org.apache.paimon.options.Options; +import org.apache.paimon.partition.PartitionPredicate; +import org.apache.paimon.predicate.PredicateBuilder; +import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.RowType; +import org.apache.paimon.utils.Range; +import org.apache.paimon.utils.RowRangeIndex; +import org.apache.paimon.utils.SerializationUtils; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.security.MessageDigest; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Base64; +import java.util.Collections; +import java.util.List; +import java.util.Properties; +import java.util.function.BiPredicate; + +import static org.apache.paimon.manifest.ManifestSidecarTest.MAX_BYTES; +import static org.apache.paimon.manifest.ManifestSidecarTest.meta; +import static org.apache.paimon.manifest.ManifestSidecarTest.settings; +import static org.apache.paimon.manifest.ManifestSidecarTest.sidecarOptions; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.verifyNoMoreInteractions; + +/** Independent partition/row-ID payloads and conservative resource degradation. */ +class ManifestBlockIndexTest { + private final RowType type = RowType.of(DataTypes.INT(), DataTypes.STRING()); + private final ManifestSidecar.Settings defaults = settings(sidecarOptions(), 2); + + private byte[] fixture(String field) throws IOException { + Properties p = new Properties(); + try (java.io.InputStream in = getClass().getResourceAsStream("/manifest-sidecar.txt")) { + p.load(in); + } + return Base64.getDecoder().decode(p.getProperty(field)); + } + + private byte[] partition(int p, String q) { + BinaryRow row = new BinaryRow(2); + BinaryRowWriter writer = new BinaryRowWriter(row); + writer.writeInt(0, p); + if (q == null) { + writer.setNullAt(1); + } else { + writer.writeString(1, BinaryString.fromString(q)); + } + writer.complete(); + return SerializationUtils.serializeBinaryRow(row); + } + + private RowRangeIndex query(long point) { + return RowRangeIndex.create(Collections.singletonList(new Range(point, point))); + } + + private static BiPredicate bucketFilter(int expected) { + return new BiPredicate() { + @Override + public boolean test(Integer bucket, Integer totalBuckets) { + return bucket == expected; + } + }; + } + + private PartitionPredicate part(int value) { + return PartitionPredicate.fromPredicate(type, new PredicateBuilder(type).equal(0, value)); + } + + @Test + void payloadGenerationCanBeDisabledIndependently() throws Exception { + byte[] header = fixture("avroHeader"); + for (int mask = 0; mask < 8; mask++) { + boolean partitionEnabled = (mask & 1) != 0; + boolean rowIdEnabled = (mask & 2) != 0; + boolean bucketEnabled = (mask & 4) != 0; + ManifestSidecar.Settings settings = + new ManifestSidecar.Settings( + 16 * 1024 * 1024L, partitionEnabled, rowIdEnabled, bucketEnabled); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + for (int block = 0; block < 2; block++) { + builder.beginBlock(header.length + block * 100L, 100, 1); + builder.add(100L + block * 100L, 10, partition(7 + block, "p"), 1, 4); + builder.endBlock(); + } + byte[] data = builder.serialize("m", header.length + 200, 2); + assertThat(ByteBuffer.wrap(data).getInt(64 + header.length)) + .isEqualTo(partitionEnabled ? 2 : 0); + for (int[] position : positions(data)) { + assertThat(data[position[1]]).isEqualTo((byte) (partitionEnabled ? 1 : 0)); + assertThat(data[position[2]]).isEqualTo((byte) (rowIdEnabled ? 1 : 0)); + assertThat(data[position[3]]).isEqualTo((byte) (bucketEnabled ? 1 : 0)); + } + ManifestFileMeta meta = meta("m", header.length + 200, 2); + assertThat(ManifestSidecar.select(data, meta, query(999), settings).blocks()) + .hasSize(rowIdEnabled ? 0 : 2); + assertThat(ManifestSidecar.select(data, meta, null, part(99), type, settings).blocks()) + .hasSize(partitionEnabled ? 0 : 2); + BiPredicate buckets = bucketFilter(99); + assertThat( + ManifestSidecar.select(data, meta, null, null, type, buckets, settings) + .blocks()) + .hasSize(bucketEnabled ? 0 : 2); + + // Generation settings do not disable payloads already stored in a sidecar. + byte[] existing = fixture("indexWithBuckets"); + ManifestFileMeta existingMeta = meta("manifest-golden", header.length + 400, 7); + assertThat( + ManifestSidecar.select(existing, existingMeta, query(999), settings) + .blocks()) + .isEmpty(); + assertThat( + ManifestSidecar.select( + existing, existingMeta, null, part(99), type, settings) + .blocks()) + .isEmpty(); + assertThat( + ManifestSidecar.select( + existing, + existingMeta, + null, + null, + type, + buckets, + settings) + .blocks()) + .isEmpty(); + } + } + + @Test + void jointGoldenPreservesTuplesNullsAndDerivedOrdinals() throws Exception { + byte[] a = partition(7, "left"); + byte[] b = partition(9, null); + assertThat(a).isEqualTo(fixture("partitionA")); + assertThat(b).isEqualTo(fixture("partitionB")); + byte[] header = fixture("avroHeader"); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + builder.beginBlock(header.length, 100, 3); + builder.add(0L, 10, a); + builder.add(5L, 5, a); + builder.add(20L, 5, b); + builder.endBlock(); + builder.beginBlock(header.length + 100, 200, 2); + builder.add((1L << 32) - 2, 5, b); + builder.add(8254058425445L, 1, a); + builder.endBlock(); + builder.beginBlock(header.length + 300, 100, 2); + builder.add(20L, 5, a); + builder.add(Long.MAX_VALUE, 1, b); + builder.endBlock(); + byte[] data = builder.serialize("manifest-golden", header.length + 400, 7); + assertThat(data).isEqualTo(fixture("indexWithPartitions")); + ManifestFileMeta meta = meta("manifest-golden", header.length + 400, 7); + PartitionPredicate filter = spy(part(7)); + assertThat(ManifestSidecar.select(data, meta, query(20), filter, type, defaults).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L, 5L); + verify(filter, times(2)).test(any(BinaryRow.class)); + PartitionPredicate nullFilter = + PartitionPredicate.fromPredicate(type, new PredicateBuilder(type).isNull(1)); + assertThat(ManifestSidecar.select(data, meta, null, nullFilter, type, defaults).blocks()) + .hasSize(3); + assertThat(ManifestSidecar.select(data, meta, null, part(99), type, defaults).blocks()) + .isEmpty(); + // Missing partition payloads cannot be pruned by dictionary misses. + assertThat( + ManifestSidecar.select( + fixture("index"), meta, null, part(99), type, defaults) + .blocks()) + .hasSize(3); + } + + @Test + void unavailableDimensionsAreIndependentAndDoNotPoisonLaterBlocks() throws Exception { + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(512)); + ManifestSidecar.Settings settings = settings(options, 2); + byte[] header = fixture("avroHeader"); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 1); + builder.add(null, 10, partition(7, "left")); + builder.endBlock(); + builder.beginBlock(header.length + 100, 100, 1); + builder.add(200L, 10, partition(9, String.join("", Collections.nCopies(600, "x")))); + builder.endBlock(); + builder.beginBlock(header.length + 200, 100, 1); + builder.add(300L, 10, partition(7, "left")); // an existing dictionary ID remains usable + builder.endBlock(); + byte[] data = builder.serialize("m", header.length + 300, 3); + ManifestFileMeta meta = meta("m", header.length + 300, 3); + assertThat(ManifestSidecar.select(data, meta, null, part(9), type, settings).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(1L); + assertThat(ManifestSidecar.select(data, meta, query(999), part(7), type, settings).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + assertThat(ManifestSidecar.select(data, meta, query(200), part(9), type, settings).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(1L); + } + + @Test + void coarseningContinuesThroughTheEntireBlockAndDetectsUnknownRows() throws Exception { + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(512)); + ManifestSidecar.Settings settings = settings(options, 2); + byte[] header = fixture("avroHeader"); + for (boolean unknown : new boolean[] {false, true}) { + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 66); + for (int i = 0; i < 64; i++) { + builder.add(100L + i * 1000L, 10, partition(7, "left")); + } + builder.add(10L, 10, partition(7, "left")); + builder.add(unknown ? null : Long.MAX_VALUE, 1, partition(7, "left")); + builder.endBlock(); + byte[] data = builder.serialize("m", header.length + 100, 66); + ManifestFileMeta meta = meta("m", header.length + 100, 66); + for (long point : new long[] {10, 100, 200, Long.MAX_VALUE}) { + assertThat(ManifestSidecar.select(data, meta, query(point), settings).blocks()) + .hasSize(1); + } + assertThat(ManifestSidecar.select(data, meta, query(0), settings).blocks()) + .hasSize(unknown ? 1 : 0); + assertThat(ManifestSidecar.select(data, meta, null, part(9), type, settings).blocks()) + .isEmpty(); + } + } + + private List positions(byte[] data) { + ByteBuffer in = ByteBuffer.wrap(data); + in.position(60); + int header = in.getInt(); + in.position(in.position() + header); + int partitions = in.getInt(); + for (int i = 0; i < partitions; i++) { + int length = in.getInt(); + in.position(in.position() + length); + } + int blocks = in.getInt(); + List result = new ArrayList<>(); + for (int i = 0; i < blocks; i++) { + int block = in.position(); + in.position(block + 24); + int partition = skipPayload(in); + int row = skipPayload(in); + int bucket = skipPayload(in); + result.add(new int[] {block, partition, row, bucket}); + } + return result; + } + + private int skipPayload(ByteBuffer in) { + int position = in.position(); + if (in.get() != 0) { + int length = in.getInt(); + in.position(in.position() + length); + } + return position; + } + + private byte[] checksum(byte[] data) throws Exception { + byte[] hash = + MessageDigest.getInstance("SHA-256").digest(Arrays.copyOf(data, data.length - 32)); + System.arraycopy(hash, 0, data, data.length - 32, 32); + return data; + } + + @Test + void rowMissSkipsPartitionAndBucketPayloads() throws Exception { + byte[] data = fixture("indexWithBuckets"); + int[] first = positions(data).get(0); + ByteBuffer.wrap(data).putInt(first[1] + 9, -1); + ByteBuffer.wrap(data).putInt(first[3] + 9, -1); + checksum(data); + ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + BiPredicate buckets = mock(BiPredicate.class); + assertThat( + ManifestSidecar.select( + data, meta, query(15), part(7), type, buckets, defaults) + .blocks()) + .isEmpty(); + verifyNoInteractions(buckets); + } + + @Test + void partitionMissSkipsBucketMatchingWithOrWithoutRowFilter() throws Exception { + byte[] data = fixture("indexWithBuckets"); + ByteBuffer.wrap(data).putInt(positions(data).get(0)[3] + 9, -1); + checksum(data); + ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + for (RowRangeIndex rows : Arrays.asList(null, query(0))) { + BiPredicate buckets = mock(BiPredicate.class); + assertThat( + ManifestSidecar.select( + data, meta, rows, part(99), type, buckets, defaults) + .blocks()) + .isEmpty(); + verifyNoInteractions(buckets); + } + } + + @Test + void absentPartitionFilterKeepsRowAndBucketMatching() throws Exception { + byte[] data = fixture("indexWithBuckets"); + ByteBuffer.wrap(data).putInt(positions(data).get(0)[1] + 9, 999); + checksum(data); + ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + BiPredicate buckets = spy(bucketFilter(1)); + assertThat( + ManifestSidecar.select(data, meta, query(20), null, type, buckets, defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + verify(buckets).test(1, 4); + verify(buckets).test(0, 1); + verify(buckets).test(3, 4); + verifyNoMoreInteractions(buckets); + } + + @Test + void absentRowOrBucketFiltersKeepRemainingDimensions() throws Exception { + byte[] data = fixture("indexWithBuckets"); + ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + assertThat( + ManifestSidecar.select( + data, meta, null, part(7), type, bucketFilter(1), defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + assertThat( + ManifestSidecar.select(data, meta, query(20), part(7), type, null, defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L, 5L); + assertThat(ManifestSidecar.select(data, meta, null, null, type, null, defaults).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L, 3L, 5L); + } + + @Test + void partitionAndBucketMatchesSkipUnusedPayloadElements() throws Exception { + byte[] partitions = fixture("indexWithBuckets"); + int[] first = positions(partitions).get(0); + ByteBuffer.wrap(partitions).putInt(first[1] + 13, -1); + checksum(partitions); + ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + assertThat( + ManifestSidecar.select(partitions, meta, query(0), part(7), type, defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + assertThatThrownBy( + () -> + ManifestSidecar.select( + partitions, meta, query(0), part(99), type, defaults)) + .isInstanceOf(IOException.class); + + byte[] buckets = fixture("indexWithBuckets"); + ByteBuffer.wrap(buckets).putInt(first[3] + 17, -1); + checksum(buckets); + assertThat( + ManifestSidecar.select( + buckets, + meta, + query(0), + null, + type, + bucketFilter(1), + defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + assertThatThrownBy( + () -> + ManifestSidecar.select( + buckets, + meta, + query(0), + null, + type, + bucketFilter(99), + defaults)) + .isInstanceOf(IOException.class); + } + + @Test + void skippedPayloadsStillRequireValidFramingAndDirectory() throws Exception { + byte[] good = fixture("indexWithBuckets"); + int[] first = positions(good).get(0); + ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + List invalid = new ArrayList<>(); + byte[] bad = good.clone(); + ByteBuffer.wrap(bad).putInt(first[1] + 5, 0); + invalid.add(bad); + bad = good.clone(); + ByteBuffer.wrap(bad).putInt(first[3] + 1, -1); + invalid.add(bad); + bad = good.clone(); + ByteBuffer.wrap(bad).putInt(first[2] + 5, 0); + invalid.add(bad); + bad = good.clone(); + ByteBuffer.wrap(bad).putLong(positions(good).get(1)[0], 0); + invalid.add(bad); + bad = good.clone(); + ByteBuffer.wrap(bad).putLong(first[0] + 16, 2); + invalid.add(bad); + for (byte[] data : invalid) { + checksum(data); + assertThatThrownBy( + () -> + ManifestSidecar.select( + data, meta, query(15), part(7), type, defaults)) + .isInstanceOf(IOException.class); + } + } + + @Test + void unknownUnsignedEncodingsSkipOnlyTheirDimensionAndMalformedPayloadsFail() throws Exception { + byte[] good = fixture("indexWithBuckets"); + int[] first = positions(good).get(0); + ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + byte[] data = good.clone(); + data[first[1]] = (byte) 200; + assertThat( + ManifestSidecar.select( + checksum(data), meta, query(0), part(99), type, defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + data = good.clone(); + data[first[2]] = (byte) 201; + assertThat( + ManifestSidecar.select( + checksum(data), meta, query(16), part(7), type, defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + data = good.clone(); + data[first[3]] = (byte) 202; + // Unknown encodings skip their payload without decoding even an invalid pair count. + ByteBuffer.wrap(data).putInt(first[3] + 5, 0); + checksum(data); + BiPredicate noBucket = bucketFilter(99); + assertThat( + ManifestSidecar.select( + data, meta, query(20), part(7), type, noBucket, defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + assertThat( + ManifestSidecar.select( + data, meta, query(999), part(7), type, noBucket, defaults) + .blocks()) + .isEmpty(); + assertThat( + ManifestSidecar.select( + data, meta, query(20), part(99), type, noBucket, defaults) + .blocks()) + .isEmpty(); + for (int position : new int[] {first[1], first[2], first[3]}) { + byte[] bad = good.clone(); + bad[position] = 0; // encoding 0 cannot have a length or payload bytes + checksum(bad); + assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, query(0), defaults)) + .isInstanceOf(IOException.class); + byte[] invalidLength = good.clone(); + invalidLength[position] = (byte) 255; + ByteBuffer.wrap(invalidLength).putInt(position + 1, -1); + checksum(invalidLength); + assertThatThrownBy( + () -> ManifestSidecar.select(invalidLength, meta, query(0), defaults)) + .isInstanceOf(IOException.class); + } + // A checksummed directory with missing bytes/entries must still be rejected. + byte[] bad = good.clone(); + ByteBuffer.wrap(bad).putLong(first[0] + 16, 2); + checksum(bad); + assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, query(0), defaults)) + .isInstanceOf(IOException.class); + byte[] badRange = good.clone(); + // Row payload begins after its encoding and length, then the range-count integer. + ByteBuffer.wrap(badRange).putLong(first[2] + 9 + 16, 9L); + checksum(badRange); + assertThatThrownBy( + () -> + ManifestSidecar.select( + badRange, meta, query(15), part(7), type, defaults)) + .isInstanceOf(IOException.class); + byte[] badId = good.clone(); + ByteBuffer.wrap(badId).putInt(first[1] + 9, 999); + checksum(badId); + assertThatThrownBy( + () -> + ManifestSidecar.select( + badId, meta, query(0), part(7), type, defaults)) + .isInstanceOf(IOException.class); + } + + @Test + void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { + byte[] header = fixture("avroHeader"); + byte[] a = partition(7, "left"); + byte[] b = partition(9, null); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + builder.beginBlock(header.length, 100, 3); + builder.add(0L, 10, a, 1, 4); + builder.add(5L, 5, a, 1, 4); + builder.add(20L, 5, b, 1, 8); + builder.endBlock(); + builder.beginBlock(header.length + 100, 200, 2); + builder.add((1L << 32) - 2, 5, b, 2, 4); + builder.add(8254058425445L, 1, a, 2, 8); + builder.endBlock(); + builder.beginBlock(header.length + 300, 100, 2); + builder.add(20L, 5, a, 0, 1); + builder.add(Long.MAX_VALUE, 1, b, 3, 4); + builder.endBlock(); + byte[] data = builder.serialize("manifest-golden", header.length + 400, 7); + assertThat(data).isEqualTo(fixture("indexWithBuckets")); + ManifestFileMeta meta = meta("manifest-golden", header.length + 400, 7); + BiPredicate bucket = bucketFilter(1); + assertThat(ManifestSidecar.select(data, meta, null, null, type, bucket, defaults).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + // Existing row-id and partition coverage remains independently usable. + assertThat( + ManifestSidecar.select( + data, + meta, + query(0), + part(7), + type, + bucketFilter(2), + defaults) + .blocks()) + .isEmpty(); + BiPredicate filter = (bucketId, total) -> bucketId == 2 && total == 8; + assertThat(ManifestSidecar.select(data, meta, null, null, type, filter, defaults).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(3L); + // The caller can omit a bucket filter which requires an actual entry partition. + assertThat(ManifestSidecar.select(data, meta, null, null, type, null, defaults).blocks()) + .hasSize(3); + for (String unavailable : new String[] {"index", "indexWithPartitions"}) { + assertThat( + ManifestSidecar.select( + fixture(unavailable), + meta, + null, + null, + type, + bucketFilter(99), + defaults) + .blocks()) + .hasSize(3); + } + } + + @Test + void unknownInvalidOrOverBudgetBucketPayloadIsUnavailable() throws Exception { + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(512)); + ManifestSidecar.Settings settings = settings(options, 2); + byte[] header = fixture("avroHeader"); + for (Integer[] pair : + Arrays.asList( + new Integer[] {null, null}, + new Integer[] {-1, 4}, + new Integer[] {4, 4}, + new Integer[] {0, 0}, + new Integer[] {2, 8})) { + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + int extraPairs = pair[0] != null && pair[0] == 2 ? 65 : 0; + builder.beginBlock(header.length, 100, 2 + extraPairs); + builder.add(100L, 10, partition(7, "left"), 1, 4); + builder.add(200L, 10, partition(7, "left"), pair[0], pair[1]); + for (int i = 0; i < extraPairs; i++) { + builder.add(200L, 10, partition(7, "left"), i, 100); + } + builder.endBlock(); + builder.beginBlock(header.length + 100, 100, 1); + builder.add(300L, 10, partition(7, "left"), 1, 4); + builder.endBlock(); + byte[] data = builder.serialize("m", header.length + 200, 3 + extraPairs); + int bucket = positions(data).get(0)[3]; + assertThat(data[bucket]).isZero(); + assertThat(positions(data).get(1)[0]).isEqualTo(bucket + 1); + ManifestFileMeta meta = meta("m", header.length + 200, 3 + extraPairs); + assertThat( + ManifestSidecar.select( + data, + meta, + null, + null, + type, + bucketFilter(99), + settings) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + assertThat( + ManifestSidecar.select( + data, + meta, + query(999), + null, + type, + bucketFilter(99), + settings) + .blocks()) + .isEmpty(); + } + } + + @Test + void malformedBucketPayloadInvalidatesTheContainer() throws Exception { + byte[] good = fixture("indexWithBuckets"); + int payload = positions(good).get(0)[3] + 1; + ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + for (int[] mutation : + new int[][] { + {payload, -2}, + {payload, Integer.MAX_VALUE}, + {payload, 0}, + {payload + 4, 0}, + {payload + 8, -1}, + {payload + 12, 1}, + {payload + 16, 0} + }) { + byte[] bad = good.clone(); + ByteBuffer.wrap(bad).putInt(mutation[0], mutation[1]); + checksum(bad); + assertThatThrownBy( + () -> + ManifestSidecar.select( + bad, + meta, + null, + null, + type, + bucketFilter(99), + defaults)) + .isInstanceOf(IOException.class); + } + } + + @Test + void payloadsCanExceedFormerLimitsWithinByteBudget() throws Exception { + byte[] header = fixture("avroHeader"); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + int blocks = 33; + int entriesPerBlock = 4097; + int entries = blocks * entriesPerBlock; + int blockBytes = 1024 * 1024; + for (int block = 0; block < blocks; block++) { + builder.beginBlock( + header.length + (long) block * blockBytes, blockBytes, entriesPerBlock); + for (int i = 0; i < entriesPerBlock; i++) { + int entry = block * entriesPerBlock + i; + builder.add(entry * 2L, 1, partition(entry, null), i, entriesPerBlock + 1); + } + builder.endBlock(); + } + long fileSize = header.length + (long) blocks * blockBytes; + byte[] data = builder.serialize("m", fileSize, entries); + assertThat(data.length).isLessThanOrEqualTo(defaults.maxBytes); + ManifestFileMeta meta = meta("m", fileSize, entries); + long last = (entries - 1L) * 2; + assertThat(ManifestSidecar.select(data, meta, query(last), defaults).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly((blocks - 1L) * entriesPerBlock); + assertThat(ManifestSidecar.select(data, meta, query(last - 1), defaults).blocks()) + .isEmpty(); + assertThat(ManifestSidecar.select(data, meta, null, part(entries), type, defaults).blocks()) + .isEmpty(); + assertThat( + ManifestSidecar.select( + data, + meta, + null, + null, + type, + bucketFilter(entriesPerBlock), + defaults) + .blocks()) + .isEmpty(); + } + + @Test + void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Exception { + byte[] header = fixture("avroHeader"); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + for (int mask = 0; mask < 8; mask++) { + builder.beginBlock(header.length + mask * 100L, 100, 1); + builder.add( + (mask & 2) == 0 ? null : 100L + mask, + 1, + (mask & 1) == 0 ? null : partition(7, "left"), + (mask & 4) == 0 ? null : 0, + (mask & 4) == 0 ? null : 1); + builder.endBlock(); + } + byte[] data = builder.serialize("m", header.length + 800, 8); + List positions = positions(data); + int[] presentSizes = {13, 25, 17}; + for (int mask = 0; mask < 8; mask++) { + for (int dimension = 0; dimension < 3; dimension++) { + int start = positions.get(mask)[dimension + 1]; + int end = + dimension < 2 + ? positions.get(mask)[dimension + 2] + : mask < 7 ? positions.get(mask + 1)[0] : data.length - 32; + boolean present = (mask & (1 << dimension)) != 0; + assertThat(data[start]).isEqualTo((byte) (present ? 1 : 0)); + assertThat(end - start).isEqualTo(present ? presentSizes[dimension] : 1); + } + } + ManifestFileMeta meta = meta("m", header.length + 800, 8); + assertThat(ManifestSidecar.select(data, meta, null, part(99), type, defaults).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L, 2L, 4L, 6L); + assertThat(ManifestSidecar.select(data, meta, query(999), defaults).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L, 1L, 4L, 5L); + BiPredicate buckets = bucketFilter(99); + assertThat(ManifestSidecar.select(data, meta, null, null, type, buckets, defaults).blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L, 1L, 2L, 3L); + assertThat( + ManifestSidecar.select( + data, meta, query(999), part(99), type, buckets, defaults) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + } + + @Test + void tightByteBudgetKeepsAllDescriptorsOrOmitsTheWholeFile() throws Exception { + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(250)); + ManifestSidecar.Settings settings = settings(options, 2); + byte[] header = fixture("avroHeader"); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + for (int i = 0; i < 3; i++) { + builder.beginBlock(header.length + 100L * i, 100, 1); + builder.add(i * 100L, 10, partition(7, "left")); + builder.endBlock(); + } + byte[] data = builder.serialize("m", header.length + 300, 3); + assertThat(data.length).isLessThanOrEqualTo(250); + assertThat( + ManifestSidecar.select( + data, + meta("m", header.length + 300, 3), + query(999), + part(99), + type, + settings) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L, 1L, 2L); + builder.beginBlock(header.length + 300, 100, 1); + builder.add(300L, 1, partition(7, "left")); + builder.endBlock(); + assertThat(builder.serialize("m", header.length + 400, 4)).isNull(); + } +} diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java new file mode 100644 index 000000000000..d00f740ec111 --- /dev/null +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -0,0 +1,1145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.manifest; + +import org.apache.paimon.CoreOptions; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.data.SingleSegments; +import org.apache.paimon.format.FileFormat; +import org.apache.paimon.format.avro.AvroFileFormat; +import org.apache.paimon.fs.ByteArraySeekableStream; +import org.apache.paimon.fs.FileIO; +import org.apache.paimon.fs.Path; +import org.apache.paimon.fs.local.LocalFileIO; +import org.apache.paimon.manifest.ManifestSidecar.ManifestSidecarSegment; +import org.apache.paimon.memory.MemorySegment; +import org.apache.paimon.options.ConfigOption; +import org.apache.paimon.options.ConfigOptions; +import org.apache.paimon.options.MemorySize; +import org.apache.paimon.options.Options; +import org.apache.paimon.schema.FileSystemSchemaManager; +import org.apache.paimon.utils.CloseableIterator; +import org.apache.paimon.utils.IOUtils; +import org.apache.paimon.utils.PathFactory; +import org.apache.paimon.utils.Range; +import org.apache.paimon.utils.RowRangeIndex; +import org.apache.paimon.utils.SegmentsCache; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.ByteArrayOutputStream; +import java.io.EOFException; +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.file.Files; +import java.security.MessageDigest; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Base64; +import java.util.Collections; +import java.util.List; +import java.util.Properties; +import java.util.concurrent.CancellationException; + +import static org.apache.paimon.TestKeyValueGenerator.DEFAULT_PART_TYPE; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +/** Cross-language format, physical block positions, completeness and allocation bounds. */ +class ManifestSidecarTest { + static final ConfigOption MAX_BYTES = + ConfigOptions.key("test.sidecar.max-bytes") + .memoryType() + .defaultValue(MemorySize.ofMebiBytes(16)); + + static ManifestSidecar.Settings settings(Options options, int partitions) { + CoreOptions core = new CoreOptions(options); + return new ManifestSidecar.Settings( + options.get(MAX_BYTES).getBytes(), + partitions > 0, + core.dataEvolutionEnabled(), + core.bucket() != -1); + } + + @TempDir java.nio.file.Path temp; + private final ManifestSidecar.Settings settings = settings(sidecarOptions(), 2); + + static Options sidecarOptions() { + Options options = new Options(); + options.set(CoreOptions.DATA_EVOLUTION_ENABLED, true); + options.set(CoreOptions.BUCKET, 4); + return options; + } + + static ManifestFileMeta meta(String name, long size, long entries) { + ManifestFileMeta meta = mock(ManifestFileMeta.class); + when(meta.fileName()).thenReturn(name); + when(meta.fileSize()).thenReturn(size); + when(meta.extraFiles()) + .thenReturn(Collections.singletonList(name + ManifestSidecar.SUFFIX)); + when(meta.numAddedFiles()).thenReturn(entries); + return meta; + } + + private Properties fixture() throws IOException { + Properties properties = new Properties(); + try (java.io.InputStream input = getClass().getResourceAsStream("/manifest-sidecar.txt")) { + properties.load(input); + } + return properties; + } + + private byte[] header() throws IOException { + return Base64.getDecoder().decode(fixture().getProperty("avroHeader")); + } + + private byte[] golden() throws IOException { + return Base64.getDecoder().decode(fixture().getProperty("index")); + } + + private ManifestFileMeta goldenMeta() throws IOException { + return meta("manifest-golden", header().length + 400, 7); + } + + @Test + void buildAndReadSelectedBlocksFromPhysicalManifests() throws Exception { + FileIO io = LocalFileIO.create(); + ManifestTestDataGenerator generator = ManifestTestDataGenerator.builder().build(); + List entries = new ArrayList<>(); + for (int i = 0; i < 4000; i++) { + ManifestEntry entry = generator.next(); + entries.add( + ManifestEntry.create( + i % 2 == 0 ? FileKind.ADD : FileKind.DELETE, + entry.partition(), + 1, + 4, + entry.file().newFirstRowId(i * 1000000L))); + } + Path sourcePath = new Path(temp.toString(), "manifest-source"); + ManifestFileMeta source; + try (ManifestAvroWriter writer = writer(io, sourcePath)) { + writer.write(entries); + writer.close(); + source = writer.result().get(0); + } + Path rewrittenPath = new Path(temp.toString(), "manifest-rewritten"); + ManifestFileMeta rewritten; + try (ManifestAvroWriter writer = writer(io, rewrittenPath); + ManifestAvroReader reader = new ManifestAvroReader(io.newInputStream(sourcePath))) { + writer.writeEncodedManifest(reader, source); + writer.close(); + rewritten = writer.result().get(0); + } + for (ManifestFileMeta meta : Arrays.asList(source, rewritten)) { + Path path = new Path(temp.toString(), meta.fileName()); + byte[] data = + ManifestSidecar.build(io, path, meta.fileSize(), entries.size(), settings); + ManifestSidecar.Selection selected = + ManifestSidecar.select( + data, + meta, + RowRangeIndex.create( + Arrays.asList( + new Range(1000000000L, 1000000000L), + new Range(3000000000L, 3000000000L))), + null, + DEFAULT_PART_TYPE, + (bucket, totalBuckets) -> bucket == 1 && totalBuckets == 4, + settings); + assertThat(selected.blocks()).hasSize(2); + List expected = new ArrayList<>(); + for (ManifestSidecar.Block block : selected.blocks()) { + expected.addAll( + entries.subList( + (int) block.firstRecord, + (int) (block.firstRecord + block.recordCount))); + } + List actual = new ArrayList<>(); + try (ManifestAvroReader reader = + new ManifestAvroReader( + ManifestSidecar.openManifest(io, path, selected)); + CloseableIterator rows = + reader.read(ManifestEntry.MANIFEST_ROW_TYPE, null, null)) { + ManifestEntrySerializer serializer = new ManifestEntrySerializer(); + while (rows.hasNext()) { + actual.add(serializer.fromRow(rows.next())); + } + } + assertThat(actual).containsExactlyElementsOf(expected); + } + } + + private ManifestAvroWriter writer(FileIO io, Path path) { + PathFactory paths = mock(PathFactory.class); + when(paths.newPath()).thenReturn(path); + return new ManifestAvroWriter( + io, + new FileSystemSchemaManager(io, new Path(temp.toUri())), + DEFAULT_PART_TYPE, + (AvroFileFormat) FileFormat.fromIdentifier("avro", new Options()), + new ManifestEntrySerializer(), + "zstd", + paths, + Long.MAX_VALUE); + } + + @Test + void crossLanguageFormatAndBlockOrdinals() throws Exception { + byte[] header = header(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 3); + builder.add(0L, 10); + builder.add(5L, 5); + builder.add(20L, 5); + builder.endBlock(); + builder.beginBlock(header.length + 100, 200, 2); + builder.add((1L << 32) - 2, 5); + builder.add(8254058425445L, 1); + builder.endBlock(); + builder.beginBlock(header.length + 300, 100, 2); + builder.add(20L, 5); + builder.add(Long.MAX_VALUE, 1); + builder.endBlock(); + byte[] data = builder.serialize("manifest-golden", header.length + 400, 7); + assertThat(data).isEqualTo(golden()); + ManifestFileMeta meta = goldenMeta(); + for (long point : + new long[] { + 0, + 9, + 20, + 24, + (1L << 32) - 2, + 1L << 32, + (1L << 32) + 2, + 8254058425445L, + Long.MAX_VALUE + }) { + assertThat(select(data, meta, point).blocks()).as("row %s", point).isNotEmpty(); + } + for (long point : + new long[] { + 10, 19, 25, (1L << 32) - 3, (1L << 32) + 3, 8254058425444L, Long.MAX_VALUE - 1 + }) { + assertThat(select(data, meta, point).blocks()).as("row %s", point).isEmpty(); + } + ManifestSidecar.Selection selected = select(data, meta, 20); + assertThat(selected.blocks()).extracting(b -> b.firstRecord).containsExactly(0L, 5L); + assertThat(selected.blocks()) + .extracting(b -> b.offset) + .containsExactly((long) header.length, header.length + 300L); + assertThat(selected.blocks()).extracting(b -> b.length).containsExactly(100L, 100L); + + ManifestSidecar.Selection gap = select(data, meta, 16); + + assertThat(gap.blocks()).isEmpty(); + RowRangeIndex query = + RowRangeIndex.create(Arrays.asList(new Range(10, 19), new Range(25, 40))); + assertThat(ManifestSidecar.select(data, meta, query, settings).blocks()).isEmpty(); + assertThat(query.ranges()).containsExactly(new Range(10, 19), new Range(25, 40)); + } + + @Test + void settingsEnforceByteArraySizeLimit() { + for (long bytes : + new long[] {0, 127, Integer.MAX_VALUE - 1L, Integer.MAX_VALUE, Long.MAX_VALUE}) { + assertThat(new ManifestSidecar.Settings(bytes, true, true, true).maxBytes) + .isEqualTo((int) Math.min(bytes, Integer.MAX_VALUE - 1L)); + } + assertThatThrownBy(() -> new ManifestSidecar.Settings(-1, true, true, true)) + .isInstanceOf(IllegalArgumentException.class); + } + + @Test + void insufficientByteBudgetSkipsSidecarIo() throws Exception { + FileIO io = mock(FileIO.class); + Path path = new Path(temp.toString(), "manifest-golden"); + ManifestFileMeta meta = goldenMeta(); + for (int bytes : new int[] {0, 1, 127}) { + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(bytes)); + ManifestSidecar.Settings settings = settings(options, 2); + assertThat(ManifestSidecar.read(io, path, meta, null, settings)).isNull(); + assertThat(ManifestSidecar.build(io, path, meta.fileSize(), 7, settings)).isNull(); + } + verifyNoInteractions(io); + } + + @Test + void minMaxSkipsExactIntersectionChecksAndHandlesOneInterval() throws Exception { + byte[] header = header(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 2); + builder.add(0L, 10); + builder.add(20L, 10); + builder.endBlock(); + builder.beginBlock(header.length + 100, 100, 2); + builder.add(100L, 10); + builder.add(200L, 10); + builder.endBlock(); + builder.beginBlock(header.length + 200, 100, 1); + builder.add(1L << 32, 10); + builder.endBlock(); + byte[] data = builder.serialize("m", header.length + 300, 5); + ManifestFileMeta meta = meta("m", header.length + 300, 5); + RowRangeIndex outside = + spy(RowRangeIndex.create(Collections.singletonList(new Range(50, 59)))); + ManifestSidecar.Selection none = ManifestSidecar.select(data, meta, outside, settings); + assertThat(none.blocks()).isEmpty(); + + // Only the three envelopes are tested; no individual interval intersection is evaluated. + verify(outside, times(3)).intersects(anyLong(), anyLong()); + verify(outside).intersects(0, 29); + verify(outside).intersects(100, 209); + verify(outside).intersects(1L << 32, (1L << 32) + 9); + + RowRangeIndex one = + spy( + RowRangeIndex.create( + Collections.singletonList( + new Range((1L << 32) + 9, (1L << 32) + 9)))); + ManifestSidecar.Selection hit = ManifestSidecar.select(data, meta, one, settings); + assertThat(hit.blocks()).extracting(b -> b.firstRecord).containsExactly(4L); + + // A one-interval block needs no second intersection check after its envelope matches. + verify(one, times(3)).intersects(anyLong(), anyLong()); + } + + @Test + void singleIntervalHandlesBoundariesAndAbsentQueries() throws Exception { + byte[] header = header(); + for (Range range : + Arrays.asList( + new Range(0, 0), + new Range(42, 51), + new Range(Long.MAX_VALUE, Long.MAX_VALUE))) { + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 1); + builder.add(range.from, range.to - range.from + 1); + builder.endBlock(); + byte[] data = builder.serialize("m", header.length + 100, 1); + ManifestFileMeta meta = meta("m", header.length + 100, 1); + assertThat(ManifestSidecar.select(data, meta, null, settings).blocks()).hasSize(1); + assertThat( + ManifestSidecar.select( + data, + meta, + RowRangeIndex.create(Collections.emptyList()), + settings) + .blocks()) + .isEmpty(); + for (long point : new long[] {range.from, range.to}) { + RowRangeIndex query = + spy( + RowRangeIndex.create( + Collections.singletonList(new Range(point, point)))); + assertThat(ManifestSidecar.select(data, meta, query, settings).blocks()).hasSize(1); + verify(query).intersects(range.from, range.to); + } + long missing = range.from > 0 ? range.from - 1 : range.to + 1; + assertThat(select(data, meta, missing).blocks()).isEmpty(); + } + } + + @Test + void malformedConsumedIntervalsStillFallBack() throws Exception { + int firstBlockIntervals = 60 + 4 + header().length + 4 + 4 + 24 + 1 + 5 + 4; + ManifestFileMeta meta = goldenMeta(); + for (long[] mutation : new long[][] {{0, -1}, {8, -1}, {8, 30}, {16, 9}, {24, 19}}) { + byte[] data = golden(); + ByteBuffer.wrap(data).putLong(firstBlockIntervals + (int) mutation[0], mutation[1]); + byte[] hash = + MessageDigest.getInstance("SHA-256") + .digest(Arrays.copyOf(data, data.length - 32)); + System.arraycopy(hash, 0, data, data.length - 32, 32); + Files.write(temp.resolve("manifest-golden" + ManifestSidecar.SUFFIX), data); + for (RowRangeIndex query : + Collections.singletonList( + RowRangeIndex.create(Collections.singletonList(new Range(15, 15))))) { + assertThat( + ManifestSidecar.read( + LocalFileIO.create(), + new Path(temp.toString(), "manifest-golden"), + meta, + query, + settings)) + .isNull(); + } + } + } + + @Test + void rowBoundsAndMatchesSkipUnusedIntervals() throws Exception { + byte[] header = header(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 3); + builder.add(0L, 10); + builder.add(20L, 10); + builder.add(40L, 10); + builder.endBlock(); + byte[] data = builder.serialize("m", header.length + 100, 3); + int intervals = 60 + 4 + header.length + 4 + 4 + 24 + 1 + 5 + 4; + // A checksummed invalid tail must not be visited once the answer is known. + ByteBuffer.wrap(data).putLong(intervals + 32, 19); + byte[] hash = + MessageDigest.getInstance("SHA-256").digest(Arrays.copyOf(data, data.length - 32)); + System.arraycopy(hash, 0, data, data.length - 32, 32); + ManifestFileMeta meta = meta("m", header.length + 100, 3); + assertThat(select(data, meta, 0).blocks()).hasSize(1); + assertThat(select(data, meta, 20).blocks()).hasSize(1); + assertThat(select(data, meta, 100).blocks()).isEmpty(); + assertThat( + ManifestSidecar.select( + data, + meta, + RowRangeIndex.create(Collections.emptyList()), + settings) + .blocks()) + .isEmpty(); + assertThatThrownBy(() -> select(data, meta, 35)).isInstanceOf(IOException.class); + } + + @Test + void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception { + byte[] header = header(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 2); + builder.add(0L, Long.MAX_VALUE); + builder.add(Long.MAX_VALUE, 1); + builder.endBlock(); + byte[] data = builder.serialize("m", header.length + 100, 2); + assertThat(data.length).isLessThan(512); + assertThat(select(data, meta("m", header.length + 100, 2), Long.MAX_VALUE).blocks()) + .hasSize(1); + for (Long first : Arrays.asList(null, -1L, Long.MAX_VALUE)) { + builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 1); + builder.add(first, 2); + builder.endBlock(); + assertThat( + select( + builder.serialize("m", header.length + 100, 1), + meta("m", header.length + 100, 1), + 100) + .blocks()) + .hasSize(1); + } + for (long count : new long[] {0, -1}) { + builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 1); + builder.add(0L, count); + builder.endBlock(); + assertThat( + select( + builder.serialize("m", header.length + 100, 1), + meta("m", header.length + 100, 1), + 100) + .blocks()) + .hasSize(1); + } + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(512)); + builder = new ManifestSidecar.Builder(settings(options, 2), header); + builder.beginBlock(header.length, 100, 64); + for (int i = 0; i < 64; i++) { + builder.add(i * 10L, 1); + } + builder.endBlock(); + assertThat( + select( + builder.serialize("m", header.length + 100, 64), + meta("m", header.length + 100, 64), + 5) + .blocks()) + .hasSize(1); + options.set(MAX_BYTES, new MemorySize(128)); + builder = new ManifestSidecar.Builder(settings(options, 2), header); + assertThat(builder.serialize("m", 1, 2)).isNull(); + } + + @Test + void cacheRespectsElementThresholdAndPerReadByteBudget() throws Exception { + byte[] data = golden(); + Path path = new Path(temp.toString(), "manifest-golden"); + Path sidecar = ManifestSidecar.path(path); + Files.write(temp.resolve(sidecar.getName()), data); + ManifestFileMeta meta = goldenMeta(); + FileIO io = spy(LocalFileIO.create()); + SegmentsCache tooSmall = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), data.length - 1L, null, false); + assertThat(readCached(io, path, meta, settings, tooSmall).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, settings, tooSmall).blocks()).hasSize(2); + assertThat(tooSmall.getIfPresents(sidecar)).isNull(); + verify(io, times(2)).newInputStream(sidecar); + + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), data.length, null, false); + assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(data.length - 1)); + assertThat(readCached(io, path, meta, settings(options, 2), cache)).isNull(); + assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + verify(io, times(3)).newInputStream(sidecar); + } + + @Test + void cachedSegmentsRequireSidecarType() throws Exception { + byte[] data = golden(); + Path path = new Path(temp.toString(), "manifest-golden"); + Path sidecar = ManifestSidecar.path(path); + Files.write(temp.resolve(sidecar.getName()), data); + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), Long.MAX_VALUE, null, false); + cache.put(sidecar, new SingleSegments(MemorySegment.wrap(data), data.length)); + FileIO io = spy(LocalFileIO.create()); + + assertThat(readCached(io, path, goldenMeta(), settings, cache).blocks()).hasSize(2); + assertThat(cache.getIfPresents(sidecar)).isInstanceOf(ManifestSidecarSegment.class); + ManifestSidecarSegment cached = (ManifestSidecarSegment) cache.getIfPresents(sidecar); + assertThat(cached.bytes()).containsExactly(data); + assertThat(cached.totalMemorySize()).isEqualTo(data.length); + assertThat(readCached(io, path, goldenMeta(), settings, cache).blocks()).hasSize(2); + verify(io, times(1)).newInputStream(sidecar); + } + + @Test + void missingAndInvalidSidecarsAreNotCached() throws Exception { + byte[] data = golden(); + Path path = new Path(temp.toString(), "manifest-golden"); + Path sidecar = ManifestSidecar.path(path); + ManifestFileMeta meta = goldenMeta(); + FileIO io = spy(LocalFileIO.create()); + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), Long.MAX_VALUE, null, false); + assertThat(readCached(io, path, meta, settings, cache)).isNull(); + assertThat(cache.getIfPresents(sidecar)).isNull(); + byte[] corrupt = data.clone(); + corrupt[0] ^= 1; + Files.write(temp.resolve(sidecar.getName()), corrupt); + assertThat(readCached(io, path, meta, settings, cache)).isNull(); + assertThat(cache.getIfPresents(sidecar)).isNull(); + Files.write(temp.resolve(sidecar.getName()), data); + assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + verify(io, times(3)).newInputStream(sidecar); + } + + @Test + void cachedBytesPreservePerQueryCancellation() throws Exception { + Path path = new Path(temp.toString(), "manifest-golden"); + Path sidecar = ManifestSidecar.path(path); + Files.write(temp.resolve(sidecar.getName()), golden()); + ManifestFileMeta meta = goldenMeta(); + FileIO io = spy(LocalFileIO.create()); + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), Long.MAX_VALUE, null, false); + RowRangeIndex cancelled = mock(RowRangeIndex.class); + when(cancelled.intersects(anyLong(), anyLong())) + .thenThrow(new CancellationException("cancelled")); + assertThatThrownBy( + () -> + ManifestSidecar.read( + io, path, meta, cancelled, null, null, null, settings, + cache)) + .isInstanceOf(CancellationException.class); + assertThat(cache.getIfPresents(sidecar)).isNull(); + + assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + assertThatThrownBy( + () -> + ManifestSidecar.read( + io, path, meta, cancelled, null, null, null, settings, + cache)) + .isInstanceOf(CancellationException.class); + assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + verify(io, times(2)).newInputStream(sidecar); + } + + private ManifestSidecar.Selection readCached( + FileIO io, + Path path, + ManifestFileMeta meta, + ManifestSidecar.Settings settings, + SegmentsCache cache) { + return ManifestSidecar.read( + io, + path, + meta, + RowRangeIndex.create(Collections.singletonList(new Range(20, 20))), + null, + null, + null, + settings, + cache); + } + + @Test + void corruptMissingIncompleteAndMismatchedIndexesFallback() throws Exception { + Path manifest = new Path(temp.toString(), "manifest-golden"); + java.nio.file.Path index = temp.resolve("manifest-golden" + ManifestSidecar.SUFFIX); + ManifestFileMeta meta = goldenMeta(); + RowRangeIndex query = RowRangeIndex.create(Collections.singletonList(new Range(11, 11))); + + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query, settings)) + .isNull(); + byte[] good = golden(); + for (int position : new int[] {0, 9, 11, 15, 16, 55, 63, 67, 75, good.length - 1}) { + byte[] bad = good.clone(); + bad[position] ^= 2; + Files.write(index, bad); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query, settings)) + .isNull(); + } + // A valid checksum cannot make an unsupported container version readable. + for (int version : new int[] {0, 2, 99}) { + byte[] bad = good.clone(); + ByteBuffer.wrap(bad).putInt(8, version); + byte[] hash = + MessageDigest.getInstance("SHA-256") + .digest(Arrays.copyOf(bad, bad.length - 32)); + System.arraycopy(hash, 0, bad, bad.length - 32, 32); + assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, query, settings)) + .isInstanceOf(IOException.class); + } + Files.write(index, Arrays.copyOf(good, good.length - 1)); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query, settings)) + .isNull(); + Files.write(index, good); + assertThat( + ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query, settings) + .blocks()) + .isEmpty(); + assertThatThrownBy( + () -> + ManifestSidecar.select( + good, meta("other", meta.fileSize(), 7), query, settings)) + .isInstanceOf(IOException.class); + } + + @Test + void ioFailuresFallBackWithoutInspectingNestedExceptions() throws Exception { + Path path = new Path(temp.toString(), "m"); + IOException suppressed = new IOException("read failed"); + suppressed.addSuppressed(new CancellationException("cancelled during close")); + for (IOException failure : + Arrays.asList( + new IOException("read failed"), + new java.net.SocketTimeoutException("timeout"), + new java.io.InterruptedIOException("no thread interruption flag"), + new IOException("wrapped", new InterruptedException("interrupted")), + suppressed)) { + FileIO fileIO = mock(FileIO.class); + when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); + assertThat(ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null, settings)) + .isNull(); + assertThat(Thread.currentThread().isInterrupted()).isFalse(); + } + } + + @Test + void interruptedThreadDoesNotFallBackOnIoFailure() throws Exception { + Path path = new Path(temp.toString(), "m"); + IOException failure = new IOException("read failed"); + FileIO fileIO = mock(FileIO.class); + when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); + try { + Thread.currentThread().interrupt(); + assertThatThrownBy( + () -> + ManifestSidecar.read( + fileIO, path, meta("m", 1, 1), null, settings)) + .isInstanceOf(java.io.UncheckedIOException.class) + .hasCauseReference(failure); + assertThat(Thread.currentThread().isInterrupted()).isTrue(); + } finally { + Thread.interrupted(); + } + } + + @Test + void uncheckedFailuresPropagateUnchanged() throws Exception { + Path path = new Path(temp.toString(), "m"); + for (Throwable failure : + Arrays.asList( + new IllegalStateException("unexpected failure"), + new java.io.UncheckedIOException(new IOException("wrapped I/O")), + new CancellationException("cancelled"), + new AssertionError("error"))) { + FileIO fileIO = mock(FileIO.class); + when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); + assertThatThrownBy( + () -> + ManifestSidecar.read( + fileIO, path, meta("m", 1, 1), null, settings)) + .isSameAs(failure); + } + } + + @Test + void indexReadsUseBoundedBulkRequests() throws Exception { + byte[] header = header(); + for (int blockCount : new int[] {5000, 25000, 131073}) { + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + for (int blockNumber = 0; blockNumber < blockCount; blockNumber++) { + builder.beginBlock(header.length + blockNumber * 100L, 100, 1); + builder.add((long) blockNumber, 1); + builder.endBlock(); + } + long size = header.length + blockCount * 100L; + byte[] data = builder.serialize("manifest-large", size, blockCount); + ManifestFileMeta meta = meta("manifest-large", size, blockCount); + CountingInput stream = new CountingInput(data, Integer.MAX_VALUE); + Path path = new Path(temp.toString(), meta.fileName()); + FileIO io = mock(FileIO.class); + when(io.newInputStream(ManifestSidecar.path(path))).thenReturn(stream); + ManifestSidecar.Selection actual = + ManifestSidecar.read( + io, + path, + meta, + RowRangeIndex.create(Collections.singletonList(new Range(0, 0))), + settings); + assertThat(actual.blocks()).hasSize(1); + assertThat(actual.blocks().get(0).offset).isEqualTo(header.length); + assertThat(stream.readLengths).hasSize((data.length + (1 << 20) - 1) / (1 << 20)); + assertThat(stream.requests).allMatch(request -> request <= 1 << 20); + assertThat(stream.closed).isTrue(); + } + } + + @Test + void indexShortReadsAndExactBudget() throws Exception { + byte[] data = golden(); + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(data.length)); + Path path = new Path(temp.toString(), "manifest-golden"); + for (int maxRead : new int[] {Integer.MAX_VALUE, 7}) { + CountingInput stream = new CountingInput(data, maxRead); + FileIO io = mock(FileIO.class); + when(io.newInputStream(ManifestSidecar.path(path))).thenReturn(stream); + ManifestSidecar.Selection actual = + ManifestSidecar.read( + io, + path, + goldenMeta(), + RowRangeIndex.create(Collections.singletonList(new Range(20, 20))), + settings(options, 2)); + assertThat(actual.blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L, 5L); + assertThat(stream.closed).isTrue(); + } + } + + @Test + void indexOverBudgetStopsAfterOneExtraByte() throws Exception { + Options options = sidecarOptions(); + options.set(MAX_BYTES, new MemorySize(128)); + Path path = new Path(temp.toString(), "manifest-golden"); + CountingInput stream = new CountingInput(golden(), Integer.MAX_VALUE); + FileIO io = mock(FileIO.class); + when(io.newInputStream(ManifestSidecar.path(path))).thenReturn(stream); + assertThat( + ManifestSidecar.read( + io, + path, + goldenMeta(), + RowRangeIndex.create(Collections.singletonList(new Range(20, 20))), + settings(options, 2))) + .isNull(); + assertThat(stream.readLengths).containsExactly(129); + assertThat(stream.closed).isTrue(); + } + + @Test + void adjacentBlocksShareReadsForSingleByteConsumers() throws Exception { + byte[] header = header(); + byte[] body = new byte[400]; + for (int position = 0; position < body.length; position++) { + body[position] = (byte) position; + } + ManifestSidecar.Selection selected = + ManifestSidecar.select( + golden(), + goldenMeta(), + RowRangeIndex.create( + Arrays.asList( + new Range(0, 0), + new Range(8254058425445L, 8254058425445L))), + settings); + byte[] manifest = Arrays.copyOf(header, header.length + body.length); + System.arraycopy(body, 0, manifest, header.length, body.length); + CountingInput stream = new CountingInput(manifest, Integer.MAX_VALUE); + Path path = new Path(temp.toString(), "manifest-golden"); + FileIO io = mock(FileIO.class); + when(io.newInputStream(path)).thenReturn(stream); + ByteArrayOutputStream actual = new ByteArrayOutputStream(); + try (InputStream input = ManifestSidecar.openManifest(io, path, selected)) { + int value; + while ((value = input.read()) != -1) { + actual.write(value); + } + assertThat(input.read(new byte[1], 0, 0)).isZero(); + } + assertThat(actual.toByteArray()).isEqualTo(Arrays.copyOf(manifest, header.length + 300)); + assertThat(stream.readLengths).containsExactly(300); + assertThat(stream.seeks).containsExactly((long) header.length); + assertThat(stream.closed).isTrue(); + } + + @Test + void blockReadsSkipGapsAndEmptySelections() throws Exception { + byte[] header = header(); + byte[] manifest = Arrays.copyOf(header, header.length + 400); + Arrays.fill(manifest, header.length + 100, header.length + 300, (byte) 7); + Path path = new Path(temp.toString(), "manifest-golden"); + for (long point : new long[] {20, 16}) { + CountingInput stream = new CountingInput(manifest, Integer.MAX_VALUE); + FileIO io = mock(FileIO.class); + when(io.newInputStream(path)).thenReturn(stream); + byte[] actual; + try (InputStream input = + ManifestSidecar.openManifest(io, path, select(golden(), goldenMeta(), point))) { + actual = IOUtils.readFully(input, false); + } + if (point == 20) { + assertThat(actual).isEqualTo(Arrays.copyOf(header, header.length + 200)); + assertThat(stream.readLengths).containsExactly(100, 100); + assertThat(stream.seeks) + .containsExactly((long) header.length, header.length + 300L); + } else { + assertThat(actual).isEqualTo(header); + assertThat(stream.readLengths).isEmpty(); + assertThat(stream.seeks).isEmpty(); + verifyNoInteractions(io); + } + assertThat(stream.closed).isEqualTo(point == 20); + } + } + + @Test + void cachedBlocksAreSharedByDifferentSelectionsWithoutOpeningTheManifest() throws Exception { + byte[] header = header(); + byte[] manifest = Arrays.copyOf(header, header.length + 400); + for (int i = header.length; i < manifest.length; i++) { + manifest[i] = (byte) i; + } + Path path = new Path(temp.toString(), "manifest-golden"); + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), 400, null, false); + cache.put( + new ManifestSidecar.BlockCacheKey(path, header.length, 100), + new SingleSegments(MemorySegment.wrap(new byte[100]), 100)); + FileIO io = mock(FileIO.class); + CountingInput stream = new CountingInput(manifest, Integer.MAX_VALUE); + when(io.newInputStream(path)).thenReturn(stream); + ManifestSidecar.Selection all = + ManifestSidecar.select( + golden(), + goldenMeta(), + RowRangeIndex.create( + Collections.singletonList(new Range(0, Long.MAX_VALUE))), + settings); + try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); + } + assertThat(stream.readLengths).containsExactly(400); + assertThat(stream.seeks).containsExactly((long) header.length); + assertThat(cache.estimatedSize()).isEqualTo(3); + assertThat(cache.getIfPresents(path)).isNull(); + when(io.newInputStream(path)).thenThrow(new IOException("Must use cached blocks")); + for (long point : new long[] {20, 8254058425445L, 0}) { + ManifestSidecar.Selection selected = select(golden(), goldenMeta(), point); + ByteArrayOutputStream expected = new ByteArrayOutputStream(); + expected.write(header); + for (ManifestSidecar.Block block : selected.blocks()) { + expected.write(manifest, (int) block.offset, (int) block.length); + } + try (InputStream in = ManifestSidecar.openManifest(io, path, selected, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(expected.toByteArray()); + } + } + verify(io, times(1)).newInputStream(path); + + Path other = new Path(temp.toString(), "other/manifest-golden"); + byte[] otherBytes = manifest.clone(); + otherBytes[header.length] ^= 1; + when(io.newInputStream(other)).thenReturn(new CountingInput(otherBytes, Integer.MAX_VALUE)); + try (InputStream in = + ManifestSidecar.openManifest(io, other, select(golden(), goldenMeta(), 0), cache)) { + assertThat(IOUtils.readFully(in, false)) + .isEqualTo(Arrays.copyOf(otherBytes, header.length + 100)); + } + verify(io).newInputStream(other); + } + + @Test + void mixedHitsAndMissesReadOnlyUncachedBlocks() throws Exception { + byte[] header = header(); + byte[] manifest = Arrays.copyOf(header, header.length + 400); + FileIO io = mock(FileIO.class); + Path path = new Path(temp.toString(), "manifest-golden"); + CountingInput cold = new CountingInput(manifest, Integer.MAX_VALUE); + CountingInput mixed = new CountingInput(manifest, Integer.MAX_VALUE); + when(io.newInputStream(path)).thenReturn(cold, mixed); + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), 400, null, false); + try (InputStream in = + ManifestSidecar.openManifest( + io, path, select(golden(), goldenMeta(), 8254058425445L), cache)) { + IOUtils.readFully(in, false); + } + ManifestSidecar.Selection all = + ManifestSidecar.select( + golden(), + goldenMeta(), + RowRangeIndex.create( + Collections.singletonList(new Range(0, Long.MAX_VALUE))), + settings); + try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); + } + assertThat(mixed.readLengths).containsExactly(100, 100); + assertThat(mixed.seeks).containsExactly((long) header.length, header.length + 300L); + try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); + } + verify(io, times(2)).newInputStream(path); + } + + @Test + void truncatedCoalescedReadsDoNotPopulateTheBlockCache() throws Exception { + byte[] header = header(); + byte[] manifest = Arrays.copyOf(header, header.length + 400); + FileIO io = mock(FileIO.class); + Path path = new Path(temp.toString(), "manifest-golden"); + CountingInput truncated = + new CountingInput(Arrays.copyOf(manifest, manifest.length - 1), 7); + CountingInput complete = new CountingInput(manifest, 7); + when(io.newInputStream(path)).thenReturn(truncated, complete); + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), 400, null, false); + ManifestSidecar.Selection all = + ManifestSidecar.select( + golden(), + goldenMeta(), + RowRangeIndex.create( + Collections.singletonList(new Range(0, Long.MAX_VALUE))), + settings); + try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { + assertThatThrownBy(() -> IOUtils.readFully(in, false)).isInstanceOf(EOFException.class); + } + assertThat(truncated.closed).isTrue(); + assertThat(cache.estimatedSize()).isZero(); + try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); + } + assertThat(complete.closed).isTrue(); + assertThat(cache.estimatedSize()).isEqualTo(3); + } + + @Test + void evictedBlocksAreReadAgainWithinTheSharedBudget() throws Exception { + byte[] header = header(); + byte[] manifest = Arrays.copyOf(header, header.length + 400); + Path path = new Path(temp.toString(), "manifest-golden"); + FileIO io = mock(FileIO.class); + when(io.newInputStream(path)) + .thenAnswer(ignored -> new CountingInput(manifest, Integer.MAX_VALUE)); + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofBytes(1300), 400, null, false); + ManifestSidecar.Selection all = + ManifestSidecar.select( + golden(), + goldenMeta(), + RowRangeIndex.create( + Collections.singletonList(new Range(0, Long.MAX_VALUE))), + settings); + for (int round = 0; round < 2; round++) { + try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); + } + assertThat(cache.totalCacheBytes()).isLessThanOrEqualTo(1300); + assertThat(cache.estimatedSize()).isEqualTo(1); + } + verify(io, times(2)).newInputStream(path); + } + + @Test + void oversizedBlocksUseBoundedReadsWithoutModifyingPreviouslyCachedBytes() throws Exception { + byte[] header = header(); + int cachedLength = (1 << 20) + 17; + int uncachedLength = 2 * (1 << 20) + 31; + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, cachedLength, 1); + builder.add(0L, 1); + builder.endBlock(); + builder.beginBlock(header.length + cachedLength, uncachedLength, 1); + builder.add(100L, 1); + builder.endBlock(); + byte[] manifest = Arrays.copyOf(header, header.length + cachedLength + uncachedLength); + Arrays.fill(manifest, header.length, header.length + cachedLength, (byte) 7); + Arrays.fill(manifest, header.length + cachedLength, manifest.length, (byte) 9); + byte[] data = builder.serialize("large", manifest.length, 2); + ManifestFileMeta meta = meta("large", manifest.length, 2); + Path path = new Path(temp.toString(), "large"); + FileIO io = mock(FileIO.class); + CountingInput cold = new CountingInput(manifest, Integer.MAX_VALUE); + CountingInput mixed = new CountingInput(manifest, Integer.MAX_VALUE); + when(io.newInputStream(path)).thenReturn(cold, mixed); + SegmentsCache cache = + new SegmentsCache<>(1024, MemorySize.ofMebiBytes(8), cachedLength, null, false); + ManifestSidecar.Selection first = select(data, meta, 0); + byte[] expected = Arrays.copyOf(manifest, header.length + cachedLength); + try (InputStream in = ManifestSidecar.openManifest(io, path, first, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(expected); + } + assertThat(cold.readLengths).containsExactly(1 << 20, 17); + ManifestSidecar.Selection all = + ManifestSidecar.select( + data, + meta, + RowRangeIndex.create( + Collections.singletonList(new Range(0, Long.MAX_VALUE))), + settings); + try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); + } + assertThat(mixed.readLengths).containsExactly(1 << 20, 1 << 20, 31); + assertThat(cache.estimatedSize()).isEqualTo(1); + assertThat( + cache.getIfPresents( + new ManifestSidecar.BlockCacheKey( + path, header.length + cachedLength, uncachedLength))) + .isNull(); + try (InputStream in = ManifestSidecar.openManifest(io, path, first, cache)) { + assertThat(IOUtils.readFully(in, false)).isEqualTo(expected); + } + verify(io, times(2)).newInputStream(path); + } + + @Test + void largeBlockSpansUseBoundedReads() throws Exception { + byte[] header = header(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + long offset = header.length; + for (int length : new int[] {512 * 1024, 512 * 1024, 257}) { + builder.beginBlock(offset, length, 1); + builder.add(20L, 1); + builder.endBlock(); + offset += length; + } + byte[] data = builder.serialize("manifest-large", offset, 3); + byte[] manifest = Arrays.copyOf(header, (int) offset); + CountingInput stream = new CountingInput(manifest, Integer.MAX_VALUE); + FileIO io = mock(FileIO.class); + Path path = new Path(temp.toString(), "manifest-large"); + when(io.newInputStream(path)).thenReturn(stream); + try (InputStream input = + ManifestSidecar.openManifest( + io, path, select(data, meta("manifest-large", offset, 3), 20))) { + assertThat(IOUtils.readFully(input, false)).isEqualTo(manifest); + } + assertThat(stream.readLengths).containsExactly(1 << 20, 257); + assertThat(stream.seeks).containsExactly((long) header.length); + assertThat(stream.closed).isTrue(); + } + + @Test + void blockShortReadsAndTruncation() throws Exception { + byte[] header = header(); + Path path = new Path(temp.toString(), "manifest-golden"); + ManifestSidecar.Selection selected = + ManifestSidecar.select( + golden(), + goldenMeta(), + RowRangeIndex.create( + Collections.singletonList(new Range(0, Long.MAX_VALUE))), + settings); + for (int bodyLength : new int[] {400, 399}) { + byte[] manifest = Arrays.copyOf(header, header.length + bodyLength); + CountingInput stream = new CountingInput(manifest, 7); + FileIO io = mock(FileIO.class); + when(io.newInputStream(path)).thenReturn(stream); + try (InputStream input = ManifestSidecar.openManifest(io, path, selected)) { + if (bodyLength == 400) { + assertThat(IOUtils.readFully(input, false)).isEqualTo(manifest); + } else { + assertThatThrownBy(() -> IOUtils.readFully(input, false)) + .isInstanceOf(EOFException.class); + } + } + assertThat(stream.closed).isTrue(); + } + } + + private static class CountingInput extends ByteArraySeekableStream { + private final int maxRead; + private final List requests = new ArrayList<>(); + private final List readLengths = new ArrayList<>(); + private final List seeks = new ArrayList<>(); + private boolean closed; + + private CountingInput(byte[] data, int maxRead) { + super(data); + this.maxRead = maxRead; + } + + @Override + public int read(byte[] bytes, int offset, int length) throws IOException { + requests.add(length); + int count = super.read(bytes, offset, Math.min(length, maxRead)); + if (count > 0) { + readLengths.add(count); + } + return count; + } + + @Override + public void seek(long position) throws IOException { + seeks.add(position); + super.seek(position); + } + + @Override + public void close() throws IOException { + closed = true; + super.close(); + } + } + + private ManifestSidecar.Selection select(byte[] data, ManifestFileMeta meta, long point) + throws IOException { + return ManifestSidecar.select( + data, + meta, + RowRangeIndex.create(Collections.singletonList(new Range(point, point))), + settings); + } +} diff --git a/paimon-core/src/test/resources/manifest-sidecar.txt b/paimon-core/src/test/resources/manifest-sidecar.txt new file mode 100644 index 000000000000..f2199ce2e4f0 --- /dev/null +++ b/paimon-core/src/test/resources/manifest-sidecar.txt @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +avroHeader=T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAA +partitionA=AAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhA== +partitionB=AAAAAgACAAAAAAAACQAAAAAAAAAAAAAAAAAAAA== +index=UEFJTVNDQVIAAAABL0eWubkHdL6ioNdIPj3jGtp9TAk83wbtAsmc+xJVwM8AAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAADkAAAAAAAAAZAAAAAAAAAADAAEAAAAkAAAAAgAAAAAAAAAAAAAAAAAAAAkAAAAAAAAAFAAAAAAAAAAYAAAAAAAAAACdAAAAAAAAAMgAAAAAAAAAAgABAAAAJAAAAAIAAAAA/////gAAAAEAAAACAAAHgcw4bGUAAAeBzDhsZQAAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIAAQAAACQAAAACAAAAAAAAABQAAAAAAAAAGH//////////f/////////8AlOHbHSWYVK7dgM+l6yo0+LjOLGYNf4lnkt2OtValVYA= +indexWithPartitions=UEFJTVNDQVIAAAABL0eWubkHdL6ioNdIPj3jGtp9TAk83wbtAsmc+xJVwM8AAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAMAAAAAgAAAAAAAAABAQAAACQAAAACAAAAAAAAAAAAAAAAAAAACQAAAAAAAAAUAAAAAAAAABgAAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAwAAAACAAAAAAAAAAEBAAAAJAAAAAIAAAAA/////gAAAAEAAAACAAAHgcw4bGUAAAeBzDhsZQAAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAADAAAAAIAAAAAAAAAAQEAAAAkAAAAAgAAAAAAAAAUAAAAAAAAABh//////////3//////////AInpuqbYtRIEgHOW0YNU5V+bvWbXg6ARpSP/hXdwewB8 +indexWithBuckets=UEFJTVNDQVIAAAABL0eWubkHdL6ioNdIPj3jGtp9TAk83wbtAsmc+xJVwM8AAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAMAAAAAgAAAAAAAAABAQAAACQAAAACAAAAAAAAAAAAAAAAAAAACQAAAAAAAAAUAAAAAAAAABgBAAAAFAAAAAIAAAABAAAABAAAAAEAAAAIAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAwAAAACAAAAAAAAAAEBAAAAJAAAAAIAAAAA/////gAAAAEAAAACAAAHgcw4bGUAAAeBzDhsZQEAAAAUAAAAAgAAAAIAAAAEAAAAAgAAAAgAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAADAAAAAIAAAAAAAAAAQEAAAAkAAAAAgAAAAAAAAAUAAAAAAAAABh//////////3//////////AQAAABQAAAACAAAAAAAAAAEAAAADAAAABGCAy0ixcqKywlCau1+E/XLEorySmCmL1zl5q9/dwdBu From eecc11e1aeb1e6d0a26b250641269c04c6882363 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 20:32:01 +0800 Subject: [PATCH 02/14] [core] Compress sidecar payloads and simplify the format --- docs/docs/concepts/spec/manifest.md | 273 +++---- .../paimon/manifest/ManifestSidecar.java | 496 ++++++------ .../manifest/ManifestBlockIndexTest.java | 754 ++++++++---------- .../paimon/manifest/ManifestSidecarTest.java | 340 +++----- .../src/test/resources/manifest-sidecar.txt | 6 +- 5 files changed, 809 insertions(+), 1060 deletions(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 97895145462b..835bcafffe3f 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -77,23 +77,23 @@ sidecar references, managing file ownership, applying entry filters and reconcil entries after block selection. `build` reads the completed physical manifest and returns sidecar bytes; it does not write or publish another file. -`Settings` takes a byte budget and separate booleans for partition, row-ID and bucket payload -generation. A disabled dimension uses encoding 0 and has no length or payload bytes. The -partition dictionary is empty when partition generation is disabled. These generation settings -do not prevent readers from using payloads already present in a sidecar. +`Settings` enables row-ID and bucket payload generation independently. Partition generation is +always enabled, including the empty partition tuple for unpartitioned tables. Missing or invalid +metadata makes only the affected block's dimension unavailable. There is no sidecar byte budget: +construction keeps complete coverage and `read` consumes the entire file once it is opened. `read` returns null for an absent sidecar reference or an `IOException`, allowing the caller to fall back to the manifest. If the thread is interrupted, the I/O failure is propagated as `UncheckedIOException`. Other exceptions and errors propagate unchanged. `select` validates supplied bytes directly and reports invalid containers with `IOException`. -Version 1 uses the following layout. Container integers and payload integers -are fixed-width big endian. Encoding IDs are unsigned bytes with separate namespaces. +Version 1 uses the following layout. Container `int` and `long` fields are signed, fixed-width +4-byte and 8-byte big-endian integers. Encoding IDs are unsigned bytes with separate namespaces. +Payload integers use the variable-length encoding described below. ```text -magic : 8 bytes // ASCII PAIMSCAR +magic : 4 bytes // ASCII PMSC formatVersion : int // 1 -manifestNameHash : 32 bytes // SHA-256 of the UTF-8 basename manifestLength : long manifestEntryCount : long // ADD + DELETE avroHeaderLength : int @@ -122,84 +122,100 @@ blocks[] // original physical order checksum : 32 bytes // SHA-256 of all preceding bytes ``` -The block ID is its position. Its first entry ordinal is the sum of preceding record -counts and is not stored. Each complete partition tuple appears once in the dictionary, -including all its fields and nulls. The scan's partition type interprets the existing -serialized tuple. Partition predicates are evaluated once per dictionary entry. +The sidecar contains no manifest-name hash. Renaming the manifest does not change sidecar +bytes. Its stored length and entry count must match the supplied manifest metadata; the +caller must associate the sidecar with the correct immutable manifest through its reference. + +The block ID is its position. Its first entry ordinal is the sum of preceding record counts +and is not stored. Each complete partition tuple appears once in the dictionary, including +all its fields and nulls. The scan's partition type interprets the existing serialized tuple. +Partition predicates are evaluated once per dictionary entry. | Dimension | Encoding | Payload | | --- | --- | --- | | Any | `0` | Unavailable; only the encoding byte is present. | -| Partition | `1` | Positive `partitionIdCount: int` followed by sorted unique dictionary IDs (`int`). | -| Row ID | `1` | Positive `rangeCount: int` followed by sorted disjoint inclusive `(start: long, end: long)` pairs. Coverage may conservatively include gaps. | -| Bucket | `1` | Positive `pairCount: int` followed by sorted unique `(bucket: int, totalBuckets: int)` pairs. | -| Any | Other nonzero ID | Skip exactly the bounded payload length; treat only this dimension as unavailable. | - -Only nonzero encodings are followed by a length and payload. Payload lengths exclude -the encoding and length fields, but include the count at the start of the payload. -For all three encoding-1 payloads below, `int` is a signed 4-byte integer and `long` is -a signed 8-byte integer, both big endian. Elements have no padding, per-element length -prefixes, or Avro variable-length integer encoding. Counts must be positive; encoding 0 -represents unavailable coverage, rather than encoding 1 with a zero count. +| Partition | `1` | Count and delta/RLE-compressed sorted unique dictionary IDs. | +| Row ID | `1` | Interval count, minimum, span, and delta/RLE-compressed interior endpoints. | +| Bucket | `1` | Count and delta/RLE-compressed sorted unique packed bucket/count pairs. | +| Any | Other nonzero ID | Skip the declared payload length; treat only this dimension as unavailable. | + +Only nonzero encodings are followed by a length and payload. Payload lengths exclude the +encoding and length fields, but include the count and other fields within the payload. +All three encoding-1 payloads have positive counts no greater than the block's record count. +Encoding 0 represents unavailable coverage, rather than encoding 1 with a zero count. + +#### Delta and RLE Encoding + +Every integer inside an encoding-1 payload is a nonnegative unsigned LEB128 varint, using +one to nine bytes for values from 0 through `Long.MAX_VALUE`. Seven value bits are stored +per byte, least significant group first; the high bit indicates another byte follows. +Encodings use the shortest representation. There is no ZigZag transformation or padding. + +A sorted sequence is delta-encoded from a specified base. Consecutive equal deltas are +stored as runs: + +```text +runs[] + repeatCount : varint // positive number of values produced + delta : varint // add delta for each value in the run +``` + +Starting with `previous = base`, a run produces `repeatCount` successive values by adding +`delta` each time. Run counts must sum to the dimension's expected value count. Decoders +consume values lazily, check overflow and the applicable value bounds, and require the +payload to end when all expected values have been consumed. They do not allocate expanded +arrays for runs. #### Partition Payload -When `partitionEncoding == 1`, the block stores the IDs of all distinct partition tuples +When `partitionEncoding == 1`, the block stores IDs of all distinct partition tuples represented by its entries: ```text partitionPayload - partitionIdCount : int // N > 0 - partitionIds[N] : int // N consecutive 4-byte dictionary IDs - -partitionPayloadLength = 4 + 4 * N + partitionIdCount : varint // N > 0 + runs[] // N dictionary IDs, base = 0 ``` -An ID is the zero-based position of a complete tuple in the sidecar's shared -`partitionDictionary`, not an individual partition field or an entry ordinal. Valid IDs -satisfy `0 <= id < partitionCount` and are strictly increasing, with no duplicates. -The tuple bytes appear only in the dictionary; they are not repeated in each block's payload. -For example, IDs `[0, 3]` are stored as the three integers `[2, 0, 3]`, occupying 12 payload -bytes, or 17 bytes including `partitionEncoding` and `partitionPayloadLength`. +An ID is the zero-based position of a complete tuple in the sidecar's shared dictionary. +IDs satisfy `0 <= id < partitionCount` and are strictly increasing. Tuple bytes appear only +in the dictionary and are not repeated in each block. For IDs `[0, 1, 2, 3, 4]`, the deltas +are `[0, 1, 1, 1, 1]` and the runs are `(1, 0), (4, 1)`. The complete payload bytes are +`[5, 1, 0, 4, 1]`: 5 bytes, or 10 bytes including the encoding and length fields. -With a partition filter, the block matches if any referenced dictionary tuple matches. -A tuple containing a null partition value can still have a valid dictionary ID. If any -entry's partition tuple is unavailable, or partition coverage cannot fit its budget, the -block uses encoding 0 so that missing dictionary coverage cannot exclude it. +With a partition filter, a block matches if any referenced tuple matches. A tuple containing +a null partition value still has a dictionary ID. Unpartitioned tables record the empty +tuple. If any entry's entire partition tuple is unavailable, the block uses encoding 0, +so a dictionary miss cannot exclude that block. Later blocks can still use existing IDs. #### Row-ID Payload -When `rowIdEncoding == 1`, the block stores inclusive row-ID intervals: +The writer merges overlapping and adjacent inclusive intervals contributed by entries. +An entry contributes `[firstRowId, firstRowId + rowCount - 1]`. The resulting intervals are +sorted and disjoint; they are never expanded into individual row IDs or coarsened to include gaps. ```text rowIdPayload - rangeCount : int // N > 0 - ranges[N] - start : long // inclusive first row ID - end : long // inclusive last row ID - -rowIdPayloadLength = 4 + 16 * N + rangeCount : varint // N > 0 + min : varint // first interval's start + span : varint // last interval's end minus min + runs[] // 2 * (N - 1) interior endpoints, base = min ``` -Each pair satisfies `0 <= start <= end <= Long.MAX_VALUE`. Pairs are ordered by `start` -and do not overlap: each `start` is greater than the preceding `end`. The writer merges -overlapping and adjacent intervals contributed by the entries. An entry contributes -`[firstRowId, firstRowId + rowCount - 1]`; these are table row IDs, not manifest entry -ordinals. `rangeCount` counts intervals, not entries or individual row IDs. - -There are no separate block min/max fields in this payload. The reader obtains the block -minimum from the first pair's `start` and the maximum from the last pair's `end`. It tests -this envelope first, then checks individual intervals if necessary. For example, -`[(10, 19), (30, 39)]` is stored as `rangeCount = 2` followed by four longs. Its payload -length is 36 bytes, or 41 bytes including the encoding and length fields. Its envelope -is `[10, 39]`, but a query for row ID 25 does not match either interval. - -If the exact union exceeds its available budget, the writer can store one conservative -`[min,max]` pair using the same encoding. That payload has `rangeCount = 1` and length -20 bytes; it can include gaps. There is no separate flag distinguishing a coarsened pair -from an exact interval, so entry filtering remains necessary. Unknown or invalid row-ID -metadata makes coverage unavailable for the block; further byte-budget degradation can -also drop the payload entirely. +The maximum is `min + span`, which must not exceed `Long.MAX_VALUE`. Flatten the intervals +as `[start0, end0, start1, end1, ...]`. The first start is supplied by `min`, and the last +end by `min + span`; only the remaining `2 * (N - 1)` interior endpoints are delta/RLE encoded. +Pairing the reconstructed endpoints recovers the intervals. Each pair satisfies +`0 <= start <= end <= Long.MAX_VALUE`; each following start must exceed the preceding end. + +For `[(10, 19), (30, 39)]`, the count is 2, minimum is 10, and span is 29. The interior +endpoints `[19, 30]` have deltas `[9, 11]` from base 10, encoded as `(1, 9), (1, 11)`. +The complete payload bytes are `[2, 10, 29, 1, 9, 1, 11]`: 7 bytes, or 12 bytes with framing. +For a single interval, the envelope completely defines the interval and no runs follow. + +The reader first tests the envelope without expanding any runs. A query for row ID 25 +passes the example's envelope check but matches neither interval. Unknown or invalid row-ID +metadata makes that block's row-ID payload unavailable; partition and bucket coverage remain usable. #### Bucket Payload @@ -207,91 +223,62 @@ When `bucketEncoding == 1`, the block stores distinct bucket/count pairs: ```text bucketPayload - pairCount : int // N > 0 - pairs[N] - bucket : int // entry's bucket number - totalBuckets : int // entry's recorded total bucket count + pairCount : varint // N > 0 + runs[] // N packed pairs, base = 0 -bucketPayloadLength = 4 + 8 * N +packedPair = ((long) bucket << 32) | totalBuckets ``` -Each pair satisfies `0 <= bucket < totalBuckets`. Pairs are sorted by `bucket`, then -`totalBuckets`, and deduplicated. `totalBuckets` comes from the entry's `_TOTAL_BUCKETS`; -it is not the number of buckets represented by this block or the table's current bucket -setting. The same bucket number can therefore occur with different totals after rescaling. -For example, `[(1, 4), (1, 8), (3, 4)]` is stored as the seven integers -`[3, 1, 4, 1, 8, 3, 4]`, occupying 28 payload bytes, or 33 bytes including the encoding -and length fields. These pairs have no partition IDs or separate bucket min/max fields. - -Missing, invalid, negative/synthetic or over-budget bucket metadata makes that block's -bucket coverage unavailable (encoding 0, no length or payload). Partition and row-ID -coverage remain independently usable; no mutual-exclusion restriction is imposed. - -A caller can supply a predicate on `(bucket, totalBuckets)` to test this payload. The -predicate must conservatively retain every potentially matching pair. Filters requiring an -entry's partition belong at the entry-filtering stage; omit the bucket predicate when no -safe partition-independent check is available. An unavailable bucket payload cannot exclude -a block. Malformed payload lengths or pair counts invalidate the container. Invalid ordering -or values encountered while matching also invalidate it; elements after the first match are skipped. - -#### Validation and Coverage - -Invalid lengths, known-payload framing, checksum mismatches or inconsistent physical -coverage invalidate the container. Invalid dictionary references or interval ordering -encountered in decoded payload contents also invalidate it. Byte spans must cover the -entire original manifest after its header; record counts must sum to the manifest entry -count. Readers validate the checksum, payload framing (including known count/length -consistency), and the complete block directory even when a block is rejected. Block -payload contents are decoded and validated only for dimensions still needed by the filters, -and only until that dimension matches. A row-ID min/max rejection skips individual -intervals; a match skips the remaining elements of that payload. Skipped payload contents -are not individually validated. - -All entries contribute, including ADD, DELETE and every file format/column group. -Row-ID ranges are never expanded into individual values. If an exact union exceeds its -available byte budget, it becomes the inclusive `[min,max]` envelope with encoding 1. Processing -continues through the end of the block to extend those bounds and detect unknown row IDs. -An unknown or invalid row-ID range makes only that block's row-ID payload unavailable. -Partition budget exhaustion independently makes that block's partition payload unavailable. -The dictionary can consequently be incomplete for the manifest: a dictionary miss never -excludes a block with unavailable partition coverage. Later blocks can still use existing IDs. - -`Settings.maxBytes` bounds the whole serialized container, including the partition -dictionary and all three payload types. The caller supplies the byte budget. It is capped -at 2147483646 bytes to fit the in-memory byte-array representation. A budget too small for a -complete sidecar causes `build` or `read` to return null; callers can keep using the manifest. -The Avro header and block directory share this byte budget without separate size or count limits. -Writers discard optional row-ID payloads, bucket payloads, then partition payloads/dictionary if necessary, -to fit the complete directory. If the directory itself cannot fit, no sidecar is published. -No emitted sidecar omits block descriptors. These are encoded-size bounds; Avro header parsing -and sidecar construction also incur object/buffer overhead. Query concurrency multiplies per-reader costs. - -For conjunctive filters a block is retained only if each dimension is either unavailable -or matches. Within each block, matching tests row ID, partition, then bucket coverage. -It skips absent filters and short-circuits after a dimension rejects a block, skipping -the contents of later payloads. Within each payload, matching stops at the first hit. Matches in different dimensions can come -from different entries in the block, so entry filtering and deletion merging remain -necessary. Block min/max is derived from the first/last interval before testing the -individual intervals. - -Readers still consume and validate the whole bounded sidecar. A partition-only query -therefore reads row-ID payload bytes too; payload lengths save decoding work for unused -payload contents and unknown encodings, not storage I/O. Selected compressed blocks are read by byte range with adjacent +Each pair satisfies `0 <= bucket < totalBuckets`. Packing places the bucket in the high +32 bits and the recorded total bucket count in the low 32 bits. Packed values are nonnegative, +sorted and unique, equivalent to sorting first by bucket and then by total bucket count. +The decoder recovers `bucket = (int) (packedPair >>> 32)` and `totalBuckets = (int) packedPair`. +The same bucket may occur with different totals after rescaling. + +For `[(1, 4), (1, 8), (3, 4)]`, the packed values are `[4294967300, 4294967304, 12884901892]` +and deltas are `[4294967300, 4, 8589934588]`. The payload contains count 3 and three runs +of length 1, occupying 15 bytes, or 20 bytes with framing. Repeated bucket strides with the +same total bucket count form a single run. + +Missing, invalid or negative/synthetic bucket metadata makes the block's bucket coverage +unavailable. A caller can supply a predicate on `(bucket, totalBuckets)` which conservatively +retains every potentially matching pair. Filters requiring an entry's partition belong at +the entry-filtering stage; omit the bucket predicate if no safe check is available. + +#### Validation and Reading + +Readers validate the checksum, fixed container fields, payload lengths and count headers, +and the complete physical block directory regardless of the query. Byte spans must cover +the whole original manifest after its header; record counts must sum to the manifest entry +count. Unknown nonzero encodings skip their declared bytes without interpreting a count. + +Compressed contents are decoded only for dimensions needed by the filters and only until +that dimension matches. A row-ID envelope rejection skips all its runs; a matching interval, +partition ID or bucket pair skips remaining values. Invalid varints, run counts, overflows, +out-of-range values or ordering encountered while decoding invalidate the container. Run +contents skipped by short-circuiting are not individually validated. + +For conjunctive filters a block is retained only if every dimension is unavailable or matches. +Matching tests row ID, partition, then bucket coverage. Absent filters are skipped, and a +rejection skips the remaining dimensions. Matches in different dimensions can come from +different entries, so entry filtering and ADD/DELETE reconciliation remain necessary. + +The entire sidecar is read in chunks of at most 1 MiB, including payloads unused by a query. +There is no size-based fallback or payload dropping. Payload lengths save decoding work, +not sidecar storage I/O. Selected compressed Avro blocks are read by byte range with adjacent spans coalesced. Building a sidecar does not modify the original manifest. `read` and `openManifest` accept an optional caller-supplied `SegmentsCache`. Complete -sidecar bytes are keyed by their explicit `Path`, subject to the cache memory budget and single-file threshold. -Only successful reads and selections populate the cache. Each query creates independent -views and reapplies its filters and byte budget; query-specific selections are not cached. - -Selected Avro blocks also share this cache. Each entry contains one complete compressed -block, keyed by the manifest's full path, original offset and encoded length, separately -from whole-manifest and sidecar entries. Different selections can reuse the same blocks. -Only successful complete reads populate the cache; oversized blocks stream through the -bounded read buffer. Adjacent uncached blocks are read together when they fit the read -buffer, then cached individually. Fully cached selections do not open the manifest file. -Block entries follow the existing memory budget, entry-size limit, expiration and eviction -settings. The Avro decoder and entry filters still run on cached bytes. +sidecar bytes are keyed by their explicit `Path`. Only successful reads and selections +populate the cache; query-specific selections are not cached. Cache entry-size limits affect +admission only: larger sidecars are still fully read, validated and used. + +Selected Avro blocks also share this cache. Each entry contains one complete compressed block, +keyed by the manifest's full path, original offset and encoded length, separately from whole-file +keys. Different selections reuse the same blocks. Only complete reads populate the cache; +oversized blocks stream through the read buffer. Adjacent uncached blocks fitting the buffer +are read together and cached individually. Fully cached selections do not open the manifest. +The cache retains its configured memory budget, entry-size limit, expiration and eviction policy. ## Manifest diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index b2807fa0796f..9edea887b156 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -43,7 +43,6 @@ import java.io.InputStream; import java.io.UncheckedIOException; import java.nio.ByteBuffer; -import java.nio.charset.StandardCharsets; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; import java.util.ArrayList; @@ -58,13 +57,15 @@ import java.util.TreeSet; import java.util.function.BiPredicate; +import static org.apache.paimon.utils.VarLengthIntUtils.encodeLong; + /** Independently usable partition, row-id and bucket coverage for each manifest block. */ public final class ManifestSidecar { public static final String SUFFIX = ".avro.sidecar"; private static final Logger LOG = LoggerFactory.getLogger(ManifestSidecar.class); - private static final long MAGIC = 0x5041494d53434152L; + private static final int MAGIC = 0x504d5343; private static final int FORMAT_VERSION = 1; - private static final int HEADER_BYTES = 60; + private static final int HEADER_BYTES = 24; private static final int BLOCK_BYTES = 27; private static final byte[] EMPTY = new byte[0]; private static final int DIGEST_BYTES = 32; @@ -90,25 +91,12 @@ public static String fileName(ManifestFileMeta manifest) { return null; } - /** Construction bounds and independently enabled payloads, supplied by the caller. */ + /** Optional payloads supplied by the caller. Partition coverage is always enabled. */ public static final class Settings { - public final int maxBytes; - public final boolean partitionEnabled; public final boolean rowIdEnabled; public final boolean bucketEnabled; - public Settings( - long maxBytes, - boolean partitionEnabled, - boolean rowIdEnabled, - boolean bucketEnabled) { - if (maxBytes < 0) { - throw new IllegalArgumentException( - "Manifest sidecar byte budget must be nonnegative"); - } - // A sidecar fits in one byte array; reserve one byte for the overflow probe. - this.maxBytes = (int) Math.min(maxBytes, Integer.MAX_VALUE - 1L); - this.partitionEnabled = partitionEnabled; + public Settings(boolean rowIdEnabled, boolean bucketEnabled) { this.rowIdEnabled = rowIdEnabled; this.bucketEnabled = bucketEnabled; } @@ -144,9 +132,7 @@ public List blocks() { } } - /** - * Builds complete block descriptors even when either optional dimension becomes unavailable. - */ + /** Builds a complete block directory with independently available coverage. */ public static final class Builder { private final Settings settings; private final byte[] header; @@ -155,7 +141,6 @@ public static final class Builder { private final TreeSet partitionIds = new TreeSet<>(); private final TreeSet bucketPairs = new TreeSet<>(); private final List blocks = new ArrayList<>(); - private boolean complete; private long nextOffset; private long nextRecord; private Block current; @@ -163,51 +148,20 @@ public static final class Builder { private boolean rowAvailable; private boolean partitionAvailable; private boolean bucketAvailable; - private boolean coarse; - private long min; - private long max; - private int dictionaryBytes; - private int optionalBytes; - public Builder(Settings settings, @Nullable byte[] header) { + public Builder(Settings settings, byte[] header) { this.settings = settings; - this.header = header; - complete = - header != null - && HEADER_BYTES + DIGEST_BYTES + 12L + header.length - <= settings.maxBytes; - nextOffset = header == null ? 0 : header.length; - } - - public boolean complete() { - return complete; + this.header = Objects.requireNonNull(header); + nextOffset = header.length; } public void beginBlock(long offset, long length, long records) throws IOException { - if (!complete) { - return; - } require(current == null && offset == nextOffset && length > 0 && records > 0); - // Optional payloads can be discarded later, but descriptors must never be truncated. - if (HEADER_BYTES - + DIGEST_BYTES - + 12L - + header.length - + (blocks.size() + 1L) * BLOCK_BYTES - > settings.maxBytes) { - complete = false; - blocks.clear(); - dictionary.clear(); - return; - } current = new Block(offset, length, nextRecord, records); entriesInBlock = 0; rowAvailable = settings.rowIdEnabled; - partitionAvailable = settings.partitionEnabled; + partitionAvailable = true; bucketAvailable = settings.bucketEnabled; - coarse = false; - min = Long.MAX_VALUE; - max = -1; ranges.clear(); partitionIds.clear(); bucketPairs.clear(); @@ -229,9 +183,6 @@ public void add( @Nullable byte[] partition, @Nullable Integer bucket, @Nullable Integer totalBuckets) { - if (!complete) { - return; - } if (current == null) { throw new IllegalStateException("No current Avro block"); } @@ -246,14 +197,8 @@ public void add( ranges.clear(); return; } - long end = first + (count - 1); - min = Math.min(min, first); - max = Math.max(max, end); - // Keep checking subsequent entries, including missing row IDs, after coarsening. - if (coarse) { - return; - } long start = first; + long end = first + (count - 1); Map.Entry before = ranges.floorEntry(start); if (before != null && before.getValue() >= start - 1) { start = before.getKey(); @@ -266,35 +211,19 @@ public void add( end = Math.max(end, next.getValue()); ranges.remove(next.getKey()); } - if (8L + 16L * (ranges.size() + 1L) > settings.maxBytes - optionalBytes) { - coarse = true; - ranges.clear(); - } else { - ranges.put(start, end); - } + ranges.put(start, end); } private void addBucket(@Nullable Integer bucket, @Nullable Integer totalBuckets) { if (!bucketAvailable) { return; } - if (bucket == null - || totalBuckets == null - || bucket < 0 - || totalBuckets <= 0 - || bucket >= totalBuckets) { + if (bucket == null || totalBuckets == null || bucket < 0 || totalBuckets <= bucket) { bucketAvailable = false; bucketPairs.clear(); return; } - long pair = ((long) bucket << 32) | totalBuckets; - if (!bucketPairs.contains(pair) - && 8L + 8L * (bucketPairs.size() + 1L) > settings.maxBytes - optionalBytes) { - bucketAvailable = false; - bucketPairs.clear(); - } else { - bucketPairs.add(pair); - } + bucketPairs.add(((long) bucket << 32) | totalBuckets); } private void addPartition(@Nullable byte[] bytes) { @@ -308,62 +237,24 @@ private void addPartition(@Nullable byte[] bytes) { } Integer id = dictionary.get(ByteBuffer.wrap(bytes)); if (id == null) { - if (bytes.length + 4L > settings.maxBytes - dictionaryBytes) { - partitionAvailable = false; - partitionIds.clear(); - return; - } id = dictionary.size(); dictionary.put(ByteBuffer.wrap(bytes.clone()), id); - dictionaryBytes += 4 + bytes.length; } partitionIds.add(id); } public void endBlock() throws IOException { - if (!complete) { - return; - } require(current != null && entriesInBlock == current.recordCount); - byte[] rowPayload = EMPTY; - byte[] partitionPayload = EMPTY; - if (rowAvailable) { - if (coarse || 8L + 16L * ranges.size() > settings.maxBytes - optionalBytes) { - ranges.clear(); - ranges.put(min, max); - } - if (8L + 16L * ranges.size() <= settings.maxBytes - optionalBytes) { - ByteBuffer out = ByteBuffer.allocate(4 + 16 * ranges.size()); - out.putInt(ranges.size()); - for (Map.Entry range : ranges.entrySet()) { - out.putLong(range.getKey()).putLong(range.getValue()); - } - rowPayload = out.array(); - optionalBytes += payloadSize(rowPayload); - } - } - if (partitionAvailable - && 8L + 4L * partitionIds.size() <= settings.maxBytes - optionalBytes) { - ByteBuffer out = ByteBuffer.allocate(4 + 4 * partitionIds.size()); - out.putInt(partitionIds.size()); - for (int id : partitionIds) { - out.putInt(id); - } - partitionPayload = out.array(); - optionalBytes += payloadSize(partitionPayload); - } - byte[] bucketPayload = EMPTY; - if (bucketAvailable - && 8L + 8L * bucketPairs.size() <= settings.maxBytes - optionalBytes) { - ByteBuffer out = ByteBuffer.allocate(4 + 8 * bucketPairs.size()); - out.putInt(bucketPairs.size()); - for (long pair : bucketPairs) { - out.putInt((int) (pair >>> 32)).putInt((int) pair); - } - bucketPayload = out.array(); - optionalBytes += payloadSize(bucketPayload); - } - blocks.add(new IndexedBlock(current, partitionPayload, rowPayload, bucketPayload)); + blocks.add( + new IndexedBlock( + current, + partitionAvailable + ? encodeValues(partitionIds, partitionIds.size()) + : EMPTY, + rowAvailable ? encodeRanges() : EMPTY, + bucketAvailable + ? encodeValues(bucketPairs, bucketPairs.size()) + : EMPTY)); nextOffset = Math.addExact(current.offset, current.length); nextRecord = Math.addExact(current.firstRecord, current.recordCount); ranges.clear(); @@ -372,53 +263,35 @@ public void endBlock() throws IOException { current = null; } - @Nullable - public byte[] serialize(String name, long fileSize, long entryCount) throws IOException { - if (!complete) { - return null; - } - require(current == null && nextOffset == fileSize && nextRecord == entryCount); - long size = - HEADER_BYTES - + DIGEST_BYTES - + 12L - + header.length - + dictionaryBytes - + blocks.size() * (long) BLOCK_BYTES - + optionalBytes; - // Give directory growth priority over optional coverage. Never remove a descriptor. - for (IndexedBlock block : blocks) { - if (size <= settings.maxBytes) { - break; - } - size -= payloadSize(block.rowIds); - optionalBytes -= payloadSize(block.rowIds); - block.rowIds = EMPTY; - } - for (IndexedBlock block : blocks) { - if (size <= settings.maxBytes) { - break; + private byte[] encodeRanges() throws IOException { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + DataOutputStream out = new DataOutputStream(buffer); + long min = ranges.firstKey(); + long max = ranges.lastEntry().getValue(); + encodeLong(out, ranges.size()); + encodeLong(out, min); + encodeLong(out, max - min); + // The envelope supplies the first start and last end. Encode only interior endpoints. + DeltaRleWriter encoder = new DeltaRleWriter(out, min); + int index = 0; + for (Map.Entry range : ranges.entrySet()) { + if (index > 0) { + encoder.add(range.getKey()); } - size -= payloadSize(block.buckets); - optionalBytes -= payloadSize(block.buckets); - block.buckets = EMPTY; - } - if (size > settings.maxBytes) { - size -= dictionaryBytes; - dictionaryBytes = 0; - dictionary.clear(); - for (IndexedBlock block : blocks) { - size -= payloadSize(block.partitions); - optionalBytes -= payloadSize(block.partitions); - block.partitions = EMPTY; + if (++index < ranges.size()) { + encoder.add(range.getValue()); } } - require(size <= settings.maxBytes); - ByteArrayOutputStream buffer = new ByteArrayOutputStream((int) size); + encoder.finish(); + return buffer.toByteArray(); + } + + public byte[] serialize(long fileSize, long entryCount) throws IOException { + require(current == null && nextOffset == fileSize && nextRecord == entryCount); + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); - out.writeLong(MAGIC); + out.writeInt(MAGIC); out.writeInt(FORMAT_VERSION); - out.write(digest(name.getBytes(StandardCharsets.UTF_8))); out.writeLong(fileSize); out.writeLong(entryCount); out.writeInt(header.length); @@ -441,8 +314,17 @@ public byte[] serialize(String name, long fileSize, long entryCount) throws IOEx return buffer.toByteArray(); } - private static int payloadSize(byte[] payload) { - return payload.length == 0 ? 0 : Integer.BYTES + payload.length; + private static byte[] encodeValues(Iterable values, int count) + throws IOException { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + DataOutputStream out = new DataOutputStream(buffer); + encodeLong(out, count); + DeltaRleWriter encoder = new DeltaRleWriter(out, 0); + for (Number value : values) { + encoder.add(value.longValue()); + } + encoder.finish(); + return buffer.toByteArray(); } private static void writePayload(DataOutputStream out, byte[] payload) throws IOException { @@ -456,9 +338,9 @@ private static void writePayload(DataOutputStream out, byte[] payload) throws IO private static final class IndexedBlock { private final Block block; - private byte[] partitions; - private byte[] rowIds; - private byte[] buckets; + private final byte[] partitions; + private final byte[] rowIds; + private final byte[] buckets; private IndexedBlock(Block block, byte[] partitions, byte[] rowIds, byte[] buckets) { this.block = block; @@ -478,43 +360,36 @@ private static ProjectedManifestEntry.Projection createBlockIndexProjection() { } /** Rebuild from the final physical blocks, including raw-copy and encoded rewrite paths. */ - @Nullable public static byte[] build(FileIO io, Path path, long size, long records, Settings settings) throws IOException { - if (settings.maxBytes < 128) { - return null; - } try (ManifestAvroReader reader = new ManifestAvroReader(io.newInputStream(path))) { Builder builder = new Builder(settings, reader.headerBytes()); ProjectedManifestEntry.Projection projection = BLOCK_INDEX_PROJECTION; ProjectedManifestEntry entry = projection.createEntry(); - while (builder.complete() && reader.hasNext()) { + while (reader.hasNext()) { ManifestAvroReader.RawBlock block = reader.next(); builder.beginBlock(reader.blockOffset(), reader.blockLength(), block.recordCount()); ManifestAvroReader.RowIterator rows = block.toRows(projection.projectedType()); - while (builder.complete() && rows.hasNext()) { + while (rows.hasNext()) { entry.replace(rows.next()); builder.add( settings.rowIdEnabled ? entry.file().firstRowId() : null, settings.rowIdEnabled ? entry.file().rowCount() : 0, - settings.partitionEnabled ? entry.partitionBytes() : null, + entry.partitionBytes(), settings.bucketEnabled ? entry.bucket() : null, settings.bucketEnabled ? entry.totalBuckets() : null); } builder.endBlock(); } - return builder.serialize(path.getName(), size, records); + return builder.serialize(size, records); } } /** Selects blocks using row-ID coverage. A null query retains every block after validation. */ public static Selection select( - byte[] data, - ManifestFileMeta manifest, - @Nullable RowRangeIndex query, - Settings settings) + byte[] data, ManifestFileMeta manifest, @Nullable RowRangeIndex query) throws IOException { - return select(data, manifest, query, null, null, settings); + return select(data, manifest, query, null, null); } /** Validates framing and tests row ID, partition, then bucket coverage. */ @@ -523,10 +398,9 @@ public static Selection select( ManifestFileMeta manifest, @Nullable RowRangeIndex query, @Nullable PartitionPredicate partitionFilter, - @Nullable RowType partitionType, - Settings settings) + @Nullable RowType partitionType) throws IOException { - return select(data, manifest, query, partitionFilter, partitionType, null, settings); + return select(data, manifest, query, partitionFilter, partitionType, null); } /** @@ -539,22 +413,16 @@ public static Selection select( @Nullable RowRangeIndex query, @Nullable PartitionPredicate partitionFilter, @Nullable RowType partitionType, - @Nullable BiPredicate bucketFilter, - Settings settings) + @Nullable BiPredicate bucketFilter) throws IOException { - require(data.length >= 128 && data.length <= settings.maxBytes); + require(data.length >= HEADER_BYTES + DIGEST_BYTES + 12 + 21); int limit = data.length - DIGEST_BYTES; require( MessageDigest.isEqual( digest(data, limit), Arrays.copyOfRange(data, limit, data.length))); ByteBuffer in = ByteBuffer.wrap(data, 0, limit).slice(); - require(in.getLong() == MAGIC); + require(in.getInt() == MAGIC); require(in.getInt() == FORMAT_VERSION); - byte[] hash = new byte[DIGEST_BYTES]; - in.get(hash); - require( - MessageDigest.isEqual( - hash, digest(manifest.fileName().getBytes(StandardCharsets.UTF_8)))); require(in.getLong() == manifest.fileSize()); long entries = Math.addExact(manifest.numAddedFiles(), manifest.numDeletedFiles()); require(in.getLong() == entries); @@ -598,33 +466,34 @@ public static Selection select( long records = in.getLong(); require(offset == nextOffset && length > 0 && length <= manifest.fileSize() - offset); require(records > 0 && records <= entries - firstRecord); - ByteBuffer partitionPayload = payload(in, Integer.BYTES); - ByteBuffer rowPayload = payload(in, 2 * Long.BYTES); - ByteBuffer bucketPayload = payload(in, 2 * Integer.BYTES); + Payload partitionPayload = payload(in, records); + Payload rowPayload = payload(in, records); + Payload bucketPayload = payload(in, records); + require(partitionPayload == null || partitionPayload.count <= partitions); long blockFirstRecord = firstRecord; nextOffset = offset + length; firstRecord += records; if (query != null && rowPayload != null) { - boolean singleRange = rowPayload.remaining() == 2 * Long.BYTES; - long min = rowPayload.getLong(); - long firstEnd = rowPayload.getLong(); - long max = - singleRange - ? firstEnd - : rowPayload.getLong(rowPayload.limit() - Long.BYTES); - require(min >= 0 && firstEnd >= min && max >= firstEnd); + long min = readVarLong(rowPayload.data); + long span = readVarLong(rowPayload.data); + require(span <= Long.MAX_VALUE - min); + long max = min + span; + DeltaRleReader endpoints = + new DeltaRleReader(rowPayload.data, 2L * (rowPayload.count - 1), min, max); if (!query.intersects(min, max)) { continue; } - boolean rowHit = singleRange || query.intersects(min, firstEnd); - long previous = firstEnd; - while (!rowHit && rowPayload.hasRemaining()) { - long rangeStart = rowPayload.getLong(); - long rangeEnd = rowPayload.getLong(); - require(rangeStart >= 0 && rangeEnd >= rangeStart && rangeStart > previous); - previous = rangeEnd; - rowHit = query.intersects(rangeStart, rangeEnd); + boolean rowHit = rowPayload.count == 1; + long start = min; + for (int range = 0; !rowHit && range < rowPayload.count; range++) { + long end = range + 1 == rowPayload.count ? max : endpoints.next(); + require(end >= start); + rowHit = query.intersects(start, end); + if (!rowHit && range + 1 < rowPayload.count) { + start = endpoints.next(); + require(start > end); + } } if (!rowHit) { continue; @@ -632,13 +501,16 @@ public static Selection select( } if (partitionFilter != null && partitionPayload != null) { + DeltaRleReader ids = + new DeltaRleReader( + partitionPayload.data, partitionPayload.count, 0, partitions - 1L); boolean partitionHit = false; - int previous = -1; - while (!partitionHit && partitionPayload.hasRemaining()) { - int id = partitionPayload.getInt(); - require(id > previous && id < partitions); + long previous = -1; + while (!partitionHit && ids.hasNext()) { + long id = ids.next(); + require(id > previous); previous = id; - partitionHit = matches[id]; + partitionHit = matches[(int) id]; } if (!partitionHit) { continue; @@ -646,14 +518,16 @@ public static Selection select( } if (bucketFilter != null && bucketPayload != null) { + DeltaRleReader pairs = + new DeltaRleReader( + bucketPayload.data, bucketPayload.count, 0, Long.MAX_VALUE); boolean bucketHit = false; long previous = -1; - while (!bucketHit && bucketPayload.hasRemaining()) { - int bucket = bucketPayload.getInt(); - int totalBuckets = bucketPayload.getInt(); - require(bucket >= 0 && totalBuckets > bucket); - long pair = ((long) bucket << 32) | totalBuckets; - require(pair > previous); + while (!bucketHit && pairs.hasNext()) { + long pair = pairs.next(); + int bucket = (int) (pair >>> 32); + int totalBuckets = (int) pair; + require(totalBuckets > bucket && pair > previous); previous = pair; bucketHit = bucketFilter.test(bucket, totalBuckets); } @@ -667,9 +541,19 @@ public static Selection select( return new Selection(header, selected); } - /** Reads framing and exposes known payload elements without decoding their contents. */ + private static final class Payload { + private final int count; + private final ByteBuffer data; + + private Payload(int count, ByteBuffer data) { + this.count = count; + this.data = data; + } + } + + /** Reads framing and counts without expanding the compressed contents. */ @Nullable - private static ByteBuffer payload(ByteBuffer in, int elementBytes) throws IOException { + private static Payload payload(ByteBuffer in, long records) throws IOException { require(in.hasRemaining()); int encoding = Byte.toUnsignedInt(in.get()); if (encoding == 0) { @@ -684,21 +568,104 @@ private static ByteBuffer payload(ByteBuffer in, int elementBytes) throws IOExce if (encoding != 1) { return null; } - require(result.remaining() >= Integer.BYTES); - int count = result.getInt(); - require(count > 0 && result.remaining() == (long) elementBytes * count); - return result; + long count = readVarLong(result); + require(count > 0 && count <= records && count <= Integer.MAX_VALUE); + // At least an envelope (row IDs) or one delta run (partitions/buckets) must follow. + require(result.remaining() >= 2); + return new Payload((int) count, result); } - /** Bounded, bulk sidecar reads. Null means read the original manifest. */ + /** Writes equal consecutive deltas as (run length, delta), both unsigned varints. */ + private static final class DeltaRleWriter { + private final DataOutputStream out; + private long previous; + private long delta; + private long repeat; + + private DeltaRleWriter(DataOutputStream out, long base) { + this.out = out; + previous = base; + } + + private void add(long value) throws IOException { + require(value >= previous); + long nextDelta = value - previous; + if (repeat != 0 && nextDelta != delta) { + finish(); + } + delta = nextDelta; + repeat++; + previous = value; + } + + private void finish() throws IOException { + if (repeat > 0) { + encodeLong(out, repeat); + encodeLong(out, delta); + repeat = 0; + } + } + } + + /** Decodes only requested values; a complete read also checks the payload boundary. */ + private static final class DeltaRleReader { + private final ByteBuffer data; + private final long max; + private long remaining; + private long value; + private long repeat; + private long delta; + + private DeltaRleReader(ByteBuffer data, long count, long base, long max) + throws IOException { + require(base >= 0 && max >= base); + this.data = data; + remaining = count; + value = base; + this.max = max; + require(count != 0 || !data.hasRemaining()); + } + + private boolean hasNext() { + return remaining > 0; + } + + private long next() throws IOException { + require(remaining > 0); + if (repeat == 0) { + repeat = readVarLong(data); + delta = readVarLong(data); + require(repeat > 0 && repeat <= remaining); + require(delta == 0 || repeat <= (max - value) / delta); + } + value += delta; + repeat--; + remaining--; + require(remaining != 0 || (repeat == 0 && !data.hasRemaining())); + return value; + } + } + + /** Nonnegative long encoded in one to nine canonical unsigned LEB128 bytes. */ + private static long readVarLong(ByteBuffer in) throws IOException { + long value = 0; + for (int shift = 0; shift < 63; shift += 7) { + require(in.hasRemaining()); + int b = Byte.toUnsignedInt(in.get()); + value |= (long) (b & 0x7f) << shift; + if ((b & 0x80) == 0) { + require(shift == 0 || (b & 0x7f) != 0); + return value; + } + } + throw new IOException("Invalid manifest sidecar varint"); + } + + /** Reads the complete sidecar. Null means read the original manifest. */ @Nullable public static Selection read( - FileIO io, - Path path, - ManifestFileMeta manifest, - @Nullable RowRangeIndex query, - Settings settings) { - return read(io, path, manifest, query, null, null, settings); + FileIO io, Path path, ManifestFileMeta manifest, @Nullable RowRangeIndex query) { + return read(io, path, manifest, query, null, null); } @Nullable @@ -708,10 +675,8 @@ public static Selection read( ManifestFileMeta manifest, @Nullable RowRangeIndex query, @Nullable PartitionPredicate partitionFilter, - @Nullable RowType partitionType, - Settings settings) { - return read( - io, path, manifest, query, partitionFilter, partitionType, null, settings, null); + @Nullable RowType partitionType) { + return read(io, path, manifest, query, partitionFilter, partitionType, null, null); } @Nullable @@ -723,10 +688,9 @@ public static Selection read( @Nullable PartitionPredicate partitionFilter, @Nullable RowType partitionType, @Nullable BiPredicate bucketFilter, - Settings settings, @Nullable SegmentsCache cache) { String sidecarFileName = fileName(manifest); - if (sidecarFileName == null || settings.maxBytes < 128) { + if (sidecarFileName == null) { return null; } try { @@ -736,16 +700,9 @@ public static Selection read( byte[] data = cacheHit ? ((ManifestSidecarSegment) cached).bytes() - : readBytes(io, sidecarPath, settings.maxBytes); + : readBytes(io, sidecarPath); Selection selection = - select( - data, - manifest, - query, - partitionFilter, - partitionType, - bucketFilter, - settings); + select(data, manifest, query, partitionFilter, partitionType, bucketFilter); if (cache != null && !cacheHit && data.length <= cache.maxElementSize()) { cache.put(sidecarPath, new ManifestSidecarSegment(data)); } @@ -777,14 +734,12 @@ public long totalMemorySize() { } } - private static byte[] readBytes(FileIO io, Path path, int maxBytes) throws IOException { + private static byte[] readBytes(FileIO io, Path path) throws IOException { try (InputStream in = io.newInputStream(path)) { ByteArrayOutputStream out = new ByteArrayOutputStream(); - byte[] buffer = new byte[Math.min(READ_BUFFER_BYTES, maxBytes + 1)]; + byte[] buffer = new byte[READ_BUFFER_BYTES]; int n; - while ((n = in.read(buffer, 0, Math.min(buffer.length, maxBytes + 1 - out.size()))) - != -1) { - require(n <= maxBytes - out.size()); + while ((n = in.read(buffer, 0, buffer.length)) != -1) { out.write(buffer, 0, n); } return out.toByteArray(); @@ -1034,8 +989,7 @@ public void close() throws IOException { private static void require(boolean valid) throws IOException { if (!valid) { - throw new IOException( - "Invalid, unsupported, mismatched or over-budget manifest sidecar"); + throw new IOException("Invalid, unsupported or mismatched manifest sidecar"); } } diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java index ba1debc9f984..2a17f6ddb158 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java @@ -21,8 +21,6 @@ import org.apache.paimon.data.BinaryRow; import org.apache.paimon.data.BinaryRowWriter; import org.apache.paimon.data.BinaryString; -import org.apache.paimon.options.MemorySize; -import org.apache.paimon.options.Options; import org.apache.paimon.partition.PartitionPredicate; import org.apache.paimon.predicate.PredicateBuilder; import org.apache.paimon.types.DataTypes; @@ -33,6 +31,8 @@ import org.junit.jupiter.api.Test; +import java.io.ByteArrayOutputStream; +import java.io.DataOutputStream; import java.io.IOException; import java.nio.ByteBuffer; import java.security.MessageDigest; @@ -44,10 +44,7 @@ import java.util.Properties; import java.util.function.BiPredicate; -import static org.apache.paimon.manifest.ManifestSidecarTest.MAX_BYTES; import static org.apache.paimon.manifest.ManifestSidecarTest.meta; -import static org.apache.paimon.manifest.ManifestSidecarTest.settings; -import static org.apache.paimon.manifest.ManifestSidecarTest.sidecarOptions; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.mockito.ArgumentMatchers.any; @@ -58,10 +55,10 @@ import static org.mockito.Mockito.verifyNoInteractions; import static org.mockito.Mockito.verifyNoMoreInteractions; -/** Independent partition/row-ID payloads and conservative resource degradation. */ +/** Complete sidecar payload format, compression and independent filtering. */ class ManifestBlockIndexTest { private final RowType type = RowType.of(DataTypes.INT(), DataTypes.STRING()); - private final ManifestSidecar.Settings defaults = settings(sidecarOptions(), 2); + private final ManifestSidecar.Settings defaults = new ManifestSidecar.Settings(true, true); private byte[] fixture(String field) throws IOException { Properties p = new Properties(); @@ -102,61 +99,46 @@ private PartitionPredicate part(int value) { } @Test - void payloadGenerationCanBeDisabledIndependently() throws Exception { + void partitionCoverageIsAlwaysGenerated() throws Exception { byte[] header = fixture("avroHeader"); - for (int mask = 0; mask < 8; mask++) { - boolean partitionEnabled = (mask & 1) != 0; - boolean rowIdEnabled = (mask & 2) != 0; - boolean bucketEnabled = (mask & 4) != 0; + for (int mask = 0; mask < 4; mask++) { + boolean rowIdEnabled = (mask & 1) != 0; + boolean bucketEnabled = (mask & 2) != 0; ManifestSidecar.Settings settings = - new ManifestSidecar.Settings( - 16 * 1024 * 1024L, partitionEnabled, rowIdEnabled, bucketEnabled); + new ManifestSidecar.Settings(rowIdEnabled, bucketEnabled); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); for (int block = 0; block < 2; block++) { builder.beginBlock(header.length + block * 100L, 100, 1); builder.add(100L + block * 100L, 10, partition(7 + block, "p"), 1, 4); builder.endBlock(); } - byte[] data = builder.serialize("m", header.length + 200, 2); - assertThat(ByteBuffer.wrap(data).getInt(64 + header.length)) - .isEqualTo(partitionEnabled ? 2 : 0); + byte[] data = builder.serialize(header.length + 200, 2); + assertThat(ByteBuffer.wrap(data).getInt(28 + header.length)).isEqualTo(2); for (int[] position : positions(data)) { - assertThat(data[position[1]]).isEqualTo((byte) (partitionEnabled ? 1 : 0)); + assertThat(data[position[1]]).isEqualTo((byte) 1); assertThat(data[position[2]]).isEqualTo((byte) (rowIdEnabled ? 1 : 0)); assertThat(data[position[3]]).isEqualTo((byte) (bucketEnabled ? 1 : 0)); } ManifestFileMeta meta = meta("m", header.length + 200, 2); - assertThat(ManifestSidecar.select(data, meta, query(999), settings).blocks()) + assertThat(ManifestSidecar.select(data, meta, query(999)).blocks()) .hasSize(rowIdEnabled ? 0 : 2); - assertThat(ManifestSidecar.select(data, meta, null, part(99), type, settings).blocks()) - .hasSize(partitionEnabled ? 0 : 2); + assertThat(ManifestSidecar.select(data, meta, null, part(99), type).blocks()).isEmpty(); BiPredicate buckets = bucketFilter(99); - assertThat( - ManifestSidecar.select(data, meta, null, null, type, buckets, settings) - .blocks()) + assertThat(ManifestSidecar.select(data, meta, null, null, type, buckets).blocks()) .hasSize(bucketEnabled ? 0 : 2); // Generation settings do not disable payloads already stored in a sidecar. byte[] existing = fixture("indexWithBuckets"); ManifestFileMeta existingMeta = meta("manifest-golden", header.length + 400, 7); - assertThat( - ManifestSidecar.select(existing, existingMeta, query(999), settings) - .blocks()) + assertThat(ManifestSidecar.select(existing, existingMeta, query(999)).blocks()) .isEmpty(); assertThat( - ManifestSidecar.select( - existing, existingMeta, null, part(99), type, settings) + ManifestSidecar.select(existing, existingMeta, null, part(99), type) .blocks()) .isEmpty(); assertThat( ManifestSidecar.select( - existing, - existingMeta, - null, - null, - type, - buckets, - settings) + existing, existingMeta, null, null, type, buckets) .blocks()) .isEmpty(); } @@ -183,88 +165,53 @@ void jointGoldenPreservesTuplesNullsAndDerivedOrdinals() throws Exception { builder.add(20L, 5, a); builder.add(Long.MAX_VALUE, 1, b); builder.endBlock(); - byte[] data = builder.serialize("manifest-golden", header.length + 400, 7); + byte[] data = builder.serialize(header.length + 400, 7); assertThat(data).isEqualTo(fixture("indexWithPartitions")); ManifestFileMeta meta = meta("manifest-golden", header.length + 400, 7); PartitionPredicate filter = spy(part(7)); - assertThat(ManifestSidecar.select(data, meta, query(20), filter, type, defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, query(20), filter, type).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L, 5L); verify(filter, times(2)).test(any(BinaryRow.class)); PartitionPredicate nullFilter = PartitionPredicate.fromPredicate(type, new PredicateBuilder(type).isNull(1)); - assertThat(ManifestSidecar.select(data, meta, null, nullFilter, type, defaults).blocks()) - .hasSize(3); - assertThat(ManifestSidecar.select(data, meta, null, part(99), type, defaults).blocks()) - .isEmpty(); + assertThat(ManifestSidecar.select(data, meta, null, nullFilter, type).blocks()).hasSize(3); + assertThat(ManifestSidecar.select(data, meta, null, part(99), type).blocks()).isEmpty(); // Missing partition payloads cannot be pruned by dictionary misses. - assertThat( - ManifestSidecar.select( - fixture("index"), meta, null, part(99), type, defaults) - .blocks()) + assertThat(ManifestSidecar.select(fixture("index"), meta, null, part(99), type).blocks()) .hasSize(3); } @Test void unavailableDimensionsAreIndependentAndDoNotPoisonLaterBlocks() throws Exception { - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(512)); - ManifestSidecar.Settings settings = settings(options, 2); + ManifestSidecar.Settings settings = defaults; byte[] header = fixture("avroHeader"); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); builder.beginBlock(header.length, 100, 1); builder.add(null, 10, partition(7, "left")); builder.endBlock(); builder.beginBlock(header.length + 100, 100, 1); - builder.add(200L, 10, partition(9, String.join("", Collections.nCopies(600, "x")))); + builder.add(200L, 10, null); builder.endBlock(); builder.beginBlock(header.length + 200, 100, 1); builder.add(300L, 10, partition(7, "left")); // an existing dictionary ID remains usable builder.endBlock(); - byte[] data = builder.serialize("m", header.length + 300, 3); + byte[] data = builder.serialize(header.length + 300, 3); ManifestFileMeta meta = meta("m", header.length + 300, 3); - assertThat(ManifestSidecar.select(data, meta, null, part(9), type, settings).blocks()) + assertThat(ManifestSidecar.select(data, meta, null, part(9), type).blocks()) .extracting(block -> block.firstRecord) .containsExactly(1L); - assertThat(ManifestSidecar.select(data, meta, query(999), part(7), type, settings).blocks()) + assertThat(ManifestSidecar.select(data, meta, query(999), part(7), type).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L); - assertThat(ManifestSidecar.select(data, meta, query(200), part(9), type, settings).blocks()) + assertThat(ManifestSidecar.select(data, meta, query(200), part(9), type).blocks()) .extracting(block -> block.firstRecord) .containsExactly(1L); } - @Test - void coarseningContinuesThroughTheEntireBlockAndDetectsUnknownRows() throws Exception { - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(512)); - ManifestSidecar.Settings settings = settings(options, 2); - byte[] header = fixture("avroHeader"); - for (boolean unknown : new boolean[] {false, true}) { - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); - builder.beginBlock(header.length, 100, 66); - for (int i = 0; i < 64; i++) { - builder.add(100L + i * 1000L, 10, partition(7, "left")); - } - builder.add(10L, 10, partition(7, "left")); - builder.add(unknown ? null : Long.MAX_VALUE, 1, partition(7, "left")); - builder.endBlock(); - byte[] data = builder.serialize("m", header.length + 100, 66); - ManifestFileMeta meta = meta("m", header.length + 100, 66); - for (long point : new long[] {10, 100, 200, Long.MAX_VALUE}) { - assertThat(ManifestSidecar.select(data, meta, query(point), settings).blocks()) - .hasSize(1); - } - assertThat(ManifestSidecar.select(data, meta, query(0), settings).blocks()) - .hasSize(unknown ? 1 : 0); - assertThat(ManifestSidecar.select(data, meta, null, part(9), type, settings).blocks()) - .isEmpty(); - } - } - private List positions(byte[] data) { ByteBuffer in = ByteBuffer.wrap(data); - in.position(60); + in.position(24); int header = in.getInt(); in.position(in.position() + header); int partitions = in.getInt(); @@ -301,237 +248,23 @@ private byte[] checksum(byte[] data) throws Exception { return data; } - @Test - void rowMissSkipsPartitionAndBucketPayloads() throws Exception { - byte[] data = fixture("indexWithBuckets"); - int[] first = positions(data).get(0); - ByteBuffer.wrap(data).putInt(first[1] + 9, -1); - ByteBuffer.wrap(data).putInt(first[3] + 9, -1); - checksum(data); - ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); - BiPredicate buckets = mock(BiPredicate.class); - assertThat( - ManifestSidecar.select( - data, meta, query(15), part(7), type, buckets, defaults) - .blocks()) - .isEmpty(); - verifyNoInteractions(buckets); - } - - @Test - void partitionMissSkipsBucketMatchingWithOrWithoutRowFilter() throws Exception { - byte[] data = fixture("indexWithBuckets"); - ByteBuffer.wrap(data).putInt(positions(data).get(0)[3] + 9, -1); - checksum(data); - ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); - for (RowRangeIndex rows : Arrays.asList(null, query(0))) { - BiPredicate buckets = mock(BiPredicate.class); - assertThat( - ManifestSidecar.select( - data, meta, rows, part(99), type, buckets, defaults) - .blocks()) - .isEmpty(); - verifyNoInteractions(buckets); - } - } - - @Test - void absentPartitionFilterKeepsRowAndBucketMatching() throws Exception { - byte[] data = fixture("indexWithBuckets"); - ByteBuffer.wrap(data).putInt(positions(data).get(0)[1] + 9, 999); - checksum(data); - ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); - BiPredicate buckets = spy(bucketFilter(1)); - assertThat( - ManifestSidecar.select(data, meta, query(20), null, type, buckets, defaults) - .blocks()) - .extracting(block -> block.firstRecord) - .containsExactly(0L); - verify(buckets).test(1, 4); - verify(buckets).test(0, 1); - verify(buckets).test(3, 4); - verifyNoMoreInteractions(buckets); - } - @Test void absentRowOrBucketFiltersKeepRemainingDimensions() throws Exception { byte[] data = fixture("indexWithBuckets"); ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); assertThat( - ManifestSidecar.select( - data, meta, null, part(7), type, bucketFilter(1), defaults) + ManifestSidecar.select(data, meta, null, part(7), type, bucketFilter(1)) .blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L); - assertThat( - ManifestSidecar.select(data, meta, query(20), part(7), type, null, defaults) - .blocks()) + assertThat(ManifestSidecar.select(data, meta, query(20), part(7), type, null).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L, 5L); - assertThat(ManifestSidecar.select(data, meta, null, null, type, null, defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, null, null, type, null).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L, 3L, 5L); } - @Test - void partitionAndBucketMatchesSkipUnusedPayloadElements() throws Exception { - byte[] partitions = fixture("indexWithBuckets"); - int[] first = positions(partitions).get(0); - ByteBuffer.wrap(partitions).putInt(first[1] + 13, -1); - checksum(partitions); - ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); - assertThat( - ManifestSidecar.select(partitions, meta, query(0), part(7), type, defaults) - .blocks()) - .extracting(block -> block.firstRecord) - .containsExactly(0L); - assertThatThrownBy( - () -> - ManifestSidecar.select( - partitions, meta, query(0), part(99), type, defaults)) - .isInstanceOf(IOException.class); - - byte[] buckets = fixture("indexWithBuckets"); - ByteBuffer.wrap(buckets).putInt(first[3] + 17, -1); - checksum(buckets); - assertThat( - ManifestSidecar.select( - buckets, - meta, - query(0), - null, - type, - bucketFilter(1), - defaults) - .blocks()) - .extracting(block -> block.firstRecord) - .containsExactly(0L); - assertThatThrownBy( - () -> - ManifestSidecar.select( - buckets, - meta, - query(0), - null, - type, - bucketFilter(99), - defaults)) - .isInstanceOf(IOException.class); - } - - @Test - void skippedPayloadsStillRequireValidFramingAndDirectory() throws Exception { - byte[] good = fixture("indexWithBuckets"); - int[] first = positions(good).get(0); - ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); - List invalid = new ArrayList<>(); - byte[] bad = good.clone(); - ByteBuffer.wrap(bad).putInt(first[1] + 5, 0); - invalid.add(bad); - bad = good.clone(); - ByteBuffer.wrap(bad).putInt(first[3] + 1, -1); - invalid.add(bad); - bad = good.clone(); - ByteBuffer.wrap(bad).putInt(first[2] + 5, 0); - invalid.add(bad); - bad = good.clone(); - ByteBuffer.wrap(bad).putLong(positions(good).get(1)[0], 0); - invalid.add(bad); - bad = good.clone(); - ByteBuffer.wrap(bad).putLong(first[0] + 16, 2); - invalid.add(bad); - for (byte[] data : invalid) { - checksum(data); - assertThatThrownBy( - () -> - ManifestSidecar.select( - data, meta, query(15), part(7), type, defaults)) - .isInstanceOf(IOException.class); - } - } - - @Test - void unknownUnsignedEncodingsSkipOnlyTheirDimensionAndMalformedPayloadsFail() throws Exception { - byte[] good = fixture("indexWithBuckets"); - int[] first = positions(good).get(0); - ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); - byte[] data = good.clone(); - data[first[1]] = (byte) 200; - assertThat( - ManifestSidecar.select( - checksum(data), meta, query(0), part(99), type, defaults) - .blocks()) - .extracting(block -> block.firstRecord) - .containsExactly(0L); - data = good.clone(); - data[first[2]] = (byte) 201; - assertThat( - ManifestSidecar.select( - checksum(data), meta, query(16), part(7), type, defaults) - .blocks()) - .extracting(block -> block.firstRecord) - .containsExactly(0L); - data = good.clone(); - data[first[3]] = (byte) 202; - // Unknown encodings skip their payload without decoding even an invalid pair count. - ByteBuffer.wrap(data).putInt(first[3] + 5, 0); - checksum(data); - BiPredicate noBucket = bucketFilter(99); - assertThat( - ManifestSidecar.select( - data, meta, query(20), part(7), type, noBucket, defaults) - .blocks()) - .extracting(block -> block.firstRecord) - .containsExactly(0L); - assertThat( - ManifestSidecar.select( - data, meta, query(999), part(7), type, noBucket, defaults) - .blocks()) - .isEmpty(); - assertThat( - ManifestSidecar.select( - data, meta, query(20), part(99), type, noBucket, defaults) - .blocks()) - .isEmpty(); - for (int position : new int[] {first[1], first[2], first[3]}) { - byte[] bad = good.clone(); - bad[position] = 0; // encoding 0 cannot have a length or payload bytes - checksum(bad); - assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, query(0), defaults)) - .isInstanceOf(IOException.class); - byte[] invalidLength = good.clone(); - invalidLength[position] = (byte) 255; - ByteBuffer.wrap(invalidLength).putInt(position + 1, -1); - checksum(invalidLength); - assertThatThrownBy( - () -> ManifestSidecar.select(invalidLength, meta, query(0), defaults)) - .isInstanceOf(IOException.class); - } - // A checksummed directory with missing bytes/entries must still be rejected. - byte[] bad = good.clone(); - ByteBuffer.wrap(bad).putLong(first[0] + 16, 2); - checksum(bad); - assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, query(0), defaults)) - .isInstanceOf(IOException.class); - byte[] badRange = good.clone(); - // Row payload begins after its encoding and length, then the range-count integer. - ByteBuffer.wrap(badRange).putLong(first[2] + 9 + 16, 9L); - checksum(badRange); - assertThatThrownBy( - () -> - ManifestSidecar.select( - badRange, meta, query(15), part(7), type, defaults)) - .isInstanceOf(IOException.class); - byte[] badId = good.clone(); - ByteBuffer.wrap(badId).putInt(first[1] + 9, 999); - checksum(badId); - assertThatThrownBy( - () -> - ManifestSidecar.select( - badId, meta, query(0), part(7), type, defaults)) - .isInstanceOf(IOException.class); - } - @Test void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { byte[] header = fixture("avroHeader"); @@ -551,32 +284,24 @@ void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { builder.add(20L, 5, a, 0, 1); builder.add(Long.MAX_VALUE, 1, b, 3, 4); builder.endBlock(); - byte[] data = builder.serialize("manifest-golden", header.length + 400, 7); + byte[] data = builder.serialize(header.length + 400, 7); assertThat(data).isEqualTo(fixture("indexWithBuckets")); ManifestFileMeta meta = meta("manifest-golden", header.length + 400, 7); BiPredicate bucket = bucketFilter(1); - assertThat(ManifestSidecar.select(data, meta, null, null, type, bucket, defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, null, null, type, bucket).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L); // Existing row-id and partition coverage remains independently usable. assertThat( - ManifestSidecar.select( - data, - meta, - query(0), - part(7), - type, - bucketFilter(2), - defaults) + ManifestSidecar.select(data, meta, query(0), part(7), type, bucketFilter(2)) .blocks()) .isEmpty(); BiPredicate filter = (bucketId, total) -> bucketId == 2 && total == 8; - assertThat(ManifestSidecar.select(data, meta, null, null, type, filter, defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, null, null, type, filter).blocks()) .extracting(block -> block.firstRecord) .containsExactly(3L); // The caller can omit a bucket filter which requires an actual entry partition. - assertThat(ManifestSidecar.select(data, meta, null, null, type, null, defaults).blocks()) - .hasSize(3); + assertThat(ManifestSidecar.select(data, meta, null, null, type, null).blocks()).hasSize(3); for (String unavailable : new String[] {"index", "indexWithPartitions"}) { assertThat( ManifestSidecar.select( @@ -585,103 +310,50 @@ void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { null, null, type, - bucketFilter(99), - defaults) + bucketFilter(99)) .blocks()) .hasSize(3); } } @Test - void unknownInvalidOrOverBudgetBucketPayloadIsUnavailable() throws Exception { - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(512)); - ManifestSidecar.Settings settings = settings(options, 2); + void unknownOrInvalidBucketPayloadIsUnavailable() throws Exception { + ManifestSidecar.Settings settings = defaults; byte[] header = fixture("avroHeader"); for (Integer[] pair : Arrays.asList( new Integer[] {null, null}, new Integer[] {-1, 4}, new Integer[] {4, 4}, - new Integer[] {0, 0}, - new Integer[] {2, 8})) { + new Integer[] {0, 0})) { ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); - int extraPairs = pair[0] != null && pair[0] == 2 ? 65 : 0; - builder.beginBlock(header.length, 100, 2 + extraPairs); + builder.beginBlock(header.length, 100, 2); builder.add(100L, 10, partition(7, "left"), 1, 4); builder.add(200L, 10, partition(7, "left"), pair[0], pair[1]); - for (int i = 0; i < extraPairs; i++) { - builder.add(200L, 10, partition(7, "left"), i, 100); - } builder.endBlock(); builder.beginBlock(header.length + 100, 100, 1); builder.add(300L, 10, partition(7, "left"), 1, 4); builder.endBlock(); - byte[] data = builder.serialize("m", header.length + 200, 3 + extraPairs); + byte[] data = builder.serialize(header.length + 200, 3); int bucket = positions(data).get(0)[3]; assertThat(data[bucket]).isZero(); assertThat(positions(data).get(1)[0]).isEqualTo(bucket + 1); - ManifestFileMeta meta = meta("m", header.length + 200, 3 + extraPairs); + ManifestFileMeta meta = meta("m", header.length + 200, 3); assertThat( - ManifestSidecar.select( - data, - meta, - null, - null, - type, - bucketFilter(99), - settings) + ManifestSidecar.select(data, meta, null, null, type, bucketFilter(99)) .blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L); assertThat( ManifestSidecar.select( - data, - meta, - query(999), - null, - type, - bucketFilter(99), - settings) + data, meta, query(999), null, type, bucketFilter(99)) .blocks()) .isEmpty(); } } @Test - void malformedBucketPayloadInvalidatesTheContainer() throws Exception { - byte[] good = fixture("indexWithBuckets"); - int payload = positions(good).get(0)[3] + 1; - ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); - for (int[] mutation : - new int[][] { - {payload, -2}, - {payload, Integer.MAX_VALUE}, - {payload, 0}, - {payload + 4, 0}, - {payload + 8, -1}, - {payload + 12, 1}, - {payload + 16, 0} - }) { - byte[] bad = good.clone(); - ByteBuffer.wrap(bad).putInt(mutation[0], mutation[1]); - checksum(bad); - assertThatThrownBy( - () -> - ManifestSidecar.select( - bad, - meta, - null, - null, - type, - bucketFilter(99), - defaults)) - .isInstanceOf(IOException.class); - } - } - - @Test - void payloadsCanExceedFormerLimitsWithinByteBudget() throws Exception { + void largePayloadsKeepExactCoverage() throws Exception { byte[] header = fixture("avroHeader"); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); int blocks = 33; @@ -698,26 +370,18 @@ void payloadsCanExceedFormerLimitsWithinByteBudget() throws Exception { builder.endBlock(); } long fileSize = header.length + (long) blocks * blockBytes; - byte[] data = builder.serialize("m", fileSize, entries); - assertThat(data.length).isLessThanOrEqualTo(defaults.maxBytes); + byte[] data = builder.serialize(fileSize, entries); ManifestFileMeta meta = meta("m", fileSize, entries); long last = (entries - 1L) * 2; - assertThat(ManifestSidecar.select(data, meta, query(last), defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, query(last)).blocks()) .extracting(block -> block.firstRecord) .containsExactly((blocks - 1L) * entriesPerBlock); - assertThat(ManifestSidecar.select(data, meta, query(last - 1), defaults).blocks()) - .isEmpty(); - assertThat(ManifestSidecar.select(data, meta, null, part(entries), type, defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, query(last - 1)).blocks()).isEmpty(); + assertThat(ManifestSidecar.select(data, meta, null, part(entries), type).blocks()) .isEmpty(); assertThat( ManifestSidecar.select( - data, - meta, - null, - null, - type, - bucketFilter(entriesPerBlock), - defaults) + data, meta, null, null, type, bucketFilter(entriesPerBlock)) .blocks()) .isEmpty(); } @@ -736,9 +400,9 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti (mask & 4) == 0 ? null : 1); builder.endBlock(); } - byte[] data = builder.serialize("m", header.length + 800, 8); + byte[] data = builder.serialize(header.length + 800, 8); List positions = positions(data); - int[] presentSizes = {13, 25, 17}; + int[] presentSizes = {8, 8, 8}; for (int mask = 0; mask < 8; mask++) { for (int dimension = 0; dimension < 3; dimension++) { int start = positions.get(mask)[dimension + 1]; @@ -752,52 +416,306 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti } } ManifestFileMeta meta = meta("m", header.length + 800, 8); - assertThat(ManifestSidecar.select(data, meta, null, part(99), type, defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, null, part(99), type).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L, 2L, 4L, 6L); - assertThat(ManifestSidecar.select(data, meta, query(999), defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, query(999)).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L, 1L, 4L, 5L); BiPredicate buckets = bucketFilter(99); - assertThat(ManifestSidecar.select(data, meta, null, null, type, buckets, defaults).blocks()) + assertThat(ManifestSidecar.select(data, meta, null, null, type, buckets).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L, 1L, 2L, 3L); - assertThat( - ManifestSidecar.select( - data, meta, query(999), part(99), type, buckets, defaults) - .blocks()) + assertThat(ManifestSidecar.select(data, meta, query(999), part(99), type, buckets).blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L); } @Test - void tightByteBudgetKeepsAllDescriptorsOrOmitsTheWholeFile() throws Exception { - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(250)); - ManifestSidecar.Settings settings = settings(options, 2); + void deltaRleCompressesSortedPayloadsWithoutCoarseningRowIds() throws Exception { byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); - for (int i = 0; i < 3; i++) { - builder.beginBlock(header.length + 100L * i, 100, 1); - builder.add(i * 100L, 10, partition(7, "left")); - builder.endBlock(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + int count = 10000; + builder.beginBlock(header.length, 100, count); + for (int i = count - 1; i >= 0; i--) { + builder.add(i * 4L, 3, partition(i, null), i, count); + } + builder.endBlock(); + byte[] data = builder.serialize(header.length + 100, count); + int[] block = positions(data).get(0); + for (int dimension = 1; dimension <= 3; dimension++) { + assertThat(ByteBuffer.wrap(data).getInt(block[dimension] + 1)).isLessThan(32); } - byte[] data = builder.serialize("m", header.length + 300, 3); - assertThat(data.length).isLessThanOrEqualTo(250); + ManifestFileMeta meta = meta("m", header.length + 100, count); + assertThat(ManifestSidecar.select(data, meta, query(3)).blocks()).isEmpty(); + assertThat( + ManifestSidecar.select( + data, meta, query(4), part(9999), type, bucketFilter(9999)) + .blocks()) + .hasSize(1); + } + + @Test + void unpartitionedTablesStillRecordTheEmptyPartition() throws Exception { + byte[] header = fixture("avroHeader"); + ManifestSidecar.Builder builder = + new ManifestSidecar.Builder(new ManifestSidecar.Settings(false, false), header); + builder.beginBlock(header.length, 100, 1); + builder.add(null, 0, SerializationUtils.serializeBinaryRow(BinaryRow.EMPTY_ROW)); + builder.endBlock(); + byte[] data = builder.serialize(header.length + 100, 1); + assertThat(ByteBuffer.wrap(data).getInt(28 + header.length)).isEqualTo(1); + int[] block = positions(data).get(0); + assertThat(data[block[1]]).isEqualTo((byte) 1); + assertThat(data[block[2]]).isZero(); + assertThat(data[block[3]]).isZero(); assertThat( ManifestSidecar.select( data, - meta("m", header.length + 300, 3), - query(999), - part(99), - type, - settings) + meta("m", header.length + 100, 1), + null, + null, + RowType.of()) + .blocks()) + .hasSize(1); + } + + @Test + void rowMissSkipsPartitionAndBucketDecoding() throws Exception { + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, varints(2, 1, 999, 1, 1)); + data = replacePayload(data, 0, 3, varints(2, 1, 0, 1, 0)); + BiPredicate buckets = mock(BiPredicate.class); + assertThat( + ManifestSidecar.select( + data, goldenMeta(), query(15), part(7), type, buckets) + .blocks()) + .isEmpty(); + verifyNoInteractions(buckets); + } + + @Test + void partitionMissSkipsBucketDecodingWithOrWithoutRowQuery() throws Exception { + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, varints(2, 1, 0, 1, 0)); + for (RowRangeIndex rows : Arrays.asList(null, query(0))) { + BiPredicate buckets = mock(BiPredicate.class); + assertThat( + ManifestSidecar.select( + data, goldenMeta(), rows, part(99), type, buckets) + .blocks()) + .isEmpty(); + verifyNoInteractions(buckets); + } + } + + @Test + void absentPartitionFilterDoesNotDecodePartitionIds() throws Exception { + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, varints(2, 1, 999, 1, 1)); + BiPredicate buckets = spy(bucketFilter(1)); + assertThat( + ManifestSidecar.select(data, goldenMeta(), query(20), null, type, buckets) .blocks()) .extracting(block -> block.firstRecord) - .containsExactly(0L, 1L, 2L); - builder.beginBlock(header.length + 300, 100, 1); - builder.add(300L, 1, partition(7, "left")); + .containsExactly(0L); + verify(buckets).test(1, 4); + verify(buckets).test(0, 1); + verify(buckets).test(3, 4); + verifyNoMoreInteractions(buckets); + } + + @Test + void matchesSkipUnusedDeltaRuns() throws Exception { + byte[] partitions = + replacePayload(fixture("indexWithBuckets"), 0, 1, varints(2, 1, 0, 1, 999)); + assertThat( + ManifestSidecar.select(partitions, goldenMeta(), query(0), part(7), type) + .blocks()) + .hasSize(1); + assertThatThrownBy( + () -> + ManifestSidecar.select( + partitions, goldenMeta(), query(0), part(99), type)) + .isInstanceOf(IOException.class); + byte[] buckets = + replacePayload( + fixture("indexWithBuckets"), + 0, + 3, + varints(2, 1, (1L << 32) | 4, 1, Long.MAX_VALUE)); + assertThat( + ManifestSidecar.select( + buckets, + goldenMeta(), + query(0), + null, + type, + bucketFilter(1)) + .blocks()) + .hasSize(1); + assertThatThrownBy( + () -> + ManifestSidecar.select( + buckets, + goldenMeta(), + query(0), + null, + type, + bucketFilter(99))) + .isInstanceOf(IOException.class); + + byte[] header = fixture("avroHeader"); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + builder.beginBlock(header.length, 100, 3); + for (long first : new long[] {0, 20, 40}) { + builder.add(first, 10); + } builder.endBlock(); - assertThat(builder.serialize("m", header.length + 400, 4)).isNull(); + byte[] rows = + replacePayload( + builder.serialize(header.length + 100, 3), + 0, + 2, + varints(3, 0, 49, 1, 9, 1, 11, 1, 9, 1, 99)); + ManifestFileMeta meta = meta("m", header.length + 100, 3); + assertThat(ManifestSidecar.select(rows, meta, query(0)).blocks()).hasSize(1); + assertThat(ManifestSidecar.select(rows, meta, query(20)).blocks()).hasSize(1); + assertThat(ManifestSidecar.select(rows, meta, query(100)).blocks()).isEmpty(); + assertThatThrownBy(() -> ManifestSidecar.select(rows, meta, query(35))) + .isInstanceOf(IOException.class); + } + + @Test + void malformedCompressedPayloadsFailWhenConsumed() throws Exception { + List badRows = + Arrays.asList( + varints(2, 0, 24, 0, 9), // Zero-length run. + varints(2, 0, 24, 3, 9), // More values than the interval count allows. + varints(2, 0, 24, 1, 9, 1), // Truncated delta. + varints(2, 0, 24, 1, 9, 1, 0), // Overlapping intervals. + varints(2, 0, 24, 2, Long.MAX_VALUE), // Run exceeds the envelope. + varints(1, Long.MAX_VALUE, 1), // Envelope overflows. + varints(1, 0, 24, 1, 0)); // Unexpected run for a single interval. + for (byte[] payload : badRows) { + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 2, payload); + assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(15))) + .isInstanceOf(IOException.class); + } + for (byte[] payload : Arrays.asList(varints(2, 1, 999, 1, 0), varints(2, 2, 0))) { + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, payload); + assertThatThrownBy( + () -> + ManifestSidecar.select( + data, goldenMeta(), query(0), part(99), type)) + .isInstanceOf(IOException.class); + } + for (byte[] payload : + Arrays.asList( + varints(2, 1, 0, 1, 4), varints(2, 1, 1L << 31, 1, 4), varints(2, 2, 0))) { + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, payload); + assertThatThrownBy( + () -> + ManifestSidecar.select( + data, + goldenMeta(), + query(0), + null, + type, + bucketFilter(99))) + .isInstanceOf(IOException.class); + } + } + + @Test + void invalidVarintsFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception { + List bad = + Arrays.asList( + new byte[] {(byte) 0x80}, + new byte[] {(byte) 0x81, 0, 1, 1}, // Noncanonical count. + new byte[] { + (byte) 0x80, + (byte) 0x80, + (byte) 0x80, + (byte) 0x80, + (byte) 0x80, + (byte) 0x80, + (byte) 0x80, + (byte) 0x80, + (byte) 0x80, + 1 + }, + varints(0, 1, 1), + varints(4, 1, 1), + varints(1)); + for (int dimension = 1; dimension <= 3; dimension++) { + for (byte[] payload : bad) { + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, dimension, payload); + assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) + .isInstanceOf(IOException.class); + } + byte[] data = fixture("indexWithBuckets"); + int position = positions(data).get(0)[dimension]; + ByteBuffer.wrap(data).putInt(position + 1, Integer.MAX_VALUE); + checksum(data); + assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) + .isInstanceOf(IOException.class); + } + byte[] data = fixture("indexWithBuckets"); + int block = positions(data).get(0)[0]; + ByteBuffer.wrap(data).putLong(block + 16, 2); + checksum(data); + assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) + .isInstanceOf(IOException.class); + } + + @Test + void unknownEncodingsRemainIndependent() throws Exception { + for (int dimension = 1; dimension <= 3; dimension++) { + byte[] data = + replacePayload( + fixture("indexWithBuckets"), 0, dimension, new byte[] {(byte) 0x80}); + data[positions(data).get(0)[dimension]] = (byte) 202; + checksum(data); + assertThat( + ManifestSidecar.select( + data, + goldenMeta(), + query(dimension == 2 ? 999 : 0), + part(dimension == 1 ? 99 : 7), + type, + bucketFilter(dimension == 3 ? 99 : 1)) + .blocks()) + .extracting(block -> block.firstRecord) + .containsExactly(0L); + } + } + + private ManifestFileMeta goldenMeta() throws Exception { + return meta("manifest-golden", fixture("avroHeader").length + 400, 7); + } + + private byte[] replacePayload(byte[] data, int block, int dimension, byte[] payload) + throws Exception { + int start = positions(data).get(block)[dimension]; + int end = + data[start] == 0 ? start + 1 : start + 5 + ByteBuffer.wrap(data).getInt(start + 1); + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + DataOutputStream out = new DataOutputStream(buffer); + out.write(data, 0, start); + out.writeByte(1); + out.writeInt(payload.length); + out.write(payload); + out.write(data, end, data.length - end); + return checksum(buffer.toByteArray()); + } + + private static byte[] varints(long... values) { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + for (long value : values) { + while ((value & ~0x7fL) != 0) { + out.write((int) (value & 0x7f) | 0x80); + value >>>= 7; + } + out.write((int) value); + } + return out.toByteArray(); } } diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index d00f740ec111..e89d0bfb596c 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -18,7 +18,8 @@ package org.apache.paimon.manifest; -import org.apache.paimon.CoreOptions; +import org.apache.paimon.data.BinaryRow; +import org.apache.paimon.data.BinaryRowWriter; import org.apache.paimon.data.InternalRow; import org.apache.paimon.data.SingleSegments; import org.apache.paimon.format.FileFormat; @@ -29,8 +30,6 @@ import org.apache.paimon.fs.local.LocalFileIO; import org.apache.paimon.manifest.ManifestSidecar.ManifestSidecarSegment; import org.apache.paimon.memory.MemorySegment; -import org.apache.paimon.options.ConfigOption; -import org.apache.paimon.options.ConfigOptions; import org.apache.paimon.options.MemorySize; import org.apache.paimon.options.Options; import org.apache.paimon.schema.FileSystemSchemaManager; @@ -40,6 +39,7 @@ import org.apache.paimon.utils.Range; import org.apache.paimon.utils.RowRangeIndex; import org.apache.paimon.utils.SegmentsCache; +import org.apache.paimon.utils.SerializationUtils; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; @@ -72,29 +72,8 @@ /** Cross-language format, physical block positions, completeness and allocation bounds. */ class ManifestSidecarTest { - static final ConfigOption MAX_BYTES = - ConfigOptions.key("test.sidecar.max-bytes") - .memoryType() - .defaultValue(MemorySize.ofMebiBytes(16)); - - static ManifestSidecar.Settings settings(Options options, int partitions) { - CoreOptions core = new CoreOptions(options); - return new ManifestSidecar.Settings( - options.get(MAX_BYTES).getBytes(), - partitions > 0, - core.dataEvolutionEnabled(), - core.bucket() != -1); - } - @TempDir java.nio.file.Path temp; - private final ManifestSidecar.Settings settings = settings(sidecarOptions(), 2); - - static Options sidecarOptions() { - Options options = new Options(); - options.set(CoreOptions.DATA_EVOLUTION_ENABLED, true); - options.set(CoreOptions.BUCKET, 4); - return options; - } + private final ManifestSidecar.Settings settings = new ManifestSidecar.Settings(true, true); static ManifestFileMeta meta(String name, long size, long entries) { ManifestFileMeta meta = mock(ManifestFileMeta.class); @@ -126,6 +105,15 @@ private ManifestFileMeta goldenMeta() throws IOException { return meta("manifest-golden", header().length + 400, 7); } + @Test + void emptyManifestHasACompleteSidecar() throws Exception { + byte[] header = header(); + byte[] data = new ManifestSidecar.Builder(settings, header).serialize(header.length, 0); + assertThat(ByteBuffer.wrap(data).getInt()).isEqualTo(0x504d5343); + assertThat(ManifestSidecar.select(data, meta("empty", header.length, 0), null).blocks()) + .isEmpty(); + } + @Test void buildAndReadSelectedBlocksFromPhysicalManifests() throws Exception { FileIO io = LocalFileIO.create(); @@ -170,8 +158,7 @@ void buildAndReadSelectedBlocksFromPhysicalManifests() throws Exception { new Range(3000000000L, 3000000000L))), null, DEFAULT_PART_TYPE, - (bucket, totalBuckets) -> bucket == 1 && totalBuckets == 4, - settings); + (bucket, totalBuckets) -> bucket == 1 && totalBuckets == 4); assertThat(selected.blocks()).hasSize(2); List expected = new ArrayList<>(); for (ManifestSidecar.Block block : selected.blocks()) { @@ -226,7 +213,7 @@ void crossLanguageFormatAndBlockOrdinals() throws Exception { builder.add(20L, 5); builder.add(Long.MAX_VALUE, 1); builder.endBlock(); - byte[] data = builder.serialize("manifest-golden", header.length + 400, 7); + byte[] data = builder.serialize(header.length + 400, 7); assertThat(data).isEqualTo(golden()); ManifestFileMeta meta = goldenMeta(); for (long point : @@ -261,36 +248,10 @@ void crossLanguageFormatAndBlockOrdinals() throws Exception { assertThat(gap.blocks()).isEmpty(); RowRangeIndex query = RowRangeIndex.create(Arrays.asList(new Range(10, 19), new Range(25, 40))); - assertThat(ManifestSidecar.select(data, meta, query, settings).blocks()).isEmpty(); + assertThat(ManifestSidecar.select(data, meta, query).blocks()).isEmpty(); assertThat(query.ranges()).containsExactly(new Range(10, 19), new Range(25, 40)); } - @Test - void settingsEnforceByteArraySizeLimit() { - for (long bytes : - new long[] {0, 127, Integer.MAX_VALUE - 1L, Integer.MAX_VALUE, Long.MAX_VALUE}) { - assertThat(new ManifestSidecar.Settings(bytes, true, true, true).maxBytes) - .isEqualTo((int) Math.min(bytes, Integer.MAX_VALUE - 1L)); - } - assertThatThrownBy(() -> new ManifestSidecar.Settings(-1, true, true, true)) - .isInstanceOf(IllegalArgumentException.class); - } - - @Test - void insufficientByteBudgetSkipsSidecarIo() throws Exception { - FileIO io = mock(FileIO.class); - Path path = new Path(temp.toString(), "manifest-golden"); - ManifestFileMeta meta = goldenMeta(); - for (int bytes : new int[] {0, 1, 127}) { - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(bytes)); - ManifestSidecar.Settings settings = settings(options, 2); - assertThat(ManifestSidecar.read(io, path, meta, null, settings)).isNull(); - assertThat(ManifestSidecar.build(io, path, meta.fileSize(), 7, settings)).isNull(); - } - verifyNoInteractions(io); - } - @Test void minMaxSkipsExactIntersectionChecksAndHandlesOneInterval() throws Exception { byte[] header = header(); @@ -306,11 +267,11 @@ void minMaxSkipsExactIntersectionChecksAndHandlesOneInterval() throws Exception builder.beginBlock(header.length + 200, 100, 1); builder.add(1L << 32, 10); builder.endBlock(); - byte[] data = builder.serialize("m", header.length + 300, 5); + byte[] data = builder.serialize(header.length + 300, 5); ManifestFileMeta meta = meta("m", header.length + 300, 5); RowRangeIndex outside = spy(RowRangeIndex.create(Collections.singletonList(new Range(50, 59)))); - ManifestSidecar.Selection none = ManifestSidecar.select(data, meta, outside, settings); + ManifestSidecar.Selection none = ManifestSidecar.select(data, meta, outside); assertThat(none.blocks()).isEmpty(); // Only the three envelopes are tested; no individual interval intersection is evaluated. @@ -324,7 +285,7 @@ void minMaxSkipsExactIntersectionChecksAndHandlesOneInterval() throws Exception RowRangeIndex.create( Collections.singletonList( new Range((1L << 32) + 9, (1L << 32) + 9)))); - ManifestSidecar.Selection hit = ManifestSidecar.select(data, meta, one, settings); + ManifestSidecar.Selection hit = ManifestSidecar.select(data, meta, one); assertThat(hit.blocks()).extracting(b -> b.firstRecord).containsExactly(4L); // A one-interval block needs no second intersection check after its envelope matches. @@ -343,15 +304,14 @@ void singleIntervalHandlesBoundariesAndAbsentQueries() throws Exception { builder.beginBlock(header.length, 100, 1); builder.add(range.from, range.to - range.from + 1); builder.endBlock(); - byte[] data = builder.serialize("m", header.length + 100, 1); + byte[] data = builder.serialize(header.length + 100, 1); ManifestFileMeta meta = meta("m", header.length + 100, 1); - assertThat(ManifestSidecar.select(data, meta, null, settings).blocks()).hasSize(1); + assertThat(ManifestSidecar.select(data, meta, null).blocks()).hasSize(1); assertThat( ManifestSidecar.select( data, meta, - RowRangeIndex.create(Collections.emptyList()), - settings) + RowRangeIndex.create(Collections.emptyList())) .blocks()) .isEmpty(); for (long point : new long[] {range.from, range.to}) { @@ -359,7 +319,7 @@ void singleIntervalHandlesBoundariesAndAbsentQueries() throws Exception { spy( RowRangeIndex.create( Collections.singletonList(new Range(point, point)))); - assertThat(ManifestSidecar.select(data, meta, query, settings).blocks()).hasSize(1); + assertThat(ManifestSidecar.select(data, meta, query).blocks()).hasSize(1); verify(query).intersects(range.from, range.to); } long missing = range.from > 0 ? range.from - 1 : range.to + 1; @@ -367,64 +327,6 @@ void singleIntervalHandlesBoundariesAndAbsentQueries() throws Exception { } } - @Test - void malformedConsumedIntervalsStillFallBack() throws Exception { - int firstBlockIntervals = 60 + 4 + header().length + 4 + 4 + 24 + 1 + 5 + 4; - ManifestFileMeta meta = goldenMeta(); - for (long[] mutation : new long[][] {{0, -1}, {8, -1}, {8, 30}, {16, 9}, {24, 19}}) { - byte[] data = golden(); - ByteBuffer.wrap(data).putLong(firstBlockIntervals + (int) mutation[0], mutation[1]); - byte[] hash = - MessageDigest.getInstance("SHA-256") - .digest(Arrays.copyOf(data, data.length - 32)); - System.arraycopy(hash, 0, data, data.length - 32, 32); - Files.write(temp.resolve("manifest-golden" + ManifestSidecar.SUFFIX), data); - for (RowRangeIndex query : - Collections.singletonList( - RowRangeIndex.create(Collections.singletonList(new Range(15, 15))))) { - assertThat( - ManifestSidecar.read( - LocalFileIO.create(), - new Path(temp.toString(), "manifest-golden"), - meta, - query, - settings)) - .isNull(); - } - } - } - - @Test - void rowBoundsAndMatchesSkipUnusedIntervals() throws Exception { - byte[] header = header(); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); - builder.beginBlock(header.length, 100, 3); - builder.add(0L, 10); - builder.add(20L, 10); - builder.add(40L, 10); - builder.endBlock(); - byte[] data = builder.serialize("m", header.length + 100, 3); - int intervals = 60 + 4 + header.length + 4 + 4 + 24 + 1 + 5 + 4; - // A checksummed invalid tail must not be visited once the answer is known. - ByteBuffer.wrap(data).putLong(intervals + 32, 19); - byte[] hash = - MessageDigest.getInstance("SHA-256").digest(Arrays.copyOf(data, data.length - 32)); - System.arraycopy(hash, 0, data, data.length - 32, 32); - ManifestFileMeta meta = meta("m", header.length + 100, 3); - assertThat(select(data, meta, 0).blocks()).hasSize(1); - assertThat(select(data, meta, 20).blocks()).hasSize(1); - assertThat(select(data, meta, 100).blocks()).isEmpty(); - assertThat( - ManifestSidecar.select( - data, - meta, - RowRangeIndex.create(Collections.emptyList()), - settings) - .blocks()) - .isEmpty(); - assertThatThrownBy(() -> select(data, meta, 35)).isInstanceOf(IOException.class); - } - @Test void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception { byte[] header = header(); @@ -433,7 +335,7 @@ void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception builder.add(0L, Long.MAX_VALUE); builder.add(Long.MAX_VALUE, 1); builder.endBlock(); - byte[] data = builder.serialize("m", header.length + 100, 2); + byte[] data = builder.serialize(header.length + 100, 2); assertThat(data.length).isLessThan(512); assertThat(select(data, meta("m", header.length + 100, 2), Long.MAX_VALUE).blocks()) .hasSize(1); @@ -444,7 +346,7 @@ void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception builder.endBlock(); assertThat( select( - builder.serialize("m", header.length + 100, 1), + builder.serialize(header.length + 100, 1), meta("m", header.length + 100, 1), 100) .blocks()) @@ -457,15 +359,13 @@ void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception builder.endBlock(); assertThat( select( - builder.serialize("m", header.length + 100, 1), + builder.serialize(header.length + 100, 1), meta("m", header.length + 100, 1), 100) .blocks()) .hasSize(1); } - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(512)); - builder = new ManifestSidecar.Builder(settings(options, 2), header); + builder = new ManifestSidecar.Builder(settings, header); builder.beginBlock(header.length, 100, 64); for (int i = 0; i < 64; i++) { builder.add(i * 10L, 1); @@ -473,18 +373,15 @@ void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception builder.endBlock(); assertThat( select( - builder.serialize("m", header.length + 100, 64), + builder.serialize(header.length + 100, 64), meta("m", header.length + 100, 64), 5) .blocks()) - .hasSize(1); - options.set(MAX_BYTES, new MemorySize(128)); - builder = new ManifestSidecar.Builder(settings(options, 2), header); - assertThat(builder.serialize("m", 1, 2)).isNull(); + .isEmpty(); } @Test - void cacheRespectsElementThresholdAndPerReadByteBudget() throws Exception { + void cacheRespectsElementThreshold() throws Exception { byte[] data = golden(); Path path = new Path(temp.toString(), "manifest-golden"); Path sidecar = ManifestSidecar.path(path); @@ -493,18 +390,15 @@ void cacheRespectsElementThresholdAndPerReadByteBudget() throws Exception { FileIO io = spy(LocalFileIO.create()); SegmentsCache tooSmall = new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), data.length - 1L, null, false); - assertThat(readCached(io, path, meta, settings, tooSmall).blocks()).hasSize(2); - assertThat(readCached(io, path, meta, settings, tooSmall).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, tooSmall).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, tooSmall).blocks()).hasSize(2); assertThat(tooSmall.getIfPresents(sidecar)).isNull(); verify(io, times(2)).newInputStream(sidecar); SegmentsCache cache = new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), data.length, null, false); - assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(data.length - 1)); - assertThat(readCached(io, path, meta, settings(options, 2), cache)).isNull(); - assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, cache).blocks()).hasSize(2); verify(io, times(3)).newInputStream(sidecar); } @@ -519,12 +413,12 @@ void cachedSegmentsRequireSidecarType() throws Exception { cache.put(sidecar, new SingleSegments(MemorySegment.wrap(data), data.length)); FileIO io = spy(LocalFileIO.create()); - assertThat(readCached(io, path, goldenMeta(), settings, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, goldenMeta(), cache).blocks()).hasSize(2); assertThat(cache.getIfPresents(sidecar)).isInstanceOf(ManifestSidecarSegment.class); ManifestSidecarSegment cached = (ManifestSidecarSegment) cache.getIfPresents(sidecar); assertThat(cached.bytes()).containsExactly(data); assertThat(cached.totalMemorySize()).isEqualTo(data.length); - assertThat(readCached(io, path, goldenMeta(), settings, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, goldenMeta(), cache).blocks()).hasSize(2); verify(io, times(1)).newInputStream(sidecar); } @@ -537,16 +431,16 @@ void missingAndInvalidSidecarsAreNotCached() throws Exception { FileIO io = spy(LocalFileIO.create()); SegmentsCache cache = new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), Long.MAX_VALUE, null, false); - assertThat(readCached(io, path, meta, settings, cache)).isNull(); + assertThat(readCached(io, path, meta, cache)).isNull(); assertThat(cache.getIfPresents(sidecar)).isNull(); byte[] corrupt = data.clone(); corrupt[0] ^= 1; Files.write(temp.resolve(sidecar.getName()), corrupt); - assertThat(readCached(io, path, meta, settings, cache)).isNull(); + assertThat(readCached(io, path, meta, cache)).isNull(); assertThat(cache.getIfPresents(sidecar)).isNull(); Files.write(temp.resolve(sidecar.getName()), data); - assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); - assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, cache).blocks()).hasSize(2); verify(io, times(3)).newInputStream(sidecar); } @@ -565,28 +459,22 @@ void cachedBytesPreservePerQueryCancellation() throws Exception { assertThatThrownBy( () -> ManifestSidecar.read( - io, path, meta, cancelled, null, null, null, settings, - cache)) + io, path, meta, cancelled, null, null, null, cache)) .isInstanceOf(CancellationException.class); assertThat(cache.getIfPresents(sidecar)).isNull(); - assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, cache).blocks()).hasSize(2); assertThatThrownBy( () -> ManifestSidecar.read( - io, path, meta, cancelled, null, null, null, settings, - cache)) + io, path, meta, cancelled, null, null, null, cache)) .isInstanceOf(CancellationException.class); - assertThat(readCached(io, path, meta, settings, cache).blocks()).hasSize(2); + assertThat(readCached(io, path, meta, cache).blocks()).hasSize(2); verify(io, times(2)).newInputStream(sidecar); } private ManifestSidecar.Selection readCached( - FileIO io, - Path path, - ManifestFileMeta meta, - ManifestSidecar.Settings settings, - SegmentsCache cache) { + FileIO io, Path path, ManifestFileMeta meta, SegmentsCache cache) { return ManifestSidecar.read( io, path, @@ -595,7 +483,6 @@ private ManifestSidecar.Selection readCached( null, null, null, - settings, cache); } @@ -606,39 +493,49 @@ void corruptMissingIncompleteAndMismatchedIndexesFallback() throws Exception { ManifestFileMeta meta = goldenMeta(); RowRangeIndex query = RowRangeIndex.create(Collections.singletonList(new Range(11, 11))); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query, settings)) - .isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); byte[] good = golden(); for (int position : new int[] {0, 9, 11, 15, 16, 55, 63, 67, 75, good.length - 1}) { byte[] bad = good.clone(); bad[position] ^= 2; Files.write(index, bad); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query, settings)) - .isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); } // A valid checksum cannot make an unsupported container version readable. for (int version : new int[] {0, 2, 99}) { byte[] bad = good.clone(); - ByteBuffer.wrap(bad).putInt(8, version); + ByteBuffer.wrap(bad).putInt(4, version); byte[] hash = MessageDigest.getInstance("SHA-256") .digest(Arrays.copyOf(bad, bad.length - 32)); System.arraycopy(hash, 0, bad, bad.length - 32, 32); - assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, query, settings)) + assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, query)) .isInstanceOf(IOException.class); } Files.write(index, Arrays.copyOf(good, good.length - 1)); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query, settings)) - .isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); Files.write(index, good); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query).blocks()) + .isEmpty(); + // The sidecar is bound to physical coverage, not to a particular file name. assertThat( - ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query, settings) + ManifestSidecar.select( + good, + meta("renamed", meta.fileSize(), 7), + RowRangeIndex.create( + Collections.singletonList(new Range(20, 20)))) .blocks()) - .isEmpty(); + .extracting(block -> block.firstRecord) + .containsExactly(0L, 5L); + assertThatThrownBy( + () -> + ManifestSidecar.select( + good, meta("renamed", meta.fileSize() + 1, 7), query)) + .isInstanceOf(IOException.class); assertThatThrownBy( () -> ManifestSidecar.select( - good, meta("other", meta.fileSize(), 7), query, settings)) + good, meta("renamed", meta.fileSize(), 8), query)) .isInstanceOf(IOException.class); } @@ -656,8 +553,7 @@ void ioFailuresFallBackWithoutInspectingNestedExceptions() throws Exception { suppressed)) { FileIO fileIO = mock(FileIO.class); when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); - assertThat(ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null, settings)) - .isNull(); + assertThat(ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)).isNull(); assertThat(Thread.currentThread().isInterrupted()).isFalse(); } } @@ -670,10 +566,7 @@ void interruptedThreadDoesNotFallBackOnIoFailure() throws Exception { when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); try { Thread.currentThread().interrupt(); - assertThatThrownBy( - () -> - ManifestSidecar.read( - fileIO, path, meta("m", 1, 1), null, settings)) + assertThatThrownBy(() -> ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)) .isInstanceOf(java.io.UncheckedIOException.class) .hasCauseReference(failure); assertThat(Thread.currentThread().isInterrupted()).isTrue(); @@ -693,10 +586,7 @@ void uncheckedFailuresPropagateUnchanged() throws Exception { new AssertionError("error"))) { FileIO fileIO = mock(FileIO.class); when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); - assertThatThrownBy( - () -> - ManifestSidecar.read( - fileIO, path, meta("m", 1, 1), null, settings)) + assertThatThrownBy(() -> ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)) .isSameAs(failure); } } @@ -712,7 +602,7 @@ void indexReadsUseBoundedBulkRequests() throws Exception { builder.endBlock(); } long size = header.length + blockCount * 100L; - byte[] data = builder.serialize("manifest-large", size, blockCount); + byte[] data = builder.serialize(size, blockCount); ManifestFileMeta meta = meta("manifest-large", size, blockCount); CountingInput stream = new CountingInput(data, Integer.MAX_VALUE); Path path = new Path(temp.toString(), meta.fileName()); @@ -723,8 +613,7 @@ void indexReadsUseBoundedBulkRequests() throws Exception { io, path, meta, - RowRangeIndex.create(Collections.singletonList(new Range(0, 0))), - settings); + RowRangeIndex.create(Collections.singletonList(new Range(0, 0)))); assertThat(actual.blocks()).hasSize(1); assertThat(actual.blocks().get(0).offset).isEqualTo(header.length); assertThat(stream.readLengths).hasSize((data.length + (1 << 20) - 1) / (1 << 20)); @@ -734,10 +623,40 @@ void indexReadsUseBoundedBulkRequests() throws Exception { } @Test - void indexShortReadsAndExactBudget() throws Exception { + void sidecarsLargerThanTheFormerDefaultLimitAreReadCompletely() throws Exception { + byte[] header = header(); + BinaryRow partition = new BinaryRow(1); + BinaryRowWriter rowWriter = new BinaryRowWriter(partition); + byte[] value = new byte[17 * 1024 * 1024]; + rowWriter.writeBinary(0, value, 0, value.length); + rowWriter.complete(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + builder.beginBlock(header.length, 100, 1); + builder.add(0L, 1, SerializationUtils.serializeBinaryRow(partition)); + builder.endBlock(); + byte[] data = builder.serialize(header.length + 100, 1); + assertThat(data.length).isGreaterThan(16 * 1024 * 1024); + Path path = new Path(temp.toString(), "manifest-large"); + CountingInput stream = new CountingInput(data, Integer.MAX_VALUE); + FileIO io = mock(FileIO.class); + when(io.newInputStream(ManifestSidecar.path(path))).thenReturn(stream); + assertThat( + ManifestSidecar.read( + io, + path, + meta("manifest-large", header.length + 100, 1), + null) + .blocks()) + .hasSize(1); + assertThat(stream.readLengths.stream().mapToInt(Integer::intValue).sum()) + .isEqualTo(data.length); + assertThat(stream.requests).allMatch(request -> request <= 1 << 20); + assertThat(stream.closed).isTrue(); + } + + @Test + void indexShortReadsReadTheWholeFile() throws Exception { byte[] data = golden(); - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(data.length)); Path path = new Path(temp.toString(), "manifest-golden"); for (int maxRead : new int[] {Integer.MAX_VALUE, 7}) { CountingInput stream = new CountingInput(data, maxRead); @@ -748,8 +667,7 @@ void indexShortReadsAndExactBudget() throws Exception { io, path, goldenMeta(), - RowRangeIndex.create(Collections.singletonList(new Range(20, 20))), - settings(options, 2)); + RowRangeIndex.create(Collections.singletonList(new Range(20, 20)))); assertThat(actual.blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L, 5L); @@ -757,26 +675,6 @@ void indexShortReadsAndExactBudget() throws Exception { } } - @Test - void indexOverBudgetStopsAfterOneExtraByte() throws Exception { - Options options = sidecarOptions(); - options.set(MAX_BYTES, new MemorySize(128)); - Path path = new Path(temp.toString(), "manifest-golden"); - CountingInput stream = new CountingInput(golden(), Integer.MAX_VALUE); - FileIO io = mock(FileIO.class); - when(io.newInputStream(ManifestSidecar.path(path))).thenReturn(stream); - assertThat( - ManifestSidecar.read( - io, - path, - goldenMeta(), - RowRangeIndex.create(Collections.singletonList(new Range(20, 20))), - settings(options, 2))) - .isNull(); - assertThat(stream.readLengths).containsExactly(129); - assertThat(stream.closed).isTrue(); - } - @Test void adjacentBlocksShareReadsForSingleByteConsumers() throws Exception { byte[] header = header(); @@ -791,8 +689,7 @@ void adjacentBlocksShareReadsForSingleByteConsumers() throws Exception { RowRangeIndex.create( Arrays.asList( new Range(0, 0), - new Range(8254058425445L, 8254058425445L))), - settings); + new Range(8254058425445L, 8254058425445L)))); byte[] manifest = Arrays.copyOf(header, header.length + body.length); System.arraycopy(body, 0, manifest, header.length, body.length); CountingInput stream = new CountingInput(manifest, Integer.MAX_VALUE); @@ -864,8 +761,7 @@ void cachedBlocksAreSharedByDifferentSelectionsWithoutOpeningTheManifest() throw golden(), goldenMeta(), RowRangeIndex.create( - Collections.singletonList(new Range(0, Long.MAX_VALUE))), - settings); + Collections.singletonList(new Range(0, Long.MAX_VALUE)))); try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); } @@ -920,8 +816,7 @@ io, path, select(golden(), goldenMeta(), 8254058425445L), cache)) { golden(), goldenMeta(), RowRangeIndex.create( - Collections.singletonList(new Range(0, Long.MAX_VALUE))), - settings); + Collections.singletonList(new Range(0, Long.MAX_VALUE)))); try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); } @@ -950,8 +845,7 @@ void truncatedCoalescedReadsDoNotPopulateTheBlockCache() throws Exception { golden(), goldenMeta(), RowRangeIndex.create( - Collections.singletonList(new Range(0, Long.MAX_VALUE))), - settings); + Collections.singletonList(new Range(0, Long.MAX_VALUE)))); try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { assertThatThrownBy(() -> IOUtils.readFully(in, false)).isInstanceOf(EOFException.class); } @@ -979,8 +873,7 @@ void evictedBlocksAreReadAgainWithinTheSharedBudget() throws Exception { golden(), goldenMeta(), RowRangeIndex.create( - Collections.singletonList(new Range(0, Long.MAX_VALUE))), - settings); + Collections.singletonList(new Range(0, Long.MAX_VALUE)))); for (int round = 0; round < 2; round++) { try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); @@ -1006,7 +899,7 @@ void oversizedBlocksUseBoundedReadsWithoutModifyingPreviouslyCachedBytes() throw byte[] manifest = Arrays.copyOf(header, header.length + cachedLength + uncachedLength); Arrays.fill(manifest, header.length, header.length + cachedLength, (byte) 7); Arrays.fill(manifest, header.length + cachedLength, manifest.length, (byte) 9); - byte[] data = builder.serialize("large", manifest.length, 2); + byte[] data = builder.serialize(manifest.length, 2); ManifestFileMeta meta = meta("large", manifest.length, 2); Path path = new Path(temp.toString(), "large"); FileIO io = mock(FileIO.class); @@ -1026,8 +919,7 @@ void oversizedBlocksUseBoundedReadsWithoutModifyingPreviouslyCachedBytes() throw data, meta, RowRangeIndex.create( - Collections.singletonList(new Range(0, Long.MAX_VALUE))), - settings); + Collections.singletonList(new Range(0, Long.MAX_VALUE)))); try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); } @@ -1055,7 +947,7 @@ void largeBlockSpansUseBoundedReads() throws Exception { builder.endBlock(); offset += length; } - byte[] data = builder.serialize("manifest-large", offset, 3); + byte[] data = builder.serialize(offset, 3); byte[] manifest = Arrays.copyOf(header, (int) offset); CountingInput stream = new CountingInput(manifest, Integer.MAX_VALUE); FileIO io = mock(FileIO.class); @@ -1080,8 +972,7 @@ void blockShortReadsAndTruncation() throws Exception { golden(), goldenMeta(), RowRangeIndex.create( - Collections.singletonList(new Range(0, Long.MAX_VALUE))), - settings); + Collections.singletonList(new Range(0, Long.MAX_VALUE)))); for (int bodyLength : new int[] {400, 399}) { byte[] manifest = Arrays.copyOf(header, header.length + bodyLength); CountingInput stream = new CountingInput(manifest, 7); @@ -1139,7 +1030,6 @@ private ManifestSidecar.Selection select(byte[] data, ManifestFileMeta meta, lon return ManifestSidecar.select( data, meta, - RowRangeIndex.create(Collections.singletonList(new Range(point, point))), - settings); + RowRangeIndex.create(Collections.singletonList(new Range(point, point)))); } } diff --git a/paimon-core/src/test/resources/manifest-sidecar.txt b/paimon-core/src/test/resources/manifest-sidecar.txt index f2199ce2e4f0..4562bd1916ef 100644 --- a/paimon-core/src/test/resources/manifest-sidecar.txt +++ b/paimon-core/src/test/resources/manifest-sidecar.txt @@ -18,6 +18,6 @@ avroHeader=T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAA partitionA=AAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhA== partitionB=AAAAAgACAAAAAAAACQAAAAAAAAAAAAAAAAAAAA== -index=UEFJTVNDQVIAAAABL0eWubkHdL6ioNdIPj3jGtp9TAk83wbtAsmc+xJVwM8AAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAADkAAAAAAAAAZAAAAAAAAAADAAEAAAAkAAAAAgAAAAAAAAAAAAAAAAAAAAkAAAAAAAAAFAAAAAAAAAAYAAAAAAAAAACdAAAAAAAAAMgAAAAAAAAAAgABAAAAJAAAAAIAAAAA/////gAAAAEAAAACAAAHgcw4bGUAAAeBzDhsZQAAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIAAQAAACQAAAACAAAAAAAAABQAAAAAAAAAGH//////////f/////////8AlOHbHSWYVK7dgM+l6yo0+LjOLGYNf4lnkt2OtValVYA= -indexWithPartitions=UEFJTVNDQVIAAAABL0eWubkHdL6ioNdIPj3jGtp9TAk83wbtAsmc+xJVwM8AAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAMAAAAAgAAAAAAAAABAQAAACQAAAACAAAAAAAAAAAAAAAAAAAACQAAAAAAAAAUAAAAAAAAABgAAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAwAAAACAAAAAAAAAAEBAAAAJAAAAAIAAAAA/////gAAAAEAAAACAAAHgcw4bGUAAAeBzDhsZQAAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAADAAAAAIAAAAAAAAAAQEAAAAkAAAAAgAAAAAAAAAUAAAAAAAAABh//////////3//////////AInpuqbYtRIEgHOW0YNU5V+bvWbXg6ARpSP/hXdwewB8 -indexWithBuckets=UEFJTVNDQVIAAAABL0eWubkHdL6ioNdIPj3jGtp9TAk83wbtAsmc+xJVwM8AAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAMAAAAAgAAAAAAAAABAQAAACQAAAACAAAAAAAAAAAAAAAAAAAACQAAAAAAAAAUAAAAAAAAABgBAAAAFAAAAAIAAAABAAAABAAAAAEAAAAIAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAwAAAACAAAAAAAAAAEBAAAAJAAAAAIAAAAA/////gAAAAEAAAACAAAHgcw4bGUAAAeBzDhsZQEAAAAUAAAAAgAAAAIAAAAEAAAAAgAAAAgAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAADAAAAAIAAAAAAAAAAQEAAAAkAAAAAgAAAAAAAAAUAAAAAAAAABh//////////3//////////AQAAABQAAAACAAAAAAAAAAEAAAADAAAABGCAy0ixcqKywlCau1+E/XLEorySmCmL1zl5q9/dwdBu +index=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAADkAAAAAAAAAZAAAAAAAAAADAAEAAAAHAgAYAQkBCwAAAAAAAAAAnQAAAAAAAADIAAAAAAAAAAIAAQAAABcC/v///w/n2OHhjPABAQQB49jh4YzwAQAAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIAAQAAABcCFOv/////////fwEEAef/////////fwDIf1W/eeutccPur0sI5iU31VsVOlXBzAZAV924R411vg== +indexWithPartitions=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAFAgEAAQEBAAAABwIAGAEJAQsAAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAUCAQABAQEAAAAXAv7///8P59jh4YzwAQEEAePY4eGM8AEAAAAAAAAAAWUAAAAAAAAAZAAAAAAAAAACAQAAAAUCAQABAQEAAAAXAhTr/////////38BBAHn/////////38ASyBR9j5LPqWbu2lXKJ/5rDuhDcH5F6h6kj6gIu5QKrs= +indexWithBuckets=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAFAgEAAQEBAAAABwIAGAEJAQsBAAAACQIBhICAgBABBAAAAAAAAACdAAAAAAAAAMgAAAAAAAAAAgEAAAAFAgEAAQEBAAAAFwL+////D+fY4eGM8AEBBAHj2OHhjPABAQAAAAkCAYSAgIAgAQQAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAABQIBAAEBAQAAABcCFOv/////////fwEEAef/////////fwEAAAAJAgEBAYOAgIAw4U9my1FPNEd5PdRGwwFFpTJWFrdan4tnQ0m2/YUbtYY= From 2ac84f3957f54c1a81364ac1475803025de018c3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 20:36:21 +0800 Subject: [PATCH 03/14] [docs] Remove redundant manifest sidecar hash explanation --- docs/docs/concepts/spec/manifest.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 835bcafffe3f..c39a81d76f69 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -122,10 +122,6 @@ blocks[] // original physical order checksum : 32 bytes // SHA-256 of all preceding bytes ``` -The sidecar contains no manifest-name hash. Renaming the manifest does not change sidecar -bytes. Its stored length and entry count must match the supplied manifest metadata; the -caller must associate the sidecar with the correct immutable manifest through its reference. - The block ID is its position. Its first entry ordinal is the sum of preceding record counts and is not stored. Each complete partition tuple appears once in the dictionary, including all its fields and nulls. The scan's partition type interprets the existing serialized tuple. From e5cb4b81f262c03b3af2db923ce7a59b0c7c0cfd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 20:42:23 +0800 Subject: [PATCH 04/14] [core] Restore manifest sidecar read and write settings --- docs/docs/concepts/spec/manifest.md | 5 +++-- .../java/org/apache/paimon/manifest/ManifestSidecar.java | 8 ++++++-- .../apache/paimon/manifest/ManifestBlockIndexTest.java | 8 +++++--- .../org/apache/paimon/manifest/ManifestSidecarTest.java | 3 ++- 4 files changed, 16 insertions(+), 8 deletions(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index c39a81d76f69..9a137f446fac 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -77,8 +77,9 @@ sidecar references, managing file ownership, applying entry filters and reconcil entries after block selection. `build` reads the completed physical manifest and returns sidecar bytes; it does not write or publish another file. -`Settings` enables row-ID and bucket payload generation independently. Partition generation is -always enabled, including the empty partition tuple for unpartitioned tables. Missing or invalid +`Settings` contains `write` and `read` switches for the calling writer and scan, and enables +row-ID and bucket payload generation independently. Partition generation is always enabled, +including the empty partition tuple for unpartitioned tables. Missing or invalid metadata makes only the affected block's dimension unavailable. There is no sidecar byte budget: construction keeps complete coverage and `read` consumes the entire file once it is opened. diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index 9edea887b156..5593ca2e28bc 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -91,12 +91,16 @@ public static String fileName(ManifestFileMeta manifest) { return null; } - /** Optional payloads supplied by the caller. Partition coverage is always enabled. */ + /** Read/write switches and optional payloads. Partition coverage is always enabled. */ public static final class Settings { + public final boolean write; + public final boolean read; public final boolean rowIdEnabled; public final boolean bucketEnabled; - public Settings(boolean rowIdEnabled, boolean bucketEnabled) { + public Settings(boolean write, boolean read, boolean rowIdEnabled, boolean bucketEnabled) { + this.write = write; + this.read = read; this.rowIdEnabled = rowIdEnabled; this.bucketEnabled = bucketEnabled; } diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java index 2a17f6ddb158..5f9ec7b1afe9 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java @@ -58,7 +58,8 @@ /** Complete sidecar payload format, compression and independent filtering. */ class ManifestBlockIndexTest { private final RowType type = RowType.of(DataTypes.INT(), DataTypes.STRING()); - private final ManifestSidecar.Settings defaults = new ManifestSidecar.Settings(true, true); + private final ManifestSidecar.Settings defaults = + new ManifestSidecar.Settings(true, true, true, true); private byte[] fixture(String field) throws IOException { Properties p = new Properties(); @@ -105,7 +106,7 @@ void partitionCoverageIsAlwaysGenerated() throws Exception { boolean rowIdEnabled = (mask & 1) != 0; boolean bucketEnabled = (mask & 2) != 0; ManifestSidecar.Settings settings = - new ManifestSidecar.Settings(rowIdEnabled, bucketEnabled); + new ManifestSidecar.Settings(true, true, rowIdEnabled, bucketEnabled); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); for (int block = 0; block < 2; block++) { builder.beginBlock(header.length + block * 100L, 100, 1); @@ -459,7 +460,8 @@ data, meta, query(4), part(9999), type, bucketFilter(9999)) void unpartitionedTablesStillRecordTheEmptyPartition() throws Exception { byte[] header = fixture("avroHeader"); ManifestSidecar.Builder builder = - new ManifestSidecar.Builder(new ManifestSidecar.Settings(false, false), header); + new ManifestSidecar.Builder( + new ManifestSidecar.Settings(true, true, false, false), header); builder.beginBlock(header.length, 100, 1); builder.add(null, 0, SerializationUtils.serializeBinaryRow(BinaryRow.EMPTY_ROW)); builder.endBlock(); diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index e89d0bfb596c..daca76c42b19 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -73,7 +73,8 @@ /** Cross-language format, physical block positions, completeness and allocation bounds. */ class ManifestSidecarTest { @TempDir java.nio.file.Path temp; - private final ManifestSidecar.Settings settings = new ManifestSidecar.Settings(true, true); + private final ManifestSidecar.Settings settings = + new ManifestSidecar.Settings(true, true, true, true); static ManifestFileMeta meta(String name, long size, long entries) { ManifestFileMeta meta = mock(ManifestFileMeta.class); From aedfb48315916344915a1fd39da357bb3b9bfb49 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 21:33:07 +0800 Subject: [PATCH 05/14] [core] Use fixed-width manifest sidecar payload prefixes --- docs/docs/concepts/spec/manifest.md | 34 +++-- .../paimon/manifest/ManifestSidecar.java | 24 ++-- .../manifest/ManifestBlockIndexTest.java | 132 +++++++++++++----- .../src/test/resources/manifest-sidecar.txt | 6 +- 4 files changed, 131 insertions(+), 65 deletions(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 9a137f446fac..1985e591901e 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -90,7 +90,8 @@ supplied bytes directly and reports invalid containers with `IOException`. Version 1 uses the following layout. Container `int` and `long` fields are signed, fixed-width 4-byte and 8-byte big-endian integers. Encoding IDs are unsigned bytes with separate namespaces. -Payload integers use the variable-length encoding described below. +Payload counts and envelopes use the same fixed-width types; delta/RLE runs use the +variable-length encoding described below. ```text magic : 4 bytes // ASCII PMSC @@ -143,9 +144,11 @@ Encoding 0 represents unavailable coverage, rather than encoding 1 with a zero c #### Delta and RLE Encoding -Every integer inside an encoding-1 payload is a nonnegative unsigned LEB128 varint, using -one to nine bytes for values from 0 through `Long.MAX_VALUE`. Seven value bits are stored -per byte, least significant group first; the high bit indicates another byte follows. +Each payload starts with a fixed-width count (`int`); row-ID payloads also have fixed-width +`min` and `span` fields (`long`). Only integers in the following delta/RLE stream use +nonnegative unsigned LEB128 varints, occupying one to nine bytes for values from 0 through +`Long.MAX_VALUE`. Seven value bits are stored per byte, least significant group first; the +high bit indicates another byte follows. Encodings use the shortest representation. There is no ZigZag transformation or padding. A sorted sequence is delta-encoded from a specified base. Consecutive equal deltas are @@ -170,15 +173,16 @@ represented by its entries: ```text partitionPayload - partitionIdCount : varint // N > 0 + partitionIdCount : int // N > 0 runs[] // N dictionary IDs, base = 0 ``` An ID is the zero-based position of a complete tuple in the sidecar's shared dictionary. IDs satisfy `0 <= id < partitionCount` and are strictly increasing. Tuple bytes appear only in the dictionary and are not repeated in each block. For IDs `[0, 1, 2, 3, 4]`, the deltas -are `[0, 1, 1, 1, 1]` and the runs are `(1, 0), (4, 1)`. The complete payload bytes are -`[5, 1, 0, 4, 1]`: 5 bytes, or 10 bytes including the encoding and length fields. +are `[0, 1, 1, 1, 1]` and the runs are `(1, 0), (4, 1)`. The payload contains a four-byte +count of 5 followed by the run bytes `[1, 0, 4, 1]`: 8 bytes, or 13 bytes including the +encoding and length fields. With a partition filter, a block matches if any referenced tuple matches. A tuple containing a null partition value still has a dictionary ID. Unpartitioned tables record the empty @@ -193,9 +197,9 @@ sorted and disjoint; they are never expanded into individual row IDs or coarsene ```text rowIdPayload - rangeCount : varint // N > 0 - min : varint // first interval's start - span : varint // last interval's end minus min + rangeCount : int // N > 0 + min : long // first interval's start + span : long // last interval's end minus min runs[] // 2 * (N - 1) interior endpoints, base = min ``` @@ -207,8 +211,10 @@ Pairing the reconstructed endpoints recovers the intervals. Each pair satisfies For `[(10, 19), (30, 39)]`, the count is 2, minimum is 10, and span is 29. The interior endpoints `[19, 30]` have deltas `[9, 11]` from base 10, encoded as `(1, 9), (1, 11)`. -The complete payload bytes are `[2, 10, 29, 1, 9, 1, 11]`: 7 bytes, or 12 bytes with framing. -For a single interval, the envelope completely defines the interval and no runs follow. +The payload starts with a four-byte count of 2, an eight-byte minimum of 10, and an eight-byte +span of 29, followed by the run bytes `[1, 9, 1, 11]`: 24 bytes, or 29 bytes with framing. +For a single interval, the 20-byte fixed-width prefix completely defines the interval and +no runs follow. The reader first tests the envelope without expanding any runs. A query for row ID 25 passes the example's envelope check but matches neither interval. Unknown or invalid row-ID @@ -220,7 +226,7 @@ When `bucketEncoding == 1`, the block stores distinct bucket/count pairs: ```text bucketPayload - pairCount : varint // N > 0 + pairCount : int // N > 0 runs[] // N packed pairs, base = 0 packedPair = ((long) bucket << 32) | totalBuckets @@ -234,7 +240,7 @@ The same bucket may occur with different totals after rescaling. For `[(1, 4), (1, 8), (3, 4)]`, the packed values are `[4294967300, 4294967304, 12884901892]` and deltas are `[4294967300, 4, 8589934588]`. The payload contains count 3 and three runs -of length 1, occupying 15 bytes, or 20 bytes with framing. Repeated bucket strides with the +of length 1, occupying 18 bytes, or 23 bytes with framing. Repeated bucket strides with the same total bucket count form a single run. Missing, invalid or negative/synthetic bucket metadata makes the block's bucket coverage diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index 5593ca2e28bc..44abefb68b82 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -272,9 +272,9 @@ private byte[] encodeRanges() throws IOException { DataOutputStream out = new DataOutputStream(buffer); long min = ranges.firstKey(); long max = ranges.lastEntry().getValue(); - encodeLong(out, ranges.size()); - encodeLong(out, min); - encodeLong(out, max - min); + out.writeInt(ranges.size()); + out.writeLong(min); + out.writeLong(max - min); // The envelope supplies the first start and last end. Encode only interior endpoints. DeltaRleWriter encoder = new DeltaRleWriter(out, min); int index = 0; @@ -322,7 +322,7 @@ private static byte[] encodeValues(Iterable values, int count) throws IOException { ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); - encodeLong(out, count); + out.writeInt(count); DeltaRleWriter encoder = new DeltaRleWriter(out, 0); for (Number value : values) { encoder.add(value.longValue()); @@ -474,14 +474,15 @@ public static Selection select( Payload rowPayload = payload(in, records); Payload bucketPayload = payload(in, records); require(partitionPayload == null || partitionPayload.count <= partitions); + require(rowPayload == null || rowPayload.data.remaining() >= 2 * Long.BYTES); long blockFirstRecord = firstRecord; nextOffset = offset + length; firstRecord += records; if (query != null && rowPayload != null) { - long min = readVarLong(rowPayload.data); - long span = readVarLong(rowPayload.data); - require(span <= Long.MAX_VALUE - min); + long min = rowPayload.data.getLong(); + long span = rowPayload.data.getLong(); + require(min >= 0 && span >= 0 && span <= Long.MAX_VALUE - min); long max = min + span; DeltaRleReader endpoints = new DeltaRleReader(rowPayload.data, 2L * (rowPayload.count - 1), min, max); @@ -572,11 +573,10 @@ private static Payload payload(ByteBuffer in, long records) throws IOException { if (encoding != 1) { return null; } - long count = readVarLong(result); - require(count > 0 && count <= records && count <= Integer.MAX_VALUE); - // At least an envelope (row IDs) or one delta run (partitions/buckets) must follow. - require(result.remaining() >= 2); - return new Payload((int) count, result); + require(result.remaining() >= Integer.BYTES + 2); + int count = result.getInt(); + require(count > 0 && count <= records); + return new Payload(count, result); } /** Writes equal consecutive deltas as (run length, delta), both unsigned varints. */ diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java index 5f9ec7b1afe9..5240b600babb 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java @@ -403,7 +403,7 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti } byte[] data = builder.serialize(header.length + 800, 8); List positions = positions(data); - int[] presentSizes = {8, 8, 8}; + int[] presentSizes = {11, 25, 11}; for (int mask = 0; mask < 8; mask++) { for (int dimension = 0; dimension < 3; dimension++) { int start = positions.get(mask)[dimension + 1]; @@ -484,8 +484,10 @@ void unpartitionedTablesStillRecordTheEmptyPartition() throws Exception { @Test void rowMissSkipsPartitionAndBucketDecoding() throws Exception { - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, varints(2, 1, 999, 1, 1)); - data = replacePayload(data, 0, 3, varints(2, 1, 0, 1, 0)); + byte[] data = + replacePayload( + fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 1, 999, 1, 1)); + data = replacePayload(data, 0, 3, compressedPayload(2, 1, 0, 1, 0)); BiPredicate buckets = mock(BiPredicate.class); assertThat( ManifestSidecar.select( @@ -497,7 +499,8 @@ data, goldenMeta(), query(15), part(7), type, buckets) @Test void partitionMissSkipsBucketDecodingWithOrWithoutRowQuery() throws Exception { - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, varints(2, 1, 0, 1, 0)); + byte[] data = + replacePayload(fixture("indexWithBuckets"), 0, 3, compressedPayload(2, 1, 0, 1, 0)); for (RowRangeIndex rows : Arrays.asList(null, query(0))) { BiPredicate buckets = mock(BiPredicate.class); assertThat( @@ -511,7 +514,9 @@ data, goldenMeta(), rows, part(99), type, buckets) @Test void absentPartitionFilterDoesNotDecodePartitionIds() throws Exception { - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, varints(2, 1, 999, 1, 1)); + byte[] data = + replacePayload( + fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 1, 999, 1, 1)); BiPredicate buckets = spy(bucketFilter(1)); assertThat( ManifestSidecar.select(data, goldenMeta(), query(20), null, type, buckets) @@ -527,7 +532,8 @@ void absentPartitionFilterDoesNotDecodePartitionIds() throws Exception { @Test void matchesSkipUnusedDeltaRuns() throws Exception { byte[] partitions = - replacePayload(fixture("indexWithBuckets"), 0, 1, varints(2, 1, 0, 1, 999)); + replacePayload( + fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 1, 0, 1, 999)); assertThat( ManifestSidecar.select(partitions, goldenMeta(), query(0), part(7), type) .blocks()) @@ -542,7 +548,7 @@ partitions, goldenMeta(), query(0), part(99), type)) fixture("indexWithBuckets"), 0, 3, - varints(2, 1, (1L << 32) | 4, 1, Long.MAX_VALUE)); + compressedPayload(2, 1, (1L << 32) | 4, 1, Long.MAX_VALUE)); assertThat( ManifestSidecar.select( buckets, @@ -576,7 +582,7 @@ partitions, goldenMeta(), query(0), part(99), type)) builder.serialize(header.length + 100, 3), 0, 2, - varints(3, 0, 49, 1, 9, 1, 11, 1, 9, 1, 99)); + rowPayload(3, 0, 49, 1, 9, 1, 11, 1, 9, 1, 99)); ManifestFileMeta meta = meta("m", header.length + 100, 3); assertThat(ManifestSidecar.select(rows, meta, query(0)).blocks()).hasSize(1); assertThat(ManifestSidecar.select(rows, meta, query(20)).blocks()).hasSize(1); @@ -589,19 +595,23 @@ partitions, goldenMeta(), query(0), part(99), type)) void malformedCompressedPayloadsFailWhenConsumed() throws Exception { List badRows = Arrays.asList( - varints(2, 0, 24, 0, 9), // Zero-length run. - varints(2, 0, 24, 3, 9), // More values than the interval count allows. - varints(2, 0, 24, 1, 9, 1), // Truncated delta. - varints(2, 0, 24, 1, 9, 1, 0), // Overlapping intervals. - varints(2, 0, 24, 2, Long.MAX_VALUE), // Run exceeds the envelope. - varints(1, Long.MAX_VALUE, 1), // Envelope overflows. - varints(1, 0, 24, 1, 0)); // Unexpected run for a single interval. + rowPayload(2, 0, 24, 0, 9), // Zero-length run. + rowPayload(2, 0, 24, 3, 9), // More values than the interval count allows. + rowPayload(2, 0, 24, 1, 9, 1), // Truncated delta. + rowPayload(2, 0, 24, 1, 9, 1, 0), // Overlapping intervals. + rowPayload(2, 0, 24, 2, Long.MAX_VALUE), // Run exceeds the envelope. + rowPayload(1, Long.MAX_VALUE, 1), // Envelope overflows. + rowPayload(1, -1, 24), // Negative minimum. + rowPayload(1, 0, -1), // Negative span. + Arrays.copyOf(rowPayload(1, 0, 24), 19), // Truncated fixed-width envelope. + rowPayload(1, 0, 24, 1, 0)); // Unexpected run for a single interval. for (byte[] payload : badRows) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 2, payload); assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(15))) .isInstanceOf(IOException.class); } - for (byte[] payload : Arrays.asList(varints(2, 1, 999, 1, 0), varints(2, 2, 0))) { + for (byte[] payload : + Arrays.asList(compressedPayload(2, 1, 999, 1, 0), compressedPayload(2, 2, 0))) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, payload); assertThatThrownBy( () -> @@ -611,7 +621,9 @@ data, goldenMeta(), query(0), part(99), type)) } for (byte[] payload : Arrays.asList( - varints(2, 1, 0, 1, 4), varints(2, 1, 1L << 31, 1, 4), varints(2, 2, 0))) { + compressedPayload(2, 1, 0, 1, 4), + compressedPayload(2, 1, 1L << 31, 1, 4), + compressedPayload(2, 2, 0))) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, payload); assertThatThrownBy( () -> @@ -627,26 +639,50 @@ data, goldenMeta(), query(0), part(99), type)) } @Test - void invalidVarintsFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception { - List bad = + void malformedRunVarintsFailWhenConsumed() throws Exception { + byte[] overlong = new byte[10]; + Arrays.fill(overlong, (byte) 0x80); + for (byte[] runs : Arrays.asList( new byte[] {(byte) 0x80}, - new byte[] {(byte) 0x81, 0, 1, 1}, // Noncanonical count. - new byte[] { - (byte) 0x80, - (byte) 0x80, - (byte) 0x80, - (byte) 0x80, - (byte) 0x80, - (byte) 0x80, - (byte) 0x80, - (byte) 0x80, - (byte) 0x80, - 1 - }, - varints(0, 1, 1), - varints(4, 1, 1), - varints(1)); + new byte[] {1, (byte) 0x80}, // Truncated delta. + new byte[] {(byte) 0x81, 0, 0}, // Noncanonical repeat count. + new byte[] {1, (byte) 0x80, 0}, // Noncanonical delta. + overlong)) { + for (int dimension = 1; dimension <= 3; dimension++) { + ByteArrayOutputStream payload = new ByteArrayOutputStream(); + payload.write(dimension == 2 ? rowPayload(2, 0, 24) : compressedPayload(2)); + payload.write(runs); + byte[] data = + replacePayload( + fixture("indexWithBuckets"), 0, dimension, payload.toByteArray()); + int dim = dimension; + assertThatThrownBy( + () -> + ManifestSidecar.select( + data, + goldenMeta(), + query(0), + dim == 1 ? part(99) : null, + type, + dim == 3 ? bucketFilter(99) : null)) + .isInstanceOf(IOException.class); + } + } + } + + @Test + void invalidCountFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception { + List bad = + Arrays.asList( + new byte[0], + new byte[3], // Incomplete int count. + compressedPayload(-1, 1, 1), + compressedPayload(0, 1, 1), + compressedPayload(4, 1, 1), + compressedPayload(Integer.MAX_VALUE, 1, 1), + compressedPayload(1), // Count without payload data. + Arrays.copyOf(compressedPayload(1, 1, 1), 5)); for (int dimension = 1; dimension <= 3; dimension++) { for (byte[] payload : bad) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, dimension, payload); @@ -660,6 +696,11 @@ void invalidVarintsFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) .isInstanceOf(IOException.class); } + byte[] missingEnvelope = + replacePayload( + fixture("indexWithBuckets"), 0, 2, Arrays.copyOf(rowPayload(1, 0, 24), 19)); + assertThatThrownBy(() -> ManifestSidecar.select(missingEnvelope, goldenMeta(), null)) + .isInstanceOf(IOException.class); byte[] data = fixture("indexWithBuckets"); int block = positions(data).get(0)[0]; ByteBuffer.wrap(data).putLong(block + 16, 2); @@ -709,7 +750,26 @@ private byte[] replacePayload(byte[] data, int block, int dimension, byte[] payl return checksum(buffer.toByteArray()); } - private static byte[] varints(long... values) { + private static byte[] compressedPayload(int count, long... values) throws IOException { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + DataOutputStream out = new DataOutputStream(buffer); + out.writeInt(count); + out.write(runBytes(values)); + return buffer.toByteArray(); + } + + private static byte[] rowPayload(int count, long min, long span, long... values) + throws IOException { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + DataOutputStream out = new DataOutputStream(buffer); + out.writeInt(count); + out.writeLong(min); + out.writeLong(span); + out.write(runBytes(values)); + return buffer.toByteArray(); + } + + private static byte[] runBytes(long... values) { ByteArrayOutputStream out = new ByteArrayOutputStream(); for (long value : values) { while ((value & ~0x7fL) != 0) { diff --git a/paimon-core/src/test/resources/manifest-sidecar.txt b/paimon-core/src/test/resources/manifest-sidecar.txt index 4562bd1916ef..1b9841ef115c 100644 --- a/paimon-core/src/test/resources/manifest-sidecar.txt +++ b/paimon-core/src/test/resources/manifest-sidecar.txt @@ -18,6 +18,6 @@ avroHeader=T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAA partitionA=AAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhA== partitionB=AAAAAgACAAAAAAAACQAAAAAAAAAAAAAAAAAAAA== -index=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAADkAAAAAAAAAZAAAAAAAAAADAAEAAAAHAgAYAQkBCwAAAAAAAAAAnQAAAAAAAADIAAAAAAAAAAIAAQAAABcC/v///w/n2OHhjPABAQQB49jh4YzwAQAAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIAAQAAABcCFOv/////////fwEEAef/////////fwDIf1W/eeutccPur0sI5iU31VsVOlXBzAZAV924R411vg== -indexWithPartitions=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAFAgEAAQEBAAAABwIAGAEJAQsAAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAUCAQABAQEAAAAXAv7///8P59jh4YzwAQEEAePY4eGM8AEAAAAAAAAAAWUAAAAAAAAAZAAAAAAAAAACAQAAAAUCAQABAQEAAAAXAhTr/////////38BBAHn/////////38ASyBR9j5LPqWbu2lXKJ/5rDuhDcH5F6h6kj6gIu5QKrs= -indexWithBuckets=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAFAgEAAQEBAAAABwIAGAEJAQsBAAAACQIBhICAgBABBAAAAAAAAACdAAAAAAAAAMgAAAAAAAAAAgEAAAAFAgEAAQEBAAAAFwL+////D+fY4eGM8AEBBAHj2OHhjPABAQAAAAkCAYSAgIAgAQQAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAABQIBAAEBAQAAABcCFOv/////////fwEEAef/////////fwEAAAAJAgEBAYOAgIAw4U9my1FPNEd5PdRGwwFFpTJWFrdan4tnQ0m2/YUbtYY= +index=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAADkAAAAAAAAAZAAAAAAAAAADAAEAAAAYAAAAAgAAAAAAAAAAAAAAAAAAABgBCQELAAAAAAAAAACdAAAAAAAAAMgAAAAAAAAAAgABAAAAHgAAAAIAAAAA/////gAAB4DMOGxnAQQB49jh4YzwAQAAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIAAQAAACAAAAACAAAAAAAAABR/////////6wEEAef/////////fwCKlRL8zDF5k5NnoTZbwWd3W4/9TLQU5ICorRug+Agf2Q== +indexWithPartitions=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAIAAAAAgEAAQEBAAAAGAAAAAIAAAAAAAAAAAAAAAAAAAAYAQkBCwAAAAAAAAAAnQAAAAAAAADIAAAAAAAAAAIBAAAACAAAAAIBAAEBAQAAAB4AAAACAAAAAP////4AAAeAzDhsZwEEAePY4eGM8AEAAAAAAAAAAWUAAAAAAAAAZAAAAAAAAAACAQAAAAgAAAACAQABAQEAAAAgAAAAAgAAAAAAAAAUf////////+sBBAHn/////////38AKbEFIt3sTGH4L0/6GN34pXMhi0acHt0pyDLHlKI0ias= +indexWithBuckets=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAIAAAAAgEAAQEBAAAAGAAAAAIAAAAAAAAAAAAAAAAAAAAYAQkBCwEAAAAMAAAAAgGEgICAEAEEAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAgAAAACAQABAQEAAAAeAAAAAgAAAAD////+AAAHgMw4bGcBBAHj2OHhjPABAQAAAAwAAAACAYSAgIAgAQQAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAACAAAAAIBAAEBAQAAACAAAAACAAAAAAAAABR/////////6wEEAef/////////fwEAAAAMAAAAAgEBAYOAgIAwO1kfZjtY7swim4i4X6yEF9xO6jMe9WyV+Sq2sbpwRPI= From 3080860e8acedfb5a6fa90982e2e7620dc57cbcf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 21:53:21 +0800 Subject: [PATCH 06/14] [core] Share delta-varint codec and remove sidecar RLE --- docs/docs/concepts/spec/manifest.md | 64 ++++---- .../apache/paimon/utils/DeltaVarintCodec.java | 89 +++++++++++ .../paimon/utils/VarLengthIntUtils.java | 20 +++ .../paimon/utils/DeltaVarintCodecTest.java | 147 ++++++++++++++++++ .../paimon/manifest/ManifestSidecar.java | 123 +++------------ .../manifest/ManifestBlockIndexTest.java | 72 ++++----- .../src/test/resources/manifest-sidecar.txt | 6 +- 7 files changed, 342 insertions(+), 179 deletions(-) create mode 100644 paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java create mode 100644 paimon-common/src/test/java/org/apache/paimon/utils/DeltaVarintCodecTest.java diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 1985e591901e..17db4bcc5a87 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -90,7 +90,7 @@ supplied bytes directly and reports invalid containers with `IOException`. Version 1 uses the following layout. Container `int` and `long` fields are signed, fixed-width 4-byte and 8-byte big-endian integers. Encoding IDs are unsigned bytes with separate namespaces. -Payload counts and envelopes use the same fixed-width types; delta/RLE runs use the +Payload counts and envelopes use the same fixed-width types; delta streams use the variable-length encoding described below. ```text @@ -132,9 +132,9 @@ Partition predicates are evaluated once per dictionary entry. | Dimension | Encoding | Payload | | --- | --- | --- | | Any | `0` | Unavailable; only the encoding byte is present. | -| Partition | `1` | Count and delta/RLE-compressed sorted unique dictionary IDs. | -| Row ID | `1` | Interval count, minimum, span, and delta/RLE-compressed interior endpoints. | -| Bucket | `1` | Count and delta/RLE-compressed sorted unique packed bucket/count pairs. | +| Partition | `1` | Count and delta/varint-compressed sorted unique dictionary IDs. | +| Row ID | `1` | Interval count, minimum, span, and delta/varint-compressed interior endpoints. | +| Bucket | `1` | Count and delta/varint-compressed sorted unique packed bucket/count pairs. | | Any | Other nonzero ID | Skip the declared payload length; treat only this dimension as unavailable. | Only nonzero encodings are followed by a length and payload. Payload lengths exclude the @@ -142,29 +142,29 @@ encoding and length fields, but include the count and other fields within the pa All three encoding-1 payloads have positive counts no greater than the block's record count. Encoding 0 represents unavailable coverage, rather than encoding 1 with a zero count. -#### Delta and RLE Encoding +#### Delta Encoding Each payload starts with a fixed-width count (`int`); row-ID payloads also have fixed-width -`min` and `span` fields (`long`). Only integers in the following delta/RLE stream use +`min` and `span` fields (`long`). Only integers in the following delta stream use nonnegative unsigned LEB128 varints, occupying one to nine bytes for values from 0 through `Long.MAX_VALUE`. Seven value bits are stored per byte, least significant group first; the high bit indicates another byte follows. Encodings use the shortest representation. There is no ZigZag transformation or padding. -A sorted sequence is delta-encoded from a specified base. Consecutive equal deltas are -stored as runs: +A sorted sequence is delta-encoded from a specified base. Each value contributes one +unsigned varint containing its difference from the preceding value. The first difference +is relative to the base: ```text -runs[] - repeatCount : varint // positive number of values produced - delta : varint // add delta for each value in the run +deltas[] : varint +value[0] = base + deltas[0] +value[i] = value[i - 1] + deltas[i] ``` -Starting with `previous = base`, a run produces `repeatCount` successive values by adding -`delta` each time. Run counts must sum to the dimension's expected value count. Decoders -consume values lazily, check overflow and the applicable value bounds, and require the -payload to end when all expected values have been consumed. They do not allocate expanded -arrays for runs. +The shared `DeltaVarintCodec` utility writes each delta immediately and reads values on +demand, using `VarLengthIntUtils` for varints. Counts and bounds are supplied by the caller. +The reader checks overflow and value bounds and requires the buffer to end after all +expected values have been consumed. It can stop early without materializing the sequence. #### Partition Payload @@ -174,15 +174,14 @@ represented by its entries: ```text partitionPayload partitionIdCount : int // N > 0 - runs[] // N dictionary IDs, base = 0 + deltas[] // N dictionary IDs, base = 0 ``` An ID is the zero-based position of a complete tuple in the sidecar's shared dictionary. IDs satisfy `0 <= id < partitionCount` and are strictly increasing. Tuple bytes appear only in the dictionary and are not repeated in each block. For IDs `[0, 1, 2, 3, 4]`, the deltas -are `[0, 1, 1, 1, 1]` and the runs are `(1, 0), (4, 1)`. The payload contains a four-byte -count of 5 followed by the run bytes `[1, 0, 4, 1]`: 8 bytes, or 13 bytes including the -encoding and length fields. +are `[0, 1, 1, 1, 1]`. The payload contains a four-byte count of 5 followed by these five +varint bytes: 9 bytes, or 14 bytes including the encoding and length fields. With a partition filter, a block matches if any referenced tuple matches. A tuple containing a null partition value still has a dictionary ID. Unpartitioned tables record the empty @@ -200,23 +199,23 @@ rowIdPayload rangeCount : int // N > 0 min : long // first interval's start span : long // last interval's end minus min - runs[] // 2 * (N - 1) interior endpoints, base = min + deltas[] // 2 * (N - 1) interior endpoints, base = min ``` The maximum is `min + span`, which must not exceed `Long.MAX_VALUE`. Flatten the intervals as `[start0, end0, start1, end1, ...]`. The first start is supplied by `min`, and the last -end by `min + span`; only the remaining `2 * (N - 1)` interior endpoints are delta/RLE encoded. +end by `min + span`; only the remaining `2 * (N - 1)` interior endpoints are delta/varint encoded. Pairing the reconstructed endpoints recovers the intervals. Each pair satisfies `0 <= start <= end <= Long.MAX_VALUE`; each following start must exceed the preceding end. For `[(10, 19), (30, 39)]`, the count is 2, minimum is 10, and span is 29. The interior -endpoints `[19, 30]` have deltas `[9, 11]` from base 10, encoded as `(1, 9), (1, 11)`. +endpoints `[19, 30]` have deltas `[9, 11]` from base 10, each encoded as one varint byte. The payload starts with a four-byte count of 2, an eight-byte minimum of 10, and an eight-byte -span of 29, followed by the run bytes `[1, 9, 1, 11]`: 24 bytes, or 29 bytes with framing. +span of 29, followed by the delta bytes `[9, 11]`: 22 bytes, or 27 bytes with framing. For a single interval, the 20-byte fixed-width prefix completely defines the interval and -no runs follow. +no deltas follow. -The reader first tests the envelope without expanding any runs. A query for row ID 25 +The reader first tests the envelope without decoding any deltas. A query for row ID 25 passes the example's envelope check but matches neither interval. Unknown or invalid row-ID metadata makes that block's row-ID payload unavailable; partition and bucket coverage remain usable. @@ -227,7 +226,7 @@ When `bucketEncoding == 1`, the block stores distinct bucket/count pairs: ```text bucketPayload pairCount : int // N > 0 - runs[] // N packed pairs, base = 0 + deltas[] // N packed pairs, base = 0 packedPair = ((long) bucket << 32) | totalBuckets ``` @@ -239,9 +238,8 @@ The decoder recovers `bucket = (int) (packedPair >>> 32)` and `totalBuckets = (i The same bucket may occur with different totals after rescaling. For `[(1, 4), (1, 8), (3, 4)]`, the packed values are `[4294967300, 4294967304, 12884901892]` -and deltas are `[4294967300, 4, 8589934588]`. The payload contains count 3 and three runs -of length 1, occupying 18 bytes, or 23 bytes with framing. Repeated bucket strides with the -same total bucket count form a single run. +and deltas are `[4294967300, 4, 8589934588]`. The payload contains a four-byte count of 3 +and three varints occupying 5, 1 and 5 bytes: 15 bytes total, or 20 bytes with framing. Missing, invalid or negative/synthetic bucket metadata makes the block's bucket coverage unavailable. A caller can supply a predicate on `(bucket, totalBuckets)` which conservatively @@ -256,9 +254,9 @@ the whole original manifest after its header; record counts must sum to the mani count. Unknown nonzero encodings skip their declared bytes without interpreting a count. Compressed contents are decoded only for dimensions needed by the filters and only until -that dimension matches. A row-ID envelope rejection skips all its runs; a matching interval, -partition ID or bucket pair skips remaining values. Invalid varints, run counts, overflows, -out-of-range values or ordering encountered while decoding invalidate the container. Run +that dimension matches. A row-ID envelope rejection skips all its deltas; a matching interval, +partition ID or bucket pair skips remaining values. Invalid varints, value counts, overflows, +out-of-range values or ordering encountered while decoding invalidate the container. Delta contents skipped by short-circuiting are not individually validated. For conjunctive filters a block is retained only if every dimension is unavailable or matches. diff --git a/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java b/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java new file mode 100644 index 000000000000..bae8e557443a --- /dev/null +++ b/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.utils; + +import java.io.DataOutput; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Objects; + +/** Streaming unsigned delta/varint encoding for nondecreasing, nonnegative long values. */ +public final class DeltaVarintCodec { + private DeltaVarintCodec() {} + + /** Writes each delta immediately. The caller owns the output and stores the value count. */ + public static final class Writer { + private final DataOutput out; + private long previous; + + public Writer(DataOutput out, long base) { + if (base < 0) { + throw new IllegalArgumentException("Delta base must be nonnegative"); + } + this.out = Objects.requireNonNull(out); + previous = base; + } + + public void write(long value) throws IOException { + require(value >= previous); + VarLengthIntUtils.encodeLong(out, value - previous); + previous = value; + } + } + + /** + * Reads values on demand and advances the supplied buffer. The buffer must contain exactly the + * encoded sequence; a complete read validates its boundary. Reading may stop early. + */ + public static final class Reader { + private final ByteBuffer data; + private final long max; + private long remaining; + private long value; + + public Reader(ByteBuffer data, long count, long base, long max) throws IOException { + this.data = Objects.requireNonNull(data); + require(count >= 0 && count <= data.remaining() && base >= 0 && max >= base); + require(count != 0 || !data.hasRemaining()); + remaining = count; + value = base; + this.max = max; + } + + public boolean hasNext() { + return remaining > 0; + } + + public long next() throws IOException { + require(remaining > 0); + long delta = VarLengthIntUtils.decodeLong(data); + require(delta <= max - value); + value += delta; + remaining--; + require(remaining != 0 || !data.hasRemaining()); + return value; + } + } + + private static void require(boolean valid) throws IOException { + if (!valid) { + throw new IOException("Invalid delta/varint sequence"); + } + } +} diff --git a/paimon-common/src/main/java/org/apache/paimon/utils/VarLengthIntUtils.java b/paimon-common/src/main/java/org/apache/paimon/utils/VarLengthIntUtils.java index a05eb8a99484..e3874a868440 100644 --- a/paimon-common/src/main/java/org/apache/paimon/utils/VarLengthIntUtils.java +++ b/paimon-common/src/main/java/org/apache/paimon/utils/VarLengthIntUtils.java @@ -24,6 +24,7 @@ import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; +import java.nio.ByteBuffer; /* This file is based on source code of LongPacker from the PalDB Project (https://github.com/linkedin/PalDB), licensed by the Apache * Software Foundation (ASF) under the Apache License, Version 2.0. See the NOTICE file distributed with this work for @@ -82,6 +83,25 @@ public static long decodeLong(DataInput is) throws IOException { throw new Error("Malformed long."); } + /** Decodes a canonical nonnegative long from the buffer and advances its position. */ + public static long decodeLong(ByteBuffer in) throws IOException { + long value = 0; + for (int shift = 0; shift < 63; shift += 7) { + if (!in.hasRemaining()) { + throw new EOFException("Truncated variable-length long"); + } + int b = Byte.toUnsignedInt(in.get()); + value |= (long) (b & 0x7f) << shift; + if ((b & 0x80) == 0) { + if (shift != 0 && (b & 0x7f) == 0) { + throw new IOException("Noncanonical variable-length long"); + } + return value; + } + } + throw new IOException("Invalid variable-length long"); + } + public static long decodeLong(byte[] ba, int index) { long result = 0; for (int offset = 0; offset < 64; offset += 7) { diff --git a/paimon-common/src/test/java/org/apache/paimon/utils/DeltaVarintCodecTest.java b/paimon-common/src/test/java/org/apache/paimon/utils/DeltaVarintCodecTest.java new file mode 100644 index 000000000000..57261c44f3b8 --- /dev/null +++ b/paimon-common/src/test/java/org/apache/paimon/utils/DeltaVarintCodecTest.java @@ -0,0 +1,147 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.utils; + +import org.apache.paimon.io.DataOutputSerializer; + +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.Random; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Tests for the shared streaming unsigned delta/varint codec. */ +class DeltaVarintCodecTest { + @Test + void fixedBytesAndSlicedBuffer() throws Exception { + long[] values = {10, 12, 14, 16, 16, 1024}; + byte[] encoded = encode(10, values); + assertThat(encoded).containsExactly(new byte[] {0, 2, 2, 2, 0, (byte) 0xf0, 7}); + ByteBuffer data = ByteBuffer.allocate(encoded.length + 4); + data.position(2); + data.put(encoded); + data.limit(data.position()).position(2); + DeltaVarintCodec.Reader reader = new DeltaVarintCodec.Reader(data, values.length, 10, 1024); + for (long value : values) { + assertThat(reader.hasNext()).isTrue(); + assertThat(reader.next()).isEqualTo(value); + } + assertThat(reader.hasNext()).isFalse(); + assertThat(data.hasRemaining()).isFalse(); + assertThatThrownBy(reader::next).isInstanceOf(IOException.class); + } + + @Test + void longBoundariesAndEmptySequence() throws Exception { + byte[] encoded = encode(0, Long.MAX_VALUE); + assertThat(encoded).hasSize(9); + assertThat( + new DeltaVarintCodec.Reader(ByteBuffer.wrap(encoded), 1, 0, Long.MAX_VALUE) + .next()) + .isEqualTo(Long.MAX_VALUE); + assertThat(encode(Long.MAX_VALUE - 1, Long.MAX_VALUE, Long.MAX_VALUE)) + .containsExactly(new byte[] {1, 0}); + DeltaVarintCodec.Reader empty = + new DeltaVarintCodec.Reader(ByteBuffer.allocate(0), 0, 0, 0); + assertThat(empty.hasNext()).isFalse(); + assertThatThrownBy(empty::next).isInstanceOf(IOException.class); + assertThat(encode(0)).isEmpty(); + } + + @Test + void stopsBeforeUnusedMalformedData() throws Exception { + DeltaVarintCodec.Reader reader = + new DeltaVarintCodec.Reader( + ByteBuffer.wrap(new byte[] {0, (byte) 0x80}), 2, 10, 100); + assertThat(reader.next()).isEqualTo(10); + assertThat(reader.hasNext()).isTrue(); + assertThatThrownBy(reader::next).isInstanceOf(IOException.class); + } + + @Test + void rejectsInvalidCountsBoundsAndOrdering() throws Exception { + assertThatThrownBy(() -> new DeltaVarintCodec.Reader(ByteBuffer.allocate(0), -1, 0, 1)) + .isInstanceOf(IOException.class); + assertThatThrownBy( + () -> new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {0}), 2, 0, 1)) + .isInstanceOf(IOException.class); + assertThatThrownBy( + () -> new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {0}), 0, 0, 1)) + .isInstanceOf(IOException.class); + assertThatThrownBy(() -> new DeltaVarintCodec.Reader(ByteBuffer.allocate(0), 0, 2, 1)) + .isInstanceOf(IOException.class); + assertThatThrownBy(() -> new DeltaVarintCodec.Writer(new DataOutputSerializer(8), -1)) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> encode(10, 9)).isInstanceOf(IOException.class); + assertThatThrownBy(() -> encode(0, 2, 1)).isInstanceOf(IOException.class); + DeltaVarintCodec.Reader trailing = + new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {1, 2}), 1, 0, 10); + assertThatThrownBy(trailing::next).isInstanceOf(IOException.class); + DeltaVarintCodec.Reader overflow = + new DeltaVarintCodec.Reader( + ByteBuffer.wrap(encode(0, Long.MAX_VALUE)), 1, 1, Long.MAX_VALUE); + assertThatThrownBy(overflow::next).isInstanceOf(IOException.class); + } + + @Test + void rejectsMalformedVarints() throws Exception { + byte[] overlong = new byte[10]; + Arrays.fill(overlong, (byte) 0x80); + for (byte[] bytes : + Arrays.asList(new byte[] {(byte) 0x80}, new byte[] {(byte) 0x81, 0}, overlong)) { + DeltaVarintCodec.Reader reader = + new DeltaVarintCodec.Reader(ByteBuffer.wrap(bytes), 1, 0, Long.MAX_VALUE); + assertThatThrownBy(reader::next).isInstanceOf(IOException.class); + } + } + + @Test + void randomizedRoundTrips() throws Exception { + Random random = new Random(9845); + for (int trial = 0; trial < 1000; trial++) { + long base = random.nextInt(10000); + long[] values = new long[random.nextInt(128)]; + long value = base; + for (int i = 0; i < values.length; i++) { + value += random.nextInt(10000); + values[i] = value; + } + DeltaVarintCodec.Reader reader = + new DeltaVarintCodec.Reader( + ByteBuffer.wrap(encode(base, values)), values.length, base, value); + for (long expected : values) { + assertThat(reader.next()).isEqualTo(expected); + } + assertThat(reader.hasNext()).isFalse(); + } + } + + private static byte[] encode(long base, long... values) throws IOException { + DataOutputSerializer out = new DataOutputSerializer(32); + DeltaVarintCodec.Writer writer = new DeltaVarintCodec.Writer(out, base); + for (long value : values) { + writer.write(value); + } + return out.getCopyOfBuffer(); + } +} diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index 44abefb68b82..42fb0d45523c 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -27,6 +27,7 @@ import org.apache.paimon.partition.PartitionPredicate; import org.apache.paimon.types.DataField; import org.apache.paimon.types.RowType; +import org.apache.paimon.utils.DeltaVarintCodec; import org.apache.paimon.utils.RowRangeIndex; import org.apache.paimon.utils.SegmentsCache; import org.apache.paimon.utils.SerializationUtils; @@ -57,8 +58,6 @@ import java.util.TreeSet; import java.util.function.BiPredicate; -import static org.apache.paimon.utils.VarLengthIntUtils.encodeLong; - /** Independently usable partition, row-id and bucket coverage for each manifest block. */ public final class ManifestSidecar { public static final String SUFFIX = ".avro.sidecar"; @@ -276,17 +275,16 @@ private byte[] encodeRanges() throws IOException { out.writeLong(min); out.writeLong(max - min); // The envelope supplies the first start and last end. Encode only interior endpoints. - DeltaRleWriter encoder = new DeltaRleWriter(out, min); + DeltaVarintCodec.Writer encoder = new DeltaVarintCodec.Writer(out, min); int index = 0; for (Map.Entry range : ranges.entrySet()) { if (index > 0) { - encoder.add(range.getKey()); + encoder.write(range.getKey()); } if (++index < ranges.size()) { - encoder.add(range.getValue()); + encoder.write(range.getValue()); } } - encoder.finish(); return buffer.toByteArray(); } @@ -323,11 +321,10 @@ private static byte[] encodeValues(Iterable values, int count) ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); out.writeInt(count); - DeltaRleWriter encoder = new DeltaRleWriter(out, 0); + DeltaVarintCodec.Writer encoder = new DeltaVarintCodec.Writer(out, 0); for (Number value : values) { - encoder.add(value.longValue()); + encoder.write(value.longValue()); } - encoder.finish(); return buffer.toByteArray(); } @@ -474,7 +471,10 @@ public static Selection select( Payload rowPayload = payload(in, records); Payload bucketPayload = payload(in, records); require(partitionPayload == null || partitionPayload.count <= partitions); - require(rowPayload == null || rowPayload.data.remaining() >= 2 * Long.BYTES); + require( + rowPayload == null + || rowPayload.data.remaining() + >= 2 * Long.BYTES + 2L * (rowPayload.count - 1)); long blockFirstRecord = firstRecord; nextOffset = offset + length; firstRecord += records; @@ -484,8 +484,9 @@ public static Selection select( long span = rowPayload.data.getLong(); require(min >= 0 && span >= 0 && span <= Long.MAX_VALUE - min); long max = min + span; - DeltaRleReader endpoints = - new DeltaRleReader(rowPayload.data, 2L * (rowPayload.count - 1), min, max); + DeltaVarintCodec.Reader endpoints = + new DeltaVarintCodec.Reader( + rowPayload.data, 2L * (rowPayload.count - 1), min, max); if (!query.intersects(min, max)) { continue; } @@ -506,8 +507,8 @@ public static Selection select( } if (partitionFilter != null && partitionPayload != null) { - DeltaRleReader ids = - new DeltaRleReader( + DeltaVarintCodec.Reader ids = + new DeltaVarintCodec.Reader( partitionPayload.data, partitionPayload.count, 0, partitions - 1L); boolean partitionHit = false; long previous = -1; @@ -523,8 +524,8 @@ public static Selection select( } if (bucketFilter != null && bucketPayload != null) { - DeltaRleReader pairs = - new DeltaRleReader( + DeltaVarintCodec.Reader pairs = + new DeltaVarintCodec.Reader( bucketPayload.data, bucketPayload.count, 0, Long.MAX_VALUE); boolean bucketHit = false; long previous = -1; @@ -573,98 +574,12 @@ private static Payload payload(ByteBuffer in, long records) throws IOException { if (encoding != 1) { return null; } - require(result.remaining() >= Integer.BYTES + 2); + require(result.remaining() >= Integer.BYTES); int count = result.getInt(); - require(count > 0 && count <= records); + require(count > 0 && count <= records && count <= result.remaining()); return new Payload(count, result); } - /** Writes equal consecutive deltas as (run length, delta), both unsigned varints. */ - private static final class DeltaRleWriter { - private final DataOutputStream out; - private long previous; - private long delta; - private long repeat; - - private DeltaRleWriter(DataOutputStream out, long base) { - this.out = out; - previous = base; - } - - private void add(long value) throws IOException { - require(value >= previous); - long nextDelta = value - previous; - if (repeat != 0 && nextDelta != delta) { - finish(); - } - delta = nextDelta; - repeat++; - previous = value; - } - - private void finish() throws IOException { - if (repeat > 0) { - encodeLong(out, repeat); - encodeLong(out, delta); - repeat = 0; - } - } - } - - /** Decodes only requested values; a complete read also checks the payload boundary. */ - private static final class DeltaRleReader { - private final ByteBuffer data; - private final long max; - private long remaining; - private long value; - private long repeat; - private long delta; - - private DeltaRleReader(ByteBuffer data, long count, long base, long max) - throws IOException { - require(base >= 0 && max >= base); - this.data = data; - remaining = count; - value = base; - this.max = max; - require(count != 0 || !data.hasRemaining()); - } - - private boolean hasNext() { - return remaining > 0; - } - - private long next() throws IOException { - require(remaining > 0); - if (repeat == 0) { - repeat = readVarLong(data); - delta = readVarLong(data); - require(repeat > 0 && repeat <= remaining); - require(delta == 0 || repeat <= (max - value) / delta); - } - value += delta; - repeat--; - remaining--; - require(remaining != 0 || (repeat == 0 && !data.hasRemaining())); - return value; - } - } - - /** Nonnegative long encoded in one to nine canonical unsigned LEB128 bytes. */ - private static long readVarLong(ByteBuffer in) throws IOException { - long value = 0; - for (int shift = 0; shift < 63; shift += 7) { - require(in.hasRemaining()); - int b = Byte.toUnsignedInt(in.get()); - value |= (long) (b & 0x7f) << shift; - if ((b & 0x80) == 0) { - require(shift == 0 || (b & 0x7f) != 0); - return value; - } - } - throw new IOException("Invalid manifest sidecar varint"); - } - /** Reads the complete sidecar. Null means read the original manifest. */ @Nullable public static Selection read( diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java index 5240b600babb..1e4aad33212d 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java @@ -403,7 +403,7 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti } byte[] data = builder.serialize(header.length + 800, 8); List positions = positions(data); - int[] presentSizes = {11, 25, 11}; + int[] presentSizes = {10, 25, 10}; for (int mask = 0; mask < 8; mask++) { for (int dimension = 0; dimension < 3; dimension++) { int start = positions.get(mask)[dimension + 1]; @@ -433,7 +433,7 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti } @Test - void deltaRleCompressesSortedPayloadsWithoutCoarseningRowIds() throws Exception { + void deltaVarintsCompressSortedPayloadsWithoutCoarseningRowIds() throws Exception { byte[] header = fixture("avroHeader"); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); int count = 10000; @@ -444,9 +444,9 @@ void deltaRleCompressesSortedPayloadsWithoutCoarseningRowIds() throws Exception builder.endBlock(); byte[] data = builder.serialize(header.length + 100, count); int[] block = positions(data).get(0); - for (int dimension = 1; dimension <= 3; dimension++) { - assertThat(ByteBuffer.wrap(data).getInt(block[dimension] + 1)).isLessThan(32); - } + assertThat(ByteBuffer.wrap(data).getInt(block[1] + 1)).isEqualTo(4 + count); + assertThat(ByteBuffer.wrap(data).getInt(block[2] + 1)).isEqualTo(20 + 2 * (count - 1)); + assertThat(ByteBuffer.wrap(data).getInt(block[3] + 1)).isEqualTo(4 + 2 + 5 * (count - 1)); ManifestFileMeta meta = meta("m", header.length + 100, count); assertThat(ManifestSidecar.select(data, meta, query(3)).blocks()).isEmpty(); assertThat( @@ -485,9 +485,8 @@ void unpartitionedTablesStillRecordTheEmptyPartition() throws Exception { @Test void rowMissSkipsPartitionAndBucketDecoding() throws Exception { byte[] data = - replacePayload( - fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 1, 999, 1, 1)); - data = replacePayload(data, 0, 3, compressedPayload(2, 1, 0, 1, 0)); + replacePayload(fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 999, 1)); + data = replacePayload(data, 0, 3, compressedPayload(2, 0, 0)); BiPredicate buckets = mock(BiPredicate.class); assertThat( ManifestSidecar.select( @@ -499,8 +498,7 @@ data, goldenMeta(), query(15), part(7), type, buckets) @Test void partitionMissSkipsBucketDecodingWithOrWithoutRowQuery() throws Exception { - byte[] data = - replacePayload(fixture("indexWithBuckets"), 0, 3, compressedPayload(2, 1, 0, 1, 0)); + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, compressedPayload(2, 0, 0)); for (RowRangeIndex rows : Arrays.asList(null, query(0))) { BiPredicate buckets = mock(BiPredicate.class); assertThat( @@ -515,8 +513,7 @@ data, goldenMeta(), rows, part(99), type, buckets) @Test void absentPartitionFilterDoesNotDecodePartitionIds() throws Exception { byte[] data = - replacePayload( - fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 1, 999, 1, 1)); + replacePayload(fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 999, 1)); BiPredicate buckets = spy(bucketFilter(1)); assertThat( ManifestSidecar.select(data, goldenMeta(), query(20), null, type, buckets) @@ -530,10 +527,9 @@ void absentPartitionFilterDoesNotDecodePartitionIds() throws Exception { } @Test - void matchesSkipUnusedDeltaRuns() throws Exception { + void matchesSkipUnusedDeltas() throws Exception { byte[] partitions = - replacePayload( - fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 1, 0, 1, 999)); + replacePayload(fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 0, 999)); assertThat( ManifestSidecar.select(partitions, goldenMeta(), query(0), part(7), type) .blocks()) @@ -548,7 +544,7 @@ partitions, goldenMeta(), query(0), part(99), type)) fixture("indexWithBuckets"), 0, 3, - compressedPayload(2, 1, (1L << 32) | 4, 1, Long.MAX_VALUE)); + compressedPayload(2, (1L << 32) | 4, Long.MAX_VALUE)); assertThat( ManifestSidecar.select( buckets, @@ -582,7 +578,7 @@ partitions, goldenMeta(), query(0), part(99), type)) builder.serialize(header.length + 100, 3), 0, 2, - rowPayload(3, 0, 49, 1, 9, 1, 11, 1, 9, 1, 99)); + rowPayload(3, 0, 49, 9, 11, 9, 99)); ManifestFileMeta meta = meta("m", header.length + 100, 3); assertThat(ManifestSidecar.select(rows, meta, query(0)).blocks()).hasSize(1); assertThat(ManifestSidecar.select(rows, meta, query(20)).blocks()).hasSize(1); @@ -595,23 +591,22 @@ partitions, goldenMeta(), query(0), part(99), type)) void malformedCompressedPayloadsFailWhenConsumed() throws Exception { List badRows = Arrays.asList( - rowPayload(2, 0, 24, 0, 9), // Zero-length run. - rowPayload(2, 0, 24, 3, 9), // More values than the interval count allows. - rowPayload(2, 0, 24, 1, 9, 1), // Truncated delta. - rowPayload(2, 0, 24, 1, 9, 1, 0), // Overlapping intervals. - rowPayload(2, 0, 24, 2, Long.MAX_VALUE), // Run exceeds the envelope. + rowPayload(2, 0, 24, 9), // Missing an endpoint. + rowPayload(2, 0, 24, 9, 0), // Overlapping intervals. + rowPayload(2, 0, 24, Long.MAX_VALUE, 0), // Exceeds the envelope. + rowPayload(2, 0, 24, 9, 11, 0), // More values than declared. rowPayload(1, Long.MAX_VALUE, 1), // Envelope overflows. - rowPayload(1, -1, 24), // Negative minimum. - rowPayload(1, 0, -1), // Negative span. + rowPayload(1, -1, 24), + rowPayload(1, 0, -1), Arrays.copyOf(rowPayload(1, 0, 24), 19), // Truncated fixed-width envelope. - rowPayload(1, 0, 24, 1, 0)); // Unexpected run for a single interval. + rowPayload(1, 0, 24, 0)); // Unexpected value for a single interval. for (byte[] payload : badRows) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 2, payload); assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(15))) .isInstanceOf(IOException.class); } for (byte[] payload : - Arrays.asList(compressedPayload(2, 1, 999, 1, 0), compressedPayload(2, 2, 0))) { + Arrays.asList(compressedPayload(2, 999, 0), compressedPayload(2, 0, 0))) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, payload); assertThatThrownBy( () -> @@ -621,9 +616,9 @@ data, goldenMeta(), query(0), part(99), type)) } for (byte[] payload : Arrays.asList( - compressedPayload(2, 1, 0, 1, 4), - compressedPayload(2, 1, 1L << 31, 1, 4), - compressedPayload(2, 2, 0))) { + compressedPayload(2, 0, 4), + compressedPayload(2, 1L << 31, 4), + compressedPayload(2, 0, 0))) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, payload); assertThatThrownBy( () -> @@ -639,20 +634,19 @@ data, goldenMeta(), query(0), part(99), type)) } @Test - void malformedRunVarintsFailWhenConsumed() throws Exception { + void malformedDeltaVarintsFailWhenConsumed() throws Exception { byte[] overlong = new byte[10]; Arrays.fill(overlong, (byte) 0x80); - for (byte[] runs : + for (byte[] deltas : Arrays.asList( new byte[] {(byte) 0x80}, - new byte[] {1, (byte) 0x80}, // Truncated delta. - new byte[] {(byte) 0x81, 0, 0}, // Noncanonical repeat count. - new byte[] {1, (byte) 0x80, 0}, // Noncanonical delta. + new byte[] {(byte) 0x80, (byte) 0x80}, // Truncated delta. + new byte[] {(byte) 0x81, 0, 0}, // Noncanonical delta. overlong)) { for (int dimension = 1; dimension <= 3; dimension++) { ByteArrayOutputStream payload = new ByteArrayOutputStream(); payload.write(dimension == 2 ? rowPayload(2, 0, 24) : compressedPayload(2)); - payload.write(runs); + payload.write(deltas); byte[] data = replacePayload( fixture("indexWithBuckets"), 0, dimension, payload.toByteArray()); @@ -682,7 +676,7 @@ void invalidCountFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception { compressedPayload(4, 1, 1), compressedPayload(Integer.MAX_VALUE, 1, 1), compressedPayload(1), // Count without payload data. - Arrays.copyOf(compressedPayload(1, 1, 1), 5)); + compressedPayload(2, 1)); for (int dimension = 1; dimension <= 3; dimension++) { for (byte[] payload : bad) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, dimension, payload); @@ -754,7 +748,7 @@ private static byte[] compressedPayload(int count, long... values) throws IOExce ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); out.writeInt(count); - out.write(runBytes(values)); + out.write(deltaBytes(values)); return buffer.toByteArray(); } @@ -765,11 +759,11 @@ private static byte[] rowPayload(int count, long min, long span, long... values) out.writeInt(count); out.writeLong(min); out.writeLong(span); - out.write(runBytes(values)); + out.write(deltaBytes(values)); return buffer.toByteArray(); } - private static byte[] runBytes(long... values) { + private static byte[] deltaBytes(long... values) { ByteArrayOutputStream out = new ByteArrayOutputStream(); for (long value : values) { while ((value & ~0x7fL) != 0) { diff --git a/paimon-core/src/test/resources/manifest-sidecar.txt b/paimon-core/src/test/resources/manifest-sidecar.txt index 1b9841ef115c..c8adbfcef042 100644 --- a/paimon-core/src/test/resources/manifest-sidecar.txt +++ b/paimon-core/src/test/resources/manifest-sidecar.txt @@ -18,6 +18,6 @@ avroHeader=T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAA partitionA=AAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhA== partitionB=AAAAAgACAAAAAAAACQAAAAAAAAAAAAAAAAAAAA== -index=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAADkAAAAAAAAAZAAAAAAAAAADAAEAAAAYAAAAAgAAAAAAAAAAAAAAAAAAABgBCQELAAAAAAAAAACdAAAAAAAAAMgAAAAAAAAAAgABAAAAHgAAAAIAAAAA/////gAAB4DMOGxnAQQB49jh4YzwAQAAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIAAQAAACAAAAACAAAAAAAAABR/////////6wEEAef/////////fwCKlRL8zDF5k5NnoTZbwWd3W4/9TLQU5ICorRug+Agf2Q== -indexWithPartitions=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAIAAAAAgEAAQEBAAAAGAAAAAIAAAAAAAAAAAAAAAAAAAAYAQkBCwAAAAAAAAAAnQAAAAAAAADIAAAAAAAAAAIBAAAACAAAAAIBAAEBAQAAAB4AAAACAAAAAP////4AAAeAzDhsZwEEAePY4eGM8AEAAAAAAAAAAWUAAAAAAAAAZAAAAAAAAAACAQAAAAgAAAACAQABAQEAAAAgAAAAAgAAAAAAAAAUf////////+sBBAHn/////////38AKbEFIt3sTGH4L0/6GN34pXMhi0acHt0pyDLHlKI0ias= -indexWithBuckets=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAIAAAAAgEAAQEBAAAAGAAAAAIAAAAAAAAAAAAAAAAAAAAYAQkBCwEAAAAMAAAAAgGEgICAEAEEAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAgAAAACAQABAQEAAAAeAAAAAgAAAAD////+AAAHgMw4bGcBBAHj2OHhjPABAQAAAAwAAAACAYSAgIAgAQQAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAACAAAAAIBAAEBAQAAACAAAAACAAAAAAAAABR/////////6wEEAef/////////fwEAAAAMAAAAAgEBAYOAgIAwO1kfZjtY7swim4i4X6yEF9xO6jMe9WyV+Sq2sbpwRPI= +index=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAADkAAAAAAAAAZAAAAAAAAAADAAEAAAAWAAAAAgAAAAAAAAAAAAAAAAAAABgJCwAAAAAAAAAAnQAAAAAAAADIAAAAAAAAAAIAAQAAABwAAAACAAAAAP////4AAAeAzDhsZwTj2OHhjPABAAAAAAAAAAFlAAAAAAAAAGQAAAAAAAAAAgABAAAAHgAAAAIAAAAAAAAAFH/////////rBOf/////////fwBDmcZxzJVsvy0z3TOchlj6Hm/yYQcj7cvI9GCUkgLTuQ== +indexWithPartitions=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAGAAAAAgABAQAAABYAAAACAAAAAAAAAAAAAAAAAAAAGAkLAAAAAAAAAACdAAAAAAAAAMgAAAAAAAAAAgEAAAAGAAAAAgABAQAAABwAAAACAAAAAP////4AAAeAzDhsZwTj2OHhjPABAAAAAAAAAAFlAAAAAAAAAGQAAAAAAAAAAgEAAAAGAAAAAgABAQAAAB4AAAACAAAAAAAAABR/////////6wTn/////////38ANcXHqVsy9rnZu8lIbURyO1ruCjRBJ6y7D1d+3De2K5c= +indexWithBuckets=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAGAAAAAgABAQAAABYAAAACAAAAAAAAAAAAAAAAAAAAGAkLAQAAAAoAAAAChICAgBAEAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAYAAAACAAEBAAAAHAAAAAIAAAAA/////gAAB4DMOGxnBOPY4eGM8AEBAAAACgAAAAKEgICAIAQAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAABgAAAAIAAQEAAAAeAAAAAgAAAAAAAAAUf////////+sE5/////////9/AQAAAAoAAAACAYOAgIAwtetSsOZbZ3HLMKzmCkypZMmlEoQ52HFpuzvnx1F2Ht0= From f46e60a24a18b81993513260fd1ddc02e969fcaf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 22:24:11 +0800 Subject: [PATCH 07/14] [core] Skip sidecar builds when writing is disabled --- docs/docs/concepts/spec/manifest.md | 5 +++-- .../apache/paimon/manifest/ManifestSidecar.java | 9 ++++++++- .../paimon/manifest/ManifestSidecarTest.java | 14 ++++++++++++++ 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 17db4bcc5a87..006e04198773 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -74,8 +74,9 @@ derived file name. The Avro schemas and `_VERSION` identifiers remain unchanged. The utility includes construction, validation, block selection and optional caching. Table writers and scans do not yet invoke it automatically. Callers are responsible for publishing sidecar references, managing file ownership, applying entry filters and reconciling ADD/DELETE -entries after block selection. `build` reads the completed physical manifest and returns sidecar -bytes; it does not write or publish another file. +entries after block selection. `build` returns null without opening files when `Settings.write` +is false. Otherwise it reads the completed physical manifest and returns sidecar bytes; it does +not write or publish another file. `Settings` contains `write` and `read` switches for the calling writer and scan, and enables row-ID and bucket payload generation independently. Partition generation is always enabled, diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index 42fb0d45523c..fbea29c217d0 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -360,9 +360,16 @@ private static ProjectedManifestEntry.Projection createBlockIndexProjection() { return ProjectedManifestEntry.Projection.create(new RowType(false, fields)); } - /** Rebuild from the final physical blocks, including raw-copy and encoded rewrite paths. */ + /** + * Rebuild from the final physical blocks, including raw-copy and encoded rewrite paths. Returns + * null without accessing files when sidecar writing is disabled. + */ + @Nullable public static byte[] build(FileIO io, Path path, long size, long records, Settings settings) throws IOException { + if (!settings.write) { + return null; + } try (ManifestAvroReader reader = new ManifestAvroReader(io.newInputStream(path))) { Builder builder = new Builder(settings, reader.headerBytes()); ProjectedManifestEntry.Projection projection = BLOCK_INDEX_PROJECTION; diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index daca76c42b19..eec1323162f9 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -115,6 +115,20 @@ void emptyManifestHasACompleteSidecar() throws Exception { .isEmpty(); } + @Test + void disabledBuildDoesNotAccessFiles() throws Exception { + FileIO io = mock(FileIO.class); + assertThat( + ManifestSidecar.build( + io, + new Path(temp.toString(), "missing-manifest"), + 100, + 1, + new ManifestSidecar.Settings(false, true, true, true))) + .isNull(); + verifyNoInteractions(io); + } + @Test void buildAndReadSelectedBlocksFromPhysicalManifests() throws Exception { FileIO io = LocalFileIO.create(); From 84d94d3e8e8eb548dd0a8e413d065438674cb14f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 22:30:00 +0800 Subject: [PATCH 08/14] [core] Honor sidecar read settings before accessing metadata --- docs/docs/concepts/spec/manifest.md | 5 +- .../paimon/manifest/ManifestSidecar.java | 18 +++++-- .../paimon/manifest/ManifestSidecarTest.java | 52 +++++++++++++++---- 3 files changed, 60 insertions(+), 15 deletions(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 006e04198773..d9e0ceee14f8 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -84,8 +84,9 @@ including the empty partition tuple for unpartitioned tables. Missing or invalid metadata makes only the affected block's dimension unavailable. There is no sidecar byte budget: construction keeps complete coverage and `read` consumes the entire file once it is opened. -`read` returns null for an absent sidecar reference or an `IOException`, allowing the caller -to fall back to the manifest. If the thread is interrupted, the I/O failure is propagated as +`read` returns null immediately when `Settings.read` is false, without inspecting metadata, +accessing the cache or opening files. An absent sidecar reference or an `IOException` also +returns null, allowing the caller to fall back to the manifest. If the thread is interrupted, the I/O failure is propagated as `UncheckedIOException`. Other exceptions and errors propagate unchanged. `select` validates supplied bytes directly and reports invalid containers with `IOException`. diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index fbea29c217d0..8fe85e88cadf 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -587,11 +587,15 @@ private static Payload payload(ByteBuffer in, long records) throws IOException { return new Payload(count, result); } - /** Reads the complete sidecar. Null means read the original manifest. */ + /** Reads the complete sidecar when enabled. Null means read the original manifest. */ @Nullable public static Selection read( - FileIO io, Path path, ManifestFileMeta manifest, @Nullable RowRangeIndex query) { - return read(io, path, manifest, query, null, null); + FileIO io, + Path path, + ManifestFileMeta manifest, + Settings settings, + @Nullable RowRangeIndex query) { + return read(io, path, manifest, settings, query, null, null); } @Nullable @@ -599,10 +603,12 @@ public static Selection read( FileIO io, Path path, ManifestFileMeta manifest, + Settings settings, @Nullable RowRangeIndex query, @Nullable PartitionPredicate partitionFilter, @Nullable RowType partitionType) { - return read(io, path, manifest, query, partitionFilter, partitionType, null, null); + return read( + io, path, manifest, settings, query, partitionFilter, partitionType, null, null); } @Nullable @@ -610,11 +616,15 @@ public static Selection read( FileIO io, Path path, ManifestFileMeta manifest, + Settings settings, @Nullable RowRangeIndex query, @Nullable PartitionPredicate partitionFilter, @Nullable RowType partitionType, @Nullable BiPredicate bucketFilter, @Nullable SegmentsCache cache) { + if (!settings.read) { + return null; + } String sidecarFileName = fileName(manifest); if (sidecarFileName == null) { return null; diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index eec1323162f9..4ca4ab68b919 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -129,6 +129,22 @@ void disabledBuildDoesNotAccessFiles() throws Exception { verifyNoInteractions(io); } + @Test + void disabledReadsDoNotAccessMetadataCacheOrFiles() { + FileIO io = mock(FileIO.class); + ManifestFileMeta manifest = mock(ManifestFileMeta.class); + SegmentsCache cache = mock(SegmentsCache.class); + Path path = new Path(temp.toString(), "missing-manifest"); + ManifestSidecar.Settings disabled = new ManifestSidecar.Settings(true, false, true, true); + assertThat(ManifestSidecar.read(io, path, manifest, disabled, null)).isNull(); + assertThat(ManifestSidecar.read(io, path, manifest, disabled, null, null, null)).isNull(); + assertThat( + ManifestSidecar.read( + io, path, manifest, disabled, null, null, null, null, cache)) + .isNull(); + verifyNoInteractions(io, manifest, cache); + } + @Test void buildAndReadSelectedBlocksFromPhysicalManifests() throws Exception { FileIO io = LocalFileIO.create(); @@ -474,7 +490,8 @@ void cachedBytesPreservePerQueryCancellation() throws Exception { assertThatThrownBy( () -> ManifestSidecar.read( - io, path, meta, cancelled, null, null, null, cache)) + io, path, meta, settings, cancelled, null, null, null, + cache)) .isInstanceOf(CancellationException.class); assertThat(cache.getIfPresents(sidecar)).isNull(); @@ -482,7 +499,8 @@ void cachedBytesPreservePerQueryCancellation() throws Exception { assertThatThrownBy( () -> ManifestSidecar.read( - io, path, meta, cancelled, null, null, null, cache)) + io, path, meta, settings, cancelled, null, null, null, + cache)) .isInstanceOf(CancellationException.class); assertThat(readCached(io, path, meta, cache).blocks()).hasSize(2); verify(io, times(2)).newInputStream(sidecar); @@ -494,6 +512,7 @@ private ManifestSidecar.Selection readCached( io, path, meta, + settings, RowRangeIndex.create(Collections.singletonList(new Range(20, 20))), null, null, @@ -508,13 +527,15 @@ void corruptMissingIncompleteAndMismatchedIndexesFallback() throws Exception { ManifestFileMeta meta = goldenMeta(); RowRangeIndex query = RowRangeIndex.create(Collections.singletonList(new Range(11, 11))); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, settings, query)) + .isNull(); byte[] good = golden(); for (int position : new int[] {0, 9, 11, 15, 16, 55, 63, 67, 75, good.length - 1}) { byte[] bad = good.clone(); bad[position] ^= 2; Files.write(index, bad); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, settings, query)) + .isNull(); } // A valid checksum cannot make an unsupported container version readable. for (int version : new int[] {0, 2, 99}) { @@ -528,9 +549,12 @@ void corruptMissingIncompleteAndMismatchedIndexesFallback() throws Exception { .isInstanceOf(IOException.class); } Files.write(index, Arrays.copyOf(good, good.length - 1)); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, settings, query)) + .isNull(); Files.write(index, good); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query).blocks()) + assertThat( + ManifestSidecar.read(LocalFileIO.create(), manifest, meta, settings, query) + .blocks()) .isEmpty(); // The sidecar is bound to physical coverage, not to a particular file name. assertThat( @@ -568,7 +592,8 @@ void ioFailuresFallBackWithoutInspectingNestedExceptions() throws Exception { suppressed)) { FileIO fileIO = mock(FileIO.class); when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); - assertThat(ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)).isNull(); + assertThat(ManifestSidecar.read(fileIO, path, meta("m", 1, 1), settings, null)) + .isNull(); assertThat(Thread.currentThread().isInterrupted()).isFalse(); } } @@ -581,7 +606,10 @@ void interruptedThreadDoesNotFallBackOnIoFailure() throws Exception { when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); try { Thread.currentThread().interrupt(); - assertThatThrownBy(() -> ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)) + assertThatThrownBy( + () -> + ManifestSidecar.read( + fileIO, path, meta("m", 1, 1), settings, null)) .isInstanceOf(java.io.UncheckedIOException.class) .hasCauseReference(failure); assertThat(Thread.currentThread().isInterrupted()).isTrue(); @@ -601,7 +629,10 @@ void uncheckedFailuresPropagateUnchanged() throws Exception { new AssertionError("error"))) { FileIO fileIO = mock(FileIO.class); when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); - assertThatThrownBy(() -> ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)) + assertThatThrownBy( + () -> + ManifestSidecar.read( + fileIO, path, meta("m", 1, 1), settings, null)) .isSameAs(failure); } } @@ -628,6 +659,7 @@ void indexReadsUseBoundedBulkRequests() throws Exception { io, path, meta, + settings, RowRangeIndex.create(Collections.singletonList(new Range(0, 0)))); assertThat(actual.blocks()).hasSize(1); assertThat(actual.blocks().get(0).offset).isEqualTo(header.length); @@ -660,6 +692,7 @@ void sidecarsLargerThanTheFormerDefaultLimitAreReadCompletely() throws Exception io, path, meta("manifest-large", header.length + 100, 1), + settings, null) .blocks()) .hasSize(1); @@ -682,6 +715,7 @@ void indexShortReadsReadTheWholeFile() throws Exception { io, path, goldenMeta(), + settings, RowRangeIndex.create(Collections.singletonList(new Range(20, 20)))); assertThat(actual.blocks()) .extracting(block -> block.firstRecord) From f0ce99722454d6e228cc886907e638b4a2509a07 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 22:58:46 +0800 Subject: [PATCH 09/14] [core] Separate sidecar and manifest block read buffers --- docs/docs/concepts/spec/manifest.md | 3 +- .../paimon/manifest/ManifestSidecar.java | 14 +++--- .../paimon/manifest/ManifestSidecarTest.java | 43 ++++++++++++------- 3 files changed, 39 insertions(+), 21 deletions(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index d9e0ceee14f8..74e7d2335afb 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -269,7 +269,8 @@ different entries, so entry filtering and ADD/DELETE reconciliation remain neces The entire sidecar is read in chunks of at most 1 MiB, including payloads unused by a query. There is no size-based fallback or payload dropping. Payload lengths save decoding work, not sidecar storage I/O. Selected compressed Avro blocks are read by byte range with adjacent -spans coalesced. Building a sidecar does not modify the original manifest. +spans coalesced and individual read requests bounded to 4 MiB. Building a sidecar does not +modify the original manifest. `read` and `openManifest` accept an optional caller-supplied `SegmentsCache`. Complete sidecar bytes are keyed by their explicit `Path`. Only successful reads and selections diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index 8fe85e88cadf..0ac640ebc380 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -68,7 +68,8 @@ public final class ManifestSidecar { private static final int BLOCK_BYTES = 27; private static final byte[] EMPTY = new byte[0]; private static final int DIGEST_BYTES = 32; - private static final int READ_BUFFER_BYTES = 1024 * 1024; + private static final int SIDECAR_READ_BUFFER_BYTES = 1024 * 1024; + private static final int BLOCK_READ_BUFFER_BYTES = 4 * 1024 * 1024; private static final ProjectedManifestEntry.Projection BLOCK_INDEX_PROJECTION = createBlockIndexProjection(); @@ -673,7 +674,7 @@ public long totalMemorySize() { private static byte[] readBytes(FileIO io, Path path) throws IOException { try (InputStream in = io.newInputStream(path)) { ByteArrayOutputStream out = new ByteArrayOutputStream(); - byte[] buffer = new byte[READ_BUFFER_BYTES]; + byte[] buffer = new byte[SIDECAR_READ_BUFFER_BYTES]; int n; while ((n = in.read(buffer, 0, buffer.length)) != -1) { out.write(buffer, 0, n); @@ -817,7 +818,7 @@ private boolean fillBuffer() throws IOException { seekInput(block.offset); remaining = end - block.offset; } - int requested = (int) Math.min(READ_BUFFER_BYTES, remaining); + int requested = (int) Math.min(BLOCK_READ_BUFFER_BYTES, remaining); // A previous buffer may be shared with other readers through the block cache. if (cache != null || buffer == null || buffer.length < requested) { buffer = new byte[requested]; @@ -842,7 +843,7 @@ private void readCachedBlocks(Block first) throws IOException { Block next = selected.blocks.get(blockPosition); if (next.offset != end || next.length > cache.maxElementSize() - || end - first.offset + next.length > READ_BUFFER_BYTES + || end - first.offset + next.length > BLOCK_READ_BUFFER_BYTES || cachedBlock(next) != null) { break; } @@ -898,7 +899,10 @@ private void readFully(byte[] bytes, int length) throws IOException { int position = 0; while (position < length) { int count = - input.read(bytes, position, Math.min(READ_BUFFER_BYTES, length - position)); + input.read( + bytes, + position, + Math.min(BLOCK_READ_BUFFER_BYTES, length - position)); if (count < 0) { throw new EOFException("Truncated manifest block"); } diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index 4ca4ab68b919..3f2466fb91ad 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -936,8 +936,8 @@ void evictedBlocksAreReadAgainWithinTheSharedBudget() throws Exception { @Test void oversizedBlocksUseBoundedReadsWithoutModifyingPreviouslyCachedBytes() throws Exception { byte[] header = header(); - int cachedLength = (1 << 20) + 17; - int uncachedLength = 2 * (1 << 20) + 31; + int cachedLength = (4 << 20) + 17; + int uncachedLength = 2 * (4 << 20) + 31; ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); builder.beginBlock(header.length, cachedLength, 1); builder.add(0L, 1); @@ -962,7 +962,7 @@ void oversizedBlocksUseBoundedReadsWithoutModifyingPreviouslyCachedBytes() throw try (InputStream in = ManifestSidecar.openManifest(io, path, first, cache)) { assertThat(IOUtils.readFully(in, false)).isEqualTo(expected); } - assertThat(cold.readLengths).containsExactly(1 << 20, 17); + assertThat(cold.readLengths).containsExactly(4 << 20, 17); ManifestSidecar.Selection all = ManifestSidecar.select( data, @@ -972,7 +972,7 @@ void oversizedBlocksUseBoundedReadsWithoutModifyingPreviouslyCachedBytes() throw try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { assertThat(IOUtils.readFully(in, false)).isEqualTo(manifest); } - assertThat(mixed.readLengths).containsExactly(1 << 20, 1 << 20, 31); + assertThat(mixed.readLengths).containsExactly(4 << 20, 4 << 20, 31); assertThat(cache.estimatedSize()).isEqualTo(1); assertThat( cache.getIfPresents( @@ -990,7 +990,7 @@ void largeBlockSpansUseBoundedReads() throws Exception { byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); long offset = header.length; - for (int length : new int[] {512 * 1024, 512 * 1024, 257}) { + for (int length : new int[] {2 << 20, 2 << 20, 257}) { builder.beginBlock(offset, length, 1); builder.add(20L, 1); builder.endBlock(); @@ -998,18 +998,31 @@ void largeBlockSpansUseBoundedReads() throws Exception { } byte[] data = builder.serialize(offset, 3); byte[] manifest = Arrays.copyOf(header, (int) offset); - CountingInput stream = new CountingInput(manifest, Integer.MAX_VALUE); - FileIO io = mock(FileIO.class); Path path = new Path(temp.toString(), "manifest-large"); - when(io.newInputStream(path)).thenReturn(stream); - try (InputStream input = - ManifestSidecar.openManifest( - io, path, select(data, meta("manifest-large", offset, 3), 20))) { - assertThat(IOUtils.readFully(input, false)).isEqualTo(manifest); + for (boolean withCache : new boolean[] {false, true}) { + CountingInput stream = new CountingInput(manifest, Integer.MAX_VALUE); + FileIO io = mock(FileIO.class); + when(io.newInputStream(path)).thenReturn(stream); + SegmentsCache cache = + withCache + ? new SegmentsCache<>( + 1024, MemorySize.ofMebiBytes(8), 4 << 20, null, false) + : null; + try (InputStream input = + ManifestSidecar.openManifest( + io, path, select(data, meta("manifest-large", offset, 3), 20), cache)) { + assertThat(IOUtils.readFully(input, false)).isEqualTo(manifest); + } + assertThat(stream.readLengths).containsExactly(4 << 20, 257); + if (withCache) { + assertThat(stream.seeks) + .containsExactly((long) header.length, header.length + (4L << 20)); + assertThat(cache.estimatedSize()).isEqualTo(3); + } else { + assertThat(stream.seeks).containsExactly((long) header.length); + } + assertThat(stream.closed).isTrue(); } - assertThat(stream.readLengths).containsExactly(1 << 20, 257); - assertThat(stream.seeks).containsExactly((long) header.length); - assertThat(stream.closed).isTrue(); } @Test From 5b8ffb515c282d693b306ba03423e8c04c8aaf56 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Tue, 15 Sep 2026 23:00:29 +0800 Subject: [PATCH 10/14] [core] Test nine MiB of consecutive manifest blocks --- .../paimon/manifest/ManifestSidecarTest.java | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index 3f2466fb91ad..4a3f0a4d4039 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -990,13 +990,13 @@ void largeBlockSpansUseBoundedReads() throws Exception { byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); long offset = header.length; - for (int length : new int[] {2 << 20, 2 << 20, 257}) { + for (int length : new int[] {2 << 20, 2 << 20, 2 << 20, 2 << 20, 1 << 20}) { builder.beginBlock(offset, length, 1); builder.add(20L, 1); builder.endBlock(); offset += length; } - byte[] data = builder.serialize(offset, 3); + byte[] data = builder.serialize(offset, 5); byte[] manifest = Arrays.copyOf(header, (int) offset); Path path = new Path(temp.toString(), "manifest-large"); for (boolean withCache : new boolean[] {false, true}) { @@ -1006,18 +1006,21 @@ void largeBlockSpansUseBoundedReads() throws Exception { SegmentsCache cache = withCache ? new SegmentsCache<>( - 1024, MemorySize.ofMebiBytes(8), 4 << 20, null, false) + 1024, MemorySize.ofMebiBytes(16), 4 << 20, null, false) : null; try (InputStream input = ManifestSidecar.openManifest( - io, path, select(data, meta("manifest-large", offset, 3), 20), cache)) { + io, path, select(data, meta("manifest-large", offset, 5), 20), cache)) { assertThat(IOUtils.readFully(input, false)).isEqualTo(manifest); } - assertThat(stream.readLengths).containsExactly(4 << 20, 257); + assertThat(stream.readLengths).containsExactly(4 << 20, 4 << 20, 1 << 20); if (withCache) { assertThat(stream.seeks) - .containsExactly((long) header.length, header.length + (4L << 20)); - assertThat(cache.estimatedSize()).isEqualTo(3); + .containsExactly( + (long) header.length, + header.length + (4L << 20), + header.length + (8L << 20)); + assertThat(cache.estimatedSize()).isEqualTo(5); } else { assertThat(stream.seeks).containsExactly((long) header.length); } From fd998cc6d814fb07e1dfc94082721439fc462c6b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Wed, 16 Sep 2026 10:05:57 +0800 Subject: [PATCH 11/14] Fix typo --- .../src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java | 1 + .../main/java/org/apache/paimon/manifest/ManifestSidecar.java | 1 + .../java/org/apache/paimon/manifest/ManifestBlockIndexTest.java | 1 + .../java/org/apache/paimon/manifest/ManifestSidecarTest.java | 1 + 4 files changed, 4 insertions(+) diff --git a/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java b/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java index bae8e557443a..75365a13dfa4 100644 --- a/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java +++ b/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java @@ -25,6 +25,7 @@ /** Streaming unsigned delta/varint encoding for nondecreasing, nonnegative long values. */ public final class DeltaVarintCodec { + private DeltaVarintCodec() {} /** Writes each delta immediately. The caller owns the output and stores the value count. */ diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index 0ac640ebc380..a01b5bbd20c7 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -60,6 +60,7 @@ /** Independently usable partition, row-id and bucket coverage for each manifest block. */ public final class ManifestSidecar { + public static final String SUFFIX = ".avro.sidecar"; private static final Logger LOG = LoggerFactory.getLogger(ManifestSidecar.class); private static final int MAGIC = 0x504d5343; diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java index 1e4aad33212d..5df35a422314 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java @@ -57,6 +57,7 @@ /** Complete sidecar payload format, compression and independent filtering. */ class ManifestBlockIndexTest { + private final RowType type = RowType.of(DataTypes.INT(), DataTypes.STRING()); private final ManifestSidecar.Settings defaults = new ManifestSidecar.Settings(true, true, true, true); diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index 4a3f0a4d4039..6b01fa54788a 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -72,6 +72,7 @@ /** Cross-language format, physical block positions, completeness and allocation bounds. */ class ManifestSidecarTest { + @TempDir java.nio.file.Path temp; private final ManifestSidecar.Settings settings = new ManifestSidecar.Settings(true, true, true, true); From 36ea6b659f72a96be50026be24da3b34e8c782d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Wed, 16 Sep 2026 13:44:31 +0800 Subject: [PATCH 12/14] [core] Simplify manifest sidecar framing and delta payloads --- docs/docs/concepts/spec/manifest.md | 163 +++++----- .../apache/paimon/utils/DeltaVarintCodec.java | 64 +++- .../paimon/utils/DeltaVarintCodecTest.java | 126 ++++++-- .../paimon/manifest/ManifestSidecar.java | 280 +++++++++--------- .../manifest/ManifestBlockIndexTest.java | 206 +++++++++---- .../paimon/manifest/ManifestSidecarTest.java | 96 ++---- .../src/test/resources/manifest-sidecar.txt | 6 +- 7 files changed, 565 insertions(+), 376 deletions(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 74e7d2335afb..ceaa8a0e4af8 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -74,54 +74,52 @@ derived file name. The Avro schemas and `_VERSION` identifiers remain unchanged. The utility includes construction, validation, block selection and optional caching. Table writers and scans do not yet invoke it automatically. Callers are responsible for publishing sidecar references, managing file ownership, applying entry filters and reconciling ADD/DELETE -entries after block selection. `build` returns null without opening files when `Settings.write` -is false. Otherwise it reads the completed physical manifest and returns sidecar bytes; it does -not write or publish another file. +entries after block selection. `build` reads the completed physical manifest and returns +sidecar bytes; it does not write or publish another file. -`Settings` contains `write` and `read` switches for the calling writer and scan, and enables -row-ID and bucket payload generation independently. Partition generation is always enabled, +Callers decide whether to invoke `build` and `read`; these utilities have no read/write switches. +`build` and `Builder` accept `rowIdEnabled` and `bucketEnabled` arguments for independent +payload generation. Partition generation is always enabled, including the empty partition tuple for unpartitioned tables. Missing or invalid metadata makes only the affected block's dimension unavailable. There is no sidecar byte budget: construction keeps complete coverage and `read` consumes the entire file once it is opened. -`read` returns null immediately when `Settings.read` is false, without inspecting metadata, -accessing the cache or opening files. An absent sidecar reference or an `IOException` also -returns null, allowing the caller to fall back to the manifest. If the thread is interrupted, the I/O failure is propagated as +`read` returns null for an absent sidecar reference or an `IOException`, allowing the caller +to fall back to the manifest. If the thread is interrupted, the I/O failure is propagated as `UncheckedIOException`. Other exceptions and errors propagate unchanged. `select` validates supplied bytes directly and reports invalid containers with `IOException`. -Version 1 uses the following layout. Container `int` and `long` fields are signed, fixed-width -4-byte and 8-byte big-endian integers. Encoding IDs are unsigned bytes with separate namespaces. -Payload counts and envelopes use the same fixed-width types; delta streams use the -variable-length encoding described below. +Version 1 uses the following layout. Counts, lengths, offsets and the version use canonical +nonnegative unsigned LEB128 varints. Counts and payload lengths are bounded by `Integer.MAX_VALUE`; +block offsets, lengths and record counts are bounded by `Long.MAX_VALUE`. Row-ID envelope +endpoints remain fixed-width, eight-byte big-endian longs. Encoding IDs are unsigned bytes +with separate namespaces. The existing serialized partition tuple bytes are unchanged. ```text magic : 4 bytes // ASCII PMSC -formatVersion : int // 1 -manifestLength : long -manifestEntryCount : long // ADD + DELETE -avroHeaderLength : int +formatVersion : varint // 1 +avroHeaderLength : varint avroHeader : bytes // original schema, codec and sync marker -partitionCount : int +partitionCount : varint partitionDictionary[] - partitionByteLength : int + partitionByteLength : varint partitionBytes : bytes // existing manifest BinaryRow serialization -blockCount : int +blockCount : varint blocks[] // original physical order - offset : long - length : long // complete encoded block, including sync marker - recordCount : long + offset : varint + length : varint // complete encoded block, including sync marker + recordCount : varint partitionEncoding : byte if partitionEncoding != 0: - partitionPayloadLength : int + partitionPayloadLength : varint partitionPayload : bytes rowIdEncoding : byte if rowIdEncoding != 0: - rowIdPayloadLength : int + rowIdPayloadLength : varint rowIdPayload : bytes bucketEncoding : byte if bucketEncoding != 0: - bucketPayloadLength : int + bucketPayloadLength : varint bucketPayload : bytes checksum : 32 bytes // SHA-256 of all preceding bytes ``` @@ -134,26 +132,34 @@ Partition predicates are evaluated once per dictionary entry. | Dimension | Encoding | Payload | | --- | --- | --- | | Any | `0` | Unavailable; only the encoding byte is present. | -| Partition | `1` | Count and delta/varint-compressed sorted unique dictionary IDs. | -| Row ID | `1` | Interval count, minimum, span, and delta/varint-compressed interior endpoints. | -| Bucket | `1` | Count and delta/varint-compressed sorted unique packed bucket/count pairs. | +| Partition | `1` | `intsDeltaPayload` of sorted unique dictionary IDs. | +| Row ID | `1` | Minimum, maximum, and `intsDeltaPayload` of sorted interior interval endpoints. | +| Bucket | `1` | Two paired `intsDeltaPayload` sequences: sorted bucket IDs and their recorded total bucket counts. | | Any | Other nonzero ID | Skip the declared payload length; treat only this dimension as unavailable. | Only nonzero encodings are followed by a length and payload. Payload lengths exclude the encoding and length fields, but include the count and other fields within the payload. -All three encoding-1 payloads have positive counts no greater than the block's record count. -Encoding 0 represents unavailable coverage, rather than encoding 1 with a zero count. +Partition IDs and bucket pairs have positive counts no greater than the block's record count. +Row-ID coverage contains one or more intervals; its interior endpoint count can be zero for +a single interval. Encoding 0 represents unavailable coverage, not an empty known set. -#### Delta Encoding +#### Integer Delta Payload -Each payload starts with a fixed-width count (`int`); row-ID payloads also have fixed-width -`min` and `span` fields (`long`). Only integers in the following delta stream use -nonnegative unsigned LEB128 varints, occupying one to nine bytes for values from 0 through +Partition, row-ID and both bucket sequences share this structure: + +```text +intsDeltaPayload + count : varint + deltas[count] : varint +``` + +The count is in `[0, Integer.MAX_VALUE]`. Nonnegative deltas use unsigned LEB128 varints, +occupying one to nine bytes for values from 0 through `Long.MAX_VALUE`. Seven value bits are stored per byte, least significant group first; the high bit indicates another byte follows. -Encodings use the shortest representation. There is no ZigZag transformation or padding. +Encodings use the shortest representation without padding. -A sorted sequence is delta-encoded from a specified base. Each value contributes one +A nondecreasing sequence is delta-encoded from a specified base. Each value contributes one unsigned varint containing its difference from the preceding value. The first difference is relative to the base: @@ -163,10 +169,18 @@ value[0] = base + deltas[0] value[i] = value[i - 1] + deltas[i] ``` -The shared `DeltaVarintCodec` utility writes each delta immediately and reads values on -demand, using `VarLengthIntUtils` for varints. Counts and bounds are supplied by the caller. -The reader checks overflow and value bounds and requires the buffer to end after all -expected values have been consumed. It can stop early without materializing the sequence. +The shared `DeltaVarintCodec` utility writes the count and then each delta immediately, +and reads values on demand using `VarLengthIntUtils`. A reader consumes exactly the declared +number of values, leaving any following sequence available in the buffer. Callers check +their enclosing payload boundaries. Counts, overflow and value bounds are checked without +materializing arrays. Reads may stop early. + +Only the `totalBuckets` sequence uses signed differences, because totals need not increase +when pairs are sorted by bucket. Its differences use ZigZag before unsigned varint encoding: +`encoded = (delta << 1) ^ (delta >> 63)` and +`delta = (encoded >>> 1) ^ -(encoded & 1)`. Values are nonnegative ints, so encoded deltas +are at most `2 * Integer.MAX_VALUE` and require at most five bytes. The field defines this +signed mode; no additional mode byte is stored. Other sequences use nonnegative differences. #### Partition Payload @@ -175,15 +189,14 @@ represented by its entries: ```text partitionPayload - partitionIdCount : int // N > 0 - deltas[] // N dictionary IDs, base = 0 + intsDeltaPayload // N > 0 dictionary IDs, base = 0 ``` An ID is the zero-based position of a complete tuple in the sidecar's shared dictionary. IDs satisfy `0 <= id < partitionCount` and are strictly increasing. Tuple bytes appear only in the dictionary and are not repeated in each block. For IDs `[0, 1, 2, 3, 4]`, the deltas -are `[0, 1, 1, 1, 1]`. The payload contains a four-byte count of 5 followed by these five -varint bytes: 9 bytes, or 14 bytes including the encoding and length fields. +are `[0, 1, 1, 1, 1]`. The payload contains a one-byte count of 5 followed by these five +varint bytes: 6 bytes, or 8 bytes including the encoding and length fields. With a partition filter, a block matches if any referenced tuple matches. A tuple containing a null partition value still has a dictionary ID. Unpartitioned tables record the empty @@ -198,24 +211,25 @@ sorted and disjoint; they are never expanded into individual row IDs or coarsene ```text rowIdPayload - rangeCount : int // N > 0 - min : long // first interval's start - span : long // last interval's end minus min - deltas[] // 2 * (N - 1) interior endpoints, base = min + minRowId : long // first interval's start + maxRowId : long // last interval's inclusive end + intsDeltaPayload // 2 * (N - 1) sorted interior endpoints, base = minRowId ``` -The maximum is `min + span`, which must not exceed `Long.MAX_VALUE`. Flatten the intervals -as `[start0, end0, start1, end1, ...]`. The first start is supplied by `min`, and the last -end by `min + span`; only the remaining `2 * (N - 1)` interior endpoints are delta/varint encoded. +The envelope satisfies `0 <= minRowId <= maxRowId <= Long.MAX_VALUE`. Flatten the intervals +as `[start0, end0, start1, end1, ...]`. The first start is supplied by `minRowId`, and the last +end by `maxRowId`; only the remaining `2 * (N - 1)` interior endpoints are delta/varint encoded. +Their count must be even, and the derived interval count `N = count / 2 + 1` must not exceed +the block's record count. There is no separate stored interval count. Pairing the reconstructed endpoints recovers the intervals. Each pair satisfies `0 <= start <= end <= Long.MAX_VALUE`; each following start must exceed the preceding end. -For `[(10, 19), (30, 39)]`, the count is 2, minimum is 10, and span is 29. The interior +For `[(10, 19), (30, 39)]`, the minimum is 10 and maximum is 39. The interior endpoints `[19, 30]` have deltas `[9, 11]` from base 10, each encoded as one varint byte. -The payload starts with a four-byte count of 2, an eight-byte minimum of 10, and an eight-byte -span of 29, followed by the delta bytes `[9, 11]`: 22 bytes, or 27 bytes with framing. -For a single interval, the 20-byte fixed-width prefix completely defines the interval and -no deltas follow. +The payload starts with an eight-byte minimum of 10 and an eight-byte maximum of 39, +followed by `intsDeltaPayload` bytes `[2, 9, 11]`: 19 bytes, or 21 bytes with framing. +For a single interval, the two eight-byte endpoints and a one-byte zero count define the +interval: 17 payload bytes and no deltas. The reader first tests the envelope without decoding any deltas. A query for row ID 25 passes the example's envelope check but matches neither interval. Unknown or invalid row-ID @@ -227,21 +241,18 @@ When `bucketEncoding == 1`, the block stores distinct bucket/count pairs: ```text bucketPayload - pairCount : int // N > 0 - deltas[] // N packed pairs, base = 0 - -packedPair = ((long) bucket << 32) | totalBuckets + buckets : intsDeltaPayload // N > 0, base = 0, nonnegative deltas + totalBuckets : intsDeltaPayload // N values, base = 0, ZigZag signed deltas ``` -Each pair satisfies `0 <= bucket < totalBuckets`. Packing places the bucket in the high -32 bits and the recorded total bucket count in the low 32 bits. Packed values are nonnegative, -sorted and unique, equivalent to sorting first by bucket and then by total bucket count. -The decoder recovers `bucket = (int) (packedPair >>> 32)` and `totalBuckets = (int) packedPair`. -The same bucket may occur with different totals after rescaling. +Pairs are sorted first by bucket, then by total bucket count, with duplicates removed. +Each pair satisfies `0 <= bucket < totalBuckets <= Integer.MAX_VALUE`. Both sequences have +the same count, and values at the same position form one pair. The totals must not be sorted +independently. The same bucket may occur with different totals after rescaling. -For `[(1, 4), (1, 8), (3, 4)]`, the packed values are `[4294967300, 4294967304, 12884901892]` -and deltas are `[4294967300, 4, 8589934588]`. The payload contains a four-byte count of 3 -and three varints occupying 5, 1 and 5 bytes: 15 bytes total, or 20 bytes with framing. +For `[(1, 4), (1, 8), (3, 4)]`, bucket values `[1, 1, 3]` have deltas `[1, 0, 2]`. +Paired totals `[4, 8, 4]` have signed deltas `[4, 4, -4]`, ZigZag-encoded as `[8, 8, 7]`. +The two payloads are `[3, 1, 0, 2]` and `[3, 8, 8, 7]`: 8 bytes total, or 10 bytes with framing. Missing, invalid or negative/synthetic bucket metadata makes the block's bucket coverage unavailable. A caller can supply a predicate on `(bucket, totalBuckets)` which conservatively @@ -250,14 +261,18 @@ the entry-filtering stage; omit the bucket predicate if no safe check is availab #### Validation and Reading -Readers validate the checksum, fixed container fields, payload lengths and count headers, +Readers validate the checksum, container fields, payload lengths and leading count headers, and the complete physical block directory regardless of the query. Byte spans must cover the whole original manifest after its header; record counts must sum to the manifest entry -count. Unknown nonzero encodings skip their declared bytes without interpreting a count. - -Compressed contents are decoded only for dimensions needed by the filters and only until -that dimension matches. A row-ID envelope rejection skips all its deltas; a matching interval, -partition ID or bucket pair skips remaining values. Invalid varints, value counts, overflows, +count. Manifest length and entry count come from the supplied `ManifestFileMeta` rather +than being duplicated in the sidecar. Unknown nonzero encodings skip their declared bytes +without interpreting a count. + +Compressed contents are decoded only for dimensions needed by the filters. A row-ID envelope +rejection skips all its deltas; a matching interval or partition ID skips remaining values. +Bucket filtering first walks the bucket sequence to locate the paired totals without +allocating arrays, then decodes pairs until a match. Unused totals may be skipped. +Invalid varints, value counts, overflows, out-of-range values or ordering encountered while decoding invalidate the container. Delta contents skipped by short-circuiting are not individually validated. diff --git a/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java b/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java index 75365a13dfa4..9c6575bb4f91 100644 --- a/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java +++ b/paimon-common/src/main/java/org/apache/paimon/utils/DeltaVarintCodec.java @@ -23,48 +23,81 @@ import java.nio.ByteBuffer; import java.util.Objects; -/** Streaming unsigned delta/varint encoding for nondecreasing, nonnegative long values. */ +/** Streaming count-prefixed delta/varint encoding of nonnegative integer sequences. */ public final class DeltaVarintCodec { private DeltaVarintCodec() {} - /** Writes each delta immediately. The caller owns the output and stores the value count. */ + /** Writes a varint count followed by deltas, without buffering the sequence. */ public static final class Writer { private final DataOutput out; + private final boolean signedDeltas; + private long remaining; private long previous; - public Writer(DataOutput out, long base) { - if (base < 0) { - throw new IllegalArgumentException("Delta base must be nonnegative"); + public Writer(DataOutput out, int count, long base) throws IOException { + this(out, count, base, false); + } + + /** Signed deltas support non-monotonic nonnegative int values using ZigZag. */ + public Writer(DataOutput out, int count, long base, boolean signedDeltas) + throws IOException { + if (base < 0 || count < 0 || (signedDeltas && base > Integer.MAX_VALUE)) { + throw new IllegalArgumentException("Invalid delta/varint count or base"); } this.out = Objects.requireNonNull(out); + this.signedDeltas = signedDeltas; + this.remaining = count; previous = base; + VarLengthIntUtils.encodeInt(out, count); } public void write(long value) throws IOException { - require(value >= previous); - VarLengthIntUtils.encodeLong(out, value - previous); + require(remaining > 0 && value >= 0); + long delta = value - previous; + if (signedDeltas) { + require(value <= Integer.MAX_VALUE); + delta = (delta << 1) ^ (delta >> 63); + } else { + require(delta >= 0); + } + VarLengthIntUtils.encodeLong(out, delta); previous = value; + remaining--; } } /** - * Reads values on demand and advances the supplied buffer. The buffer must contain exactly the - * encoded sequence; a complete read validates its boundary. Reading may stop early. + * Reads the count and then values on demand. A complete read leaves the buffer at the next + * payload, allowing count-prefixed sequences to be concatenated. Reading may stop early. */ public static final class Reader { private final ByteBuffer data; private final long max; + private final int count; + private final boolean signedDeltas; private long remaining; private long value; - public Reader(ByteBuffer data, long count, long base, long max) throws IOException { + public Reader(ByteBuffer data, long base, long max) throws IOException { + this(data, base, max, false); + } + + public Reader(ByteBuffer data, long base, long max, boolean signedDeltas) + throws IOException { this.data = Objects.requireNonNull(data); - require(count >= 0 && count <= data.remaining() && base >= 0 && max >= base); - require(count != 0 || !data.hasRemaining()); + long encodedCount = VarLengthIntUtils.decodeLong(data); + require(encodedCount <= Integer.MAX_VALUE && encodedCount <= data.remaining()); + require(base >= 0 && max >= base && (!signedDeltas || max <= Integer.MAX_VALUE)); + count = (int) encodedCount; remaining = count; value = base; this.max = max; + this.signedDeltas = signedDeltas; + } + + public int count() { + return count; } public boolean hasNext() { @@ -74,10 +107,13 @@ public boolean hasNext() { public long next() throws IOException { require(remaining > 0); long delta = VarLengthIntUtils.decodeLong(data); - require(delta <= max - value); + if (signedDeltas) { + require(delta <= 2L * Integer.MAX_VALUE); + delta = (delta >>> 1) ^ -(delta & 1); + } + require(delta >= -value && delta <= max - value); value += delta; remaining--; - require(remaining != 0 || !data.hasRemaining()); return value; } } diff --git a/paimon-common/src/test/java/org/apache/paimon/utils/DeltaVarintCodecTest.java b/paimon-common/src/test/java/org/apache/paimon/utils/DeltaVarintCodecTest.java index 57261c44f3b8..dd667f475d35 100644 --- a/paimon-common/src/test/java/org/apache/paimon/utils/DeltaVarintCodecTest.java +++ b/paimon-common/src/test/java/org/apache/paimon/utils/DeltaVarintCodecTest.java @@ -30,18 +30,19 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; -/** Tests for the shared streaming unsigned delta/varint codec. */ +/** Tests for count-prefixed streaming delta/varint payloads. */ class DeltaVarintCodecTest { @Test void fixedBytesAndSlicedBuffer() throws Exception { long[] values = {10, 12, 14, 16, 16, 1024}; byte[] encoded = encode(10, values); - assertThat(encoded).containsExactly(new byte[] {0, 2, 2, 2, 0, (byte) 0xf0, 7}); + assertThat(encoded).containsExactly(new byte[] {6, 0, 2, 2, 2, 0, (byte) 0xf0, 7}); ByteBuffer data = ByteBuffer.allocate(encoded.length + 4); data.position(2); data.put(encoded); data.limit(data.position()).position(2); - DeltaVarintCodec.Reader reader = new DeltaVarintCodec.Reader(data, values.length, 10, 1024); + DeltaVarintCodec.Reader reader = new DeltaVarintCodec.Reader(data, 10, 1024); + assertThat(reader.count()).isEqualTo(values.length); for (long value : values) { assertThat(reader.hasNext()).isTrue(); assertThat(reader.next()).isEqualTo(value); @@ -54,25 +55,23 @@ void fixedBytesAndSlicedBuffer() throws Exception { @Test void longBoundariesAndEmptySequence() throws Exception { byte[] encoded = encode(0, Long.MAX_VALUE); - assertThat(encoded).hasSize(9); - assertThat( - new DeltaVarintCodec.Reader(ByteBuffer.wrap(encoded), 1, 0, Long.MAX_VALUE) - .next()) + assertThat(encoded).hasSize(10); + assertThat(new DeltaVarintCodec.Reader(ByteBuffer.wrap(encoded), 0, Long.MAX_VALUE).next()) .isEqualTo(Long.MAX_VALUE); assertThat(encode(Long.MAX_VALUE - 1, Long.MAX_VALUE, Long.MAX_VALUE)) - .containsExactly(new byte[] {1, 0}); + .containsExactly(new byte[] {2, 1, 0}); DeltaVarintCodec.Reader empty = - new DeltaVarintCodec.Reader(ByteBuffer.allocate(0), 0, 0, 0); + new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {0}), 0, 0); assertThat(empty.hasNext()).isFalse(); assertThatThrownBy(empty::next).isInstanceOf(IOException.class); - assertThat(encode(0)).isEmpty(); + assertThat(encode(0)).containsExactly((byte) 0); } @Test void stopsBeforeUnusedMalformedData() throws Exception { DeltaVarintCodec.Reader reader = new DeltaVarintCodec.Reader( - ByteBuffer.wrap(new byte[] {0, (byte) 0x80}), 2, 10, 100); + ByteBuffer.wrap(new byte[] {2, 0, (byte) 0x80}), 10, 100); assertThat(reader.next()).isEqualTo(10); assertThat(reader.hasNext()).isTrue(); assertThatThrownBy(reader::next).isInstanceOf(IOException.class); @@ -80,26 +79,25 @@ void stopsBeforeUnusedMalformedData() throws Exception { @Test void rejectsInvalidCountsBoundsAndOrdering() throws Exception { - assertThatThrownBy(() -> new DeltaVarintCodec.Reader(ByteBuffer.allocate(0), -1, 0, 1)) + assertThatThrownBy(() -> new DeltaVarintCodec.Reader(ByteBuffer.allocate(0), 0, 1)) .isInstanceOf(IOException.class); assertThatThrownBy( - () -> new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {0}), 2, 0, 1)) + () -> new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {2, 0}), 0, 1)) .isInstanceOf(IOException.class); assertThatThrownBy( - () -> new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {0}), 0, 0, 1)) + () -> + new DeltaVarintCodec.Reader( + ByteBuffer.wrap(new byte[] {(byte) 0x81, 0, 0}), 0, 1)) .isInstanceOf(IOException.class); - assertThatThrownBy(() -> new DeltaVarintCodec.Reader(ByteBuffer.allocate(0), 0, 2, 1)) + assertThatThrownBy(() -> new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {0}), 2, 1)) .isInstanceOf(IOException.class); - assertThatThrownBy(() -> new DeltaVarintCodec.Writer(new DataOutputSerializer(8), -1)) + assertThatThrownBy(() -> new DeltaVarintCodec.Writer(new DataOutputSerializer(8), 0, -1)) .isInstanceOf(IllegalArgumentException.class); assertThatThrownBy(() -> encode(10, 9)).isInstanceOf(IOException.class); assertThatThrownBy(() -> encode(0, 2, 1)).isInstanceOf(IOException.class); - DeltaVarintCodec.Reader trailing = - new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {1, 2}), 1, 0, 10); - assertThatThrownBy(trailing::next).isInstanceOf(IOException.class); DeltaVarintCodec.Reader overflow = new DeltaVarintCodec.Reader( - ByteBuffer.wrap(encode(0, Long.MAX_VALUE)), 1, 1, Long.MAX_VALUE); + ByteBuffer.wrap(encode(0, Long.MAX_VALUE)), 1, Long.MAX_VALUE); assertThatThrownBy(overflow::next).isInstanceOf(IOException.class); } @@ -109,8 +107,11 @@ void rejectsMalformedVarints() throws Exception { Arrays.fill(overlong, (byte) 0x80); for (byte[] bytes : Arrays.asList(new byte[] {(byte) 0x80}, new byte[] {(byte) 0x81, 0}, overlong)) { + byte[] payload = new byte[bytes.length + 1]; + payload[0] = 1; + System.arraycopy(bytes, 0, payload, 1, bytes.length); DeltaVarintCodec.Reader reader = - new DeltaVarintCodec.Reader(ByteBuffer.wrap(bytes), 1, 0, Long.MAX_VALUE); + new DeltaVarintCodec.Reader(ByteBuffer.wrap(payload), 0, Long.MAX_VALUE); assertThatThrownBy(reader::next).isInstanceOf(IOException.class); } } @@ -127,8 +128,7 @@ void randomizedRoundTrips() throws Exception { values[i] = value; } DeltaVarintCodec.Reader reader = - new DeltaVarintCodec.Reader( - ByteBuffer.wrap(encode(base, values)), values.length, base, value); + new DeltaVarintCodec.Reader(ByteBuffer.wrap(encode(base, values)), base, value); for (long expected : values) { assertThat(reader.next()).isEqualTo(expected); } @@ -138,10 +138,88 @@ void randomizedRoundTrips() throws Exception { private static byte[] encode(long base, long... values) throws IOException { DataOutputSerializer out = new DataOutputSerializer(32); - DeltaVarintCodec.Writer writer = new DeltaVarintCodec.Writer(out, base); + DeltaVarintCodec.Writer writer = new DeltaVarintCodec.Writer(out, values.length, base); for (long value : values) { writer.write(value); } return out.getCopyOfBuffer(); } + + @Test + void concatenatedPayloadsRetainTheirOwnBoundaries() throws Exception { + byte[] first = encode(0, 1, 1, 3); + byte[] second = encode(0, 4, 8, 16); + ByteBuffer buffer = ByteBuffer.allocate(first.length + second.length); + buffer.put(first).put(second).flip(); + DeltaVarintCodec.Reader reader = new DeltaVarintCodec.Reader(buffer, 0, 3); + assertThat(reader.count()).isEqualTo(3); + assertThat(reader.next()).isEqualTo(1); + assertThat(reader.next()).isEqualTo(1); + assertThat(reader.next()).isEqualTo(3); + assertThat(reader.hasNext()).isFalse(); + assertThat(buffer.position()).isEqualTo(first.length); + reader = new DeltaVarintCodec.Reader(buffer, 0, 16); + assertThat(reader.next()).isEqualTo(4); + assertThat(reader.next()).isEqualTo(8); + assertThat(reader.next()).isEqualTo(16); + assertThat(buffer.hasRemaining()).isFalse(); + } + + @Test + void signedIntDeltasPreserveDecreasingTotals() throws Exception { + DataOutputSerializer out = new DataOutputSerializer(32); + DeltaVarintCodec.Writer writer = new DeltaVarintCodec.Writer(out, 3, 0, true); + for (int value : new int[] {4, 8, 4}) { + writer.write(value); + } + assertThat(out.getCopyOfBuffer()).containsExactly(new byte[] {3, 8, 8, 7}); + DeltaVarintCodec.Reader reader = + new DeltaVarintCodec.Reader(ByteBuffer.wrap(out.getCopyOfBuffer()), 0, 8, true); + assertThat(reader.next()).isEqualTo(4); + assertThat(reader.next()).isEqualTo(8); + assertThat(reader.next()).isEqualTo(4); + assertThat(reader.hasNext()).isFalse(); + assertThatThrownBy(() -> writer.write(4)).isInstanceOf(IOException.class); + } + + @Test + void signedIntBoundariesAndRandomSequences() throws Exception { + Random random = new Random(20260916); + for (int trial = 0; trial < 100; trial++) { + int[] values = new int[128]; + for (int i = 0; i < values.length; i++) { + values[i] = random.nextInt(Integer.MAX_VALUE); + } + values[0] = Integer.MAX_VALUE; + values[1] = 0; + values[2] = Integer.MAX_VALUE; + DataOutputSerializer out = new DataOutputSerializer(32); + DeltaVarintCodec.Writer writer = + new DeltaVarintCodec.Writer(out, values.length, 0, true); + for (int value : values) { + writer.write(value); + } + ByteBuffer buffer = ByteBuffer.wrap(out.getCopyOfBuffer()); + DeltaVarintCodec.Reader reader = + new DeltaVarintCodec.Reader(buffer, 0, Integer.MAX_VALUE, true); + for (int value : values) { + assertThat(reader.next()).isEqualTo(value); + } + assertThat(buffer.hasRemaining()).isFalse(); + } + DeltaVarintCodec.Reader negative = + new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {1, 1}), 0, 10, true); + assertThatThrownBy(negative::next).isInstanceOf(IOException.class); + DeltaVarintCodec.Reader tooLarge = + new DeltaVarintCodec.Reader(ByteBuffer.wrap(new byte[] {1, 22}), 0, 10, true); + assertThatThrownBy(tooLarge::next).isInstanceOf(IOException.class); + assertThatThrownBy( + () -> + new DeltaVarintCodec.Writer( + new DataOutputSerializer(8), + 0, + (long) Integer.MAX_VALUE + 1, + true)) + .isInstanceOf(IllegalArgumentException.class); + } } diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index a01b5bbd20c7..6fc39844f62a 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -31,6 +31,7 @@ import org.apache.paimon.utils.RowRangeIndex; import org.apache.paimon.utils.SegmentsCache; import org.apache.paimon.utils.SerializationUtils; +import org.apache.paimon.utils.VarLengthIntUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -58,6 +59,8 @@ import java.util.TreeSet; import java.util.function.BiPredicate; +import static org.apache.paimon.utils.VarLengthIntUtils.encodeLong; + /** Independently usable partition, row-id and bucket coverage for each manifest block. */ public final class ManifestSidecar { @@ -65,8 +68,8 @@ public final class ManifestSidecar { private static final Logger LOG = LoggerFactory.getLogger(ManifestSidecar.class); private static final int MAGIC = 0x504d5343; private static final int FORMAT_VERSION = 1; - private static final int HEADER_BYTES = 24; - private static final int BLOCK_BYTES = 27; + private static final int MIN_HEADER_BYTES = 29; + private static final int MIN_BLOCK_BYTES = 6; private static final byte[] EMPTY = new byte[0]; private static final int DIGEST_BYTES = 32; private static final int SIDECAR_READ_BUFFER_BYTES = 1024 * 1024; @@ -92,21 +95,6 @@ public static String fileName(ManifestFileMeta manifest) { return null; } - /** Read/write switches and optional payloads. Partition coverage is always enabled. */ - public static final class Settings { - public final boolean write; - public final boolean read; - public final boolean rowIdEnabled; - public final boolean bucketEnabled; - - public Settings(boolean write, boolean read, boolean rowIdEnabled, boolean bucketEnabled) { - this.write = write; - this.read = read; - this.rowIdEnabled = rowIdEnabled; - this.bucketEnabled = bucketEnabled; - } - } - /** Original file offset/length and zero-based manifest entry ordinal, not table row id. */ public static final class Block { public final long offset; @@ -139,7 +127,8 @@ public List blocks() { /** Builds a complete block directory with independently available coverage. */ public static final class Builder { - private final Settings settings; + private final boolean rowIdEnabled; + private final boolean bucketEnabled; private final byte[] header; private final TreeMap ranges = new TreeMap<>(); private final Map dictionary = new LinkedHashMap<>(); @@ -154,8 +143,9 @@ public static final class Builder { private boolean partitionAvailable; private boolean bucketAvailable; - public Builder(Settings settings, byte[] header) { - this.settings = settings; + public Builder(byte[] header, boolean rowIdEnabled, boolean bucketEnabled) { + this.rowIdEnabled = rowIdEnabled; + this.bucketEnabled = bucketEnabled; this.header = Objects.requireNonNull(header); nextOffset = header.length; } @@ -164,9 +154,9 @@ public void beginBlock(long offset, long length, long records) throws IOExceptio require(current == null && offset == nextOffset && length > 0 && records > 0); current = new Block(offset, length, nextRecord, records); entriesInBlock = 0; - rowAvailable = settings.rowIdEnabled; + rowAvailable = rowIdEnabled; partitionAvailable = true; - bucketAvailable = settings.bucketEnabled; + bucketAvailable = bucketEnabled; ranges.clear(); partitionIds.clear(); bucketPairs.clear(); @@ -257,9 +247,7 @@ public void endBlock() throws IOException { ? encodeValues(partitionIds, partitionIds.size()) : EMPTY, rowAvailable ? encodeRanges() : EMPTY, - bucketAvailable - ? encodeValues(bucketPairs, bucketPairs.size()) - : EMPTY)); + bucketAvailable ? encodeBuckets() : EMPTY)); nextOffset = Math.addExact(current.offset, current.length); nextRecord = Math.addExact(current.firstRecord, current.recordCount); ranges.clear(); @@ -273,11 +261,11 @@ private byte[] encodeRanges() throws IOException { DataOutputStream out = new DataOutputStream(buffer); long min = ranges.firstKey(); long max = ranges.lastEntry().getValue(); - out.writeInt(ranges.size()); out.writeLong(min); - out.writeLong(max - min); + out.writeLong(max); // The envelope supplies the first start and last end. Encode only interior endpoints. - DeltaVarintCodec.Writer encoder = new DeltaVarintCodec.Writer(out, min); + DeltaVarintCodec.Writer encoder = + new DeltaVarintCodec.Writer(out, Math.multiplyExact(ranges.size() - 1, 2), min); int index = 0; for (Map.Entry range : ranges.entrySet()) { if (index > 0) { @@ -290,26 +278,40 @@ private byte[] encodeRanges() throws IOException { return buffer.toByteArray(); } + private byte[] encodeBuckets() throws IOException { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + DataOutputStream out = new DataOutputStream(buffer); + DeltaVarintCodec.Writer buckets = + new DeltaVarintCodec.Writer(out, bucketPairs.size(), 0); + for (long pair : bucketPairs) { + buckets.write(pair >>> 32); + } + DeltaVarintCodec.Writer totals = + new DeltaVarintCodec.Writer(out, bucketPairs.size(), 0, true); + for (long pair : bucketPairs) { + totals.write((int) pair); + } + return buffer.toByteArray(); + } + public byte[] serialize(long fileSize, long entryCount) throws IOException { require(current == null && nextOffset == fileSize && nextRecord == entryCount); ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); out.writeInt(MAGIC); - out.writeInt(FORMAT_VERSION); - out.writeLong(fileSize); - out.writeLong(entryCount); - out.writeInt(header.length); + encodeLong(out, FORMAT_VERSION); + encodeLong(out, header.length); out.write(header); - out.writeInt(dictionary.size()); + encodeLong(out, dictionary.size()); for (ByteBuffer bytes : dictionary.keySet()) { - out.writeInt(bytes.remaining()); + encodeLong(out, bytes.remaining()); out.write(bytes.array()); } - out.writeInt(blocks.size()); + encodeLong(out, blocks.size()); for (IndexedBlock block : blocks) { - out.writeLong(block.block.offset); - out.writeLong(block.block.length); - out.writeLong(block.block.recordCount); + encodeLong(out, block.block.offset); + encodeLong(out, block.block.length); + encodeLong(out, block.block.recordCount); writePayload(out, block.partitions); writePayload(out, block.rowIds); writePayload(out, block.buckets); @@ -322,8 +324,7 @@ private static byte[] encodeValues(Iterable values, int count) throws IOException { ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); - out.writeInt(count); - DeltaVarintCodec.Writer encoder = new DeltaVarintCodec.Writer(out, 0); + DeltaVarintCodec.Writer encoder = new DeltaVarintCodec.Writer(out, count, 0); for (Number value : values) { encoder.write(value.longValue()); } @@ -333,7 +334,7 @@ private static byte[] encodeValues(Iterable values, int count) private static void writePayload(DataOutputStream out, byte[] payload) throws IOException { out.writeByte(payload.length == 0 ? 0 : 1); if (payload.length > 0) { - out.writeInt(payload.length); + encodeLong(out, payload.length); out.write(payload); } } @@ -363,17 +364,19 @@ private static ProjectedManifestEntry.Projection createBlockIndexProjection() { } /** - * Rebuild from the final physical blocks, including raw-copy and encoded rewrite paths. Returns - * null without accessing files when sidecar writing is disabled. + * Rebuild from the final physical blocks, including raw-copy and encoded rewrite paths. Callers + * decide whether to build a sidecar. Partition coverage is always generated. */ - @Nullable - public static byte[] build(FileIO io, Path path, long size, long records, Settings settings) + public static byte[] build( + FileIO io, + Path path, + long size, + long records, + boolean rowIdEnabled, + boolean bucketEnabled) throws IOException { - if (!settings.write) { - return null; - } try (ManifestAvroReader reader = new ManifestAvroReader(io.newInputStream(path))) { - Builder builder = new Builder(settings, reader.headerBytes()); + Builder builder = new Builder(reader.headerBytes(), rowIdEnabled, bucketEnabled); ProjectedManifestEntry.Projection projection = BLOCK_INDEX_PROJECTION; ProjectedManifestEntry entry = projection.createEntry(); while (reader.hasNext()) { @@ -383,11 +386,11 @@ public static byte[] build(FileIO io, Path path, long size, long records, Settin while (rows.hasNext()) { entry.replace(rows.next()); builder.add( - settings.rowIdEnabled ? entry.file().firstRowId() : null, - settings.rowIdEnabled ? entry.file().rowCount() : 0, + rowIdEnabled ? entry.file().firstRowId() : null, + rowIdEnabled ? entry.file().rowCount() : 0, entry.partitionBytes(), - settings.bucketEnabled ? entry.bucket() : null, - settings.bucketEnabled ? entry.totalBuckets() : null); + bucketEnabled ? entry.bucket() : null, + bucketEnabled ? entry.totalBuckets() : null); } builder.endBlock(); } @@ -425,29 +428,28 @@ public static Selection select( @Nullable RowType partitionType, @Nullable BiPredicate bucketFilter) throws IOException { - require(data.length >= HEADER_BYTES + DIGEST_BYTES + 12 + 21); + require(data.length >= MIN_HEADER_BYTES + DIGEST_BYTES); int limit = data.length - DIGEST_BYTES; require( MessageDigest.isEqual( digest(data, limit), Arrays.copyOfRange(data, limit, data.length))); ByteBuffer in = ByteBuffer.wrap(data, 0, limit).slice(); require(in.getInt() == MAGIC); - require(in.getInt() == FORMAT_VERSION); - require(in.getLong() == manifest.fileSize()); + require(readInt(in) == FORMAT_VERSION); long entries = Math.addExact(manifest.numAddedFiles(), manifest.numDeletedFiles()); - require(in.getLong() == entries); - int headerLength = in.getInt(); - require(headerLength >= 21 && headerLength <= in.remaining() - 8); + require(entries >= 0); + int headerLength = readInt(in); + require(headerLength >= 21 && headerLength <= in.remaining() - 2); + require(headerLength <= manifest.fileSize()); byte[] header = new byte[headerLength]; in.get(header); require(header[0] == 'O' && header[1] == 'b' && header[2] == 'j' && header[3] == 1); - int partitions = in.getInt(); - require(partitions >= 0 && partitions <= in.remaining() / 16); + int partitions = readInt(in); + require(partitions <= in.remaining() / 13); boolean[] matches = partitionFilter == null ? null : new boolean[partitions]; Set unique = new java.util.HashSet<>(); for (int id = 0; id < partitions; id++) { - require(in.remaining() >= 4); - int length = in.getInt(); + int length = readInt(in); require(length >= 12 && length <= in.remaining()); ByteBuffer encoded = in.slice(); encoded.limit(length); @@ -463,52 +465,62 @@ public static Selection select( } in.position(in.position() + length); } - require(in.remaining() >= 4); - int count = in.getInt(); - require(count >= 0 && count <= in.remaining() / BLOCK_BYTES); + int count = readInt(in); + require(count <= in.remaining() / MIN_BLOCK_BYTES); long nextOffset = headerLength; long firstRecord = 0; List selected = new ArrayList<>(); for (int i = 0; i < count; i++) { - require(in.remaining() >= BLOCK_BYTES); - long offset = in.getLong(); - long length = in.getLong(); - long records = in.getLong(); + require(in.remaining() >= MIN_BLOCK_BYTES); + long offset = VarLengthIntUtils.decodeLong(in); + long length = VarLengthIntUtils.decodeLong(in); + long records = VarLengthIntUtils.decodeLong(in); require(offset == nextOffset && length > 0 && length <= manifest.fileSize() - offset); require(records > 0 && records <= entries - firstRecord); - Payload partitionPayload = payload(in, records); - Payload rowPayload = payload(in, records); - Payload bucketPayload = payload(in, records); - require(partitionPayload == null || partitionPayload.count <= partitions); - require( - rowPayload == null - || rowPayload.data.remaining() - >= 2 * Long.BYTES + 2L * (rowPayload.count - 1)); + ByteBuffer partitionPayload = payload(in); + ByteBuffer rowPayload = payload(in); + ByteBuffer bucketPayload = payload(in); + DeltaVarintCodec.Reader ids = null; + if (partitionPayload != null) { + ids = new DeltaVarintCodec.Reader(partitionPayload, 0, partitions - 1L); + require(ids.count() > 0 && ids.count() <= records && ids.count() <= partitions); + } + long min = 0; + long max = 0; + DeltaVarintCodec.Reader endpoints = null; + if (rowPayload != null) { + require(rowPayload.remaining() >= 2 * Long.BYTES + 1); + min = rowPayload.getLong(); + max = rowPayload.getLong(); + endpoints = new DeltaVarintCodec.Reader(rowPayload, min, max); + require(endpoints.count() % 2 == 0 && endpoints.count() / 2L < records); + require(endpoints.count() != 0 || !rowPayload.hasRemaining()); + } + if (bucketPayload != null) { + ByteBuffer prefix = bucketPayload.duplicate(); + int pairs = readInt(prefix); + require(pairs > 0 && pairs <= records && 2L * pairs + 1 <= prefix.remaining()); + } long blockFirstRecord = firstRecord; nextOffset = offset + length; firstRecord += records; if (query != null && rowPayload != null) { - long min = rowPayload.data.getLong(); - long span = rowPayload.data.getLong(); - require(min >= 0 && span >= 0 && span <= Long.MAX_VALUE - min); - long max = min + span; - DeltaVarintCodec.Reader endpoints = - new DeltaVarintCodec.Reader( - rowPayload.data, 2L * (rowPayload.count - 1), min, max); if (!query.intersects(min, max)) { continue; } - boolean rowHit = rowPayload.count == 1; + int rangeCount = endpoints.count() / 2 + 1; + boolean rowHit = rangeCount == 1; long start = min; - for (int range = 0; !rowHit && range < rowPayload.count; range++) { - long end = range + 1 == rowPayload.count ? max : endpoints.next(); + for (int range = 0; !rowHit && range < rangeCount; range++) { + long end = range + 1 == rangeCount ? max : endpoints.next(); require(end >= start); rowHit = query.intersects(start, end); - if (!rowHit && range + 1 < rowPayload.count) { + if (!rowHit && range + 1 < rangeCount) { start = endpoints.next(); require(start > end); } + require(endpoints.hasNext() || !rowPayload.hasRemaining()); } if (!rowHit) { continue; @@ -516,14 +528,12 @@ public static Selection select( } if (partitionFilter != null && partitionPayload != null) { - DeltaVarintCodec.Reader ids = - new DeltaVarintCodec.Reader( - partitionPayload.data, partitionPayload.count, 0, partitions - 1L); boolean partitionHit = false; long previous = -1; while (!partitionHit && ids.hasNext()) { long id = ids.next(); require(id > previous); + require(ids.hasNext() || !partitionPayload.hasRemaining()); previous = id; partitionHit = matches[(int) id]; } @@ -533,17 +543,35 @@ public static Selection select( } if (bucketFilter != null && bucketPayload != null) { - DeltaVarintCodec.Reader pairs = - new DeltaVarintCodec.Reader( - bucketPayload.data, bucketPayload.count, 0, Long.MAX_VALUE); + // Locate the second count-prefixed sequence without allocating value arrays. + ByteBuffer totalsData = bucketPayload.duplicate(); + DeltaVarintCodec.Reader directory = + new DeltaVarintCodec.Reader(totalsData, 0, Integer.MAX_VALUE); + while (directory.hasNext()) { + directory.next(); + } + bucketPayload.limit(totalsData.position()); + DeltaVarintCodec.Reader buckets = + new DeltaVarintCodec.Reader(bucketPayload, 0, Integer.MAX_VALUE); + DeltaVarintCodec.Reader totals = + new DeltaVarintCodec.Reader(totalsData, 0, Integer.MAX_VALUE, true); + require(totals.count() == buckets.count()); boolean bucketHit = false; - long previous = -1; - while (!bucketHit && pairs.hasNext()) { - long pair = pairs.next(); - int bucket = (int) (pair >>> 32); - int totalBuckets = (int) pair; - require(totalBuckets > bucket && pair > previous); - previous = pair; + int previousBucket = -1; + int previousTotal = -1; + while (!bucketHit && buckets.hasNext()) { + int bucket = (int) buckets.next(); + int totalBuckets = (int) totals.next(); + require(totalBuckets > bucket); + require( + bucket > previousBucket + || (bucket == previousBucket && totalBuckets > previousTotal)); + require( + buckets.hasNext() + || (!bucketPayload.hasRemaining() + && !totalsData.hasRemaining())); + previousBucket = bucket; + previousTotal = totalBuckets; bucketHit = bucketFilter.test(bucket, totalBuckets); } if (!bucketHit) { @@ -556,48 +584,36 @@ public static Selection select( return new Selection(header, selected); } - private static final class Payload { - private final int count; - private final ByteBuffer data; - - private Payload(int count, ByteBuffer data) { - this.count = count; - this.data = data; - } - } - - /** Reads framing and counts without expanding the compressed contents. */ + /** Reads framing without expanding the compressed contents. */ @Nullable - private static Payload payload(ByteBuffer in, long records) throws IOException { + private static ByteBuffer payload(ByteBuffer in) throws IOException { require(in.hasRemaining()); int encoding = Byte.toUnsignedInt(in.get()); if (encoding == 0) { return null; } - require(in.remaining() >= Integer.BYTES); - int length = in.getInt(); - require(length >= 0 && length <= in.remaining()); + int length = readInt(in); + require(length <= in.remaining()); ByteBuffer result = in.slice(); result.limit(length); in.position(in.position() + length); if (encoding != 1) { return null; } - require(result.remaining() >= Integer.BYTES); - int count = result.getInt(); - require(count > 0 && count <= records && count <= result.remaining()); - return new Payload(count, result); + return result; + } + + private static int readInt(ByteBuffer in) throws IOException { + long value = VarLengthIntUtils.decodeLong(in); + require(value <= Integer.MAX_VALUE); + return (int) value; } - /** Reads the complete sidecar when enabled. Null means read the original manifest. */ + /** Reads the complete sidecar. Null means read the original manifest. */ @Nullable public static Selection read( - FileIO io, - Path path, - ManifestFileMeta manifest, - Settings settings, - @Nullable RowRangeIndex query) { - return read(io, path, manifest, settings, query, null, null); + FileIO io, Path path, ManifestFileMeta manifest, @Nullable RowRangeIndex query) { + return read(io, path, manifest, query, null, null); } @Nullable @@ -605,12 +621,10 @@ public static Selection read( FileIO io, Path path, ManifestFileMeta manifest, - Settings settings, @Nullable RowRangeIndex query, @Nullable PartitionPredicate partitionFilter, @Nullable RowType partitionType) { - return read( - io, path, manifest, settings, query, partitionFilter, partitionType, null, null); + return read(io, path, manifest, query, partitionFilter, partitionType, null, null); } @Nullable @@ -618,15 +632,11 @@ public static Selection read( FileIO io, Path path, ManifestFileMeta manifest, - Settings settings, @Nullable RowRangeIndex query, @Nullable PartitionPredicate partitionFilter, @Nullable RowType partitionType, @Nullable BiPredicate bucketFilter, @Nullable SegmentsCache cache) { - if (!settings.read) { - return null; - } String sidecarFileName = fileName(manifest); if (sidecarFileName == null) { return null; diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java index 5df35a422314..f412c14f2f74 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java @@ -28,6 +28,7 @@ import org.apache.paimon.utils.Range; import org.apache.paimon.utils.RowRangeIndex; import org.apache.paimon.utils.SerializationUtils; +import org.apache.paimon.utils.VarLengthIntUtils; import org.junit.jupiter.api.Test; @@ -59,8 +60,6 @@ class ManifestBlockIndexTest { private final RowType type = RowType.of(DataTypes.INT(), DataTypes.STRING()); - private final ManifestSidecar.Settings defaults = - new ManifestSidecar.Settings(true, true, true, true); private byte[] fixture(String field) throws IOException { Properties p = new Properties(); @@ -106,16 +105,15 @@ void partitionCoverageIsAlwaysGenerated() throws Exception { for (int mask = 0; mask < 4; mask++) { boolean rowIdEnabled = (mask & 1) != 0; boolean bucketEnabled = (mask & 2) != 0; - ManifestSidecar.Settings settings = - new ManifestSidecar.Settings(true, true, rowIdEnabled, bucketEnabled); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = + new ManifestSidecar.Builder(header, rowIdEnabled, bucketEnabled); for (int block = 0; block < 2; block++) { builder.beginBlock(header.length + block * 100L, 100, 1); builder.add(100L + block * 100L, 10, partition(7 + block, "p"), 1, 4); builder.endBlock(); } byte[] data = builder.serialize(header.length + 200, 2); - assertThat(ByteBuffer.wrap(data).getInt(28 + header.length)).isEqualTo(2); + assertThat(partitionCount(data)).isEqualTo(2); for (int[] position : positions(data)) { assertThat(data[position[1]]).isEqualTo((byte) 1); assertThat(data[position[2]]).isEqualTo((byte) (rowIdEnabled ? 1 : 0)); @@ -153,7 +151,7 @@ void jointGoldenPreservesTuplesNullsAndDerivedOrdinals() throws Exception { assertThat(a).isEqualTo(fixture("partitionA")); assertThat(b).isEqualTo(fixture("partitionB")); byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 3); builder.add(0L, 10, a); builder.add(5L, 5, a); @@ -186,9 +184,8 @@ void jointGoldenPreservesTuplesNullsAndDerivedOrdinals() throws Exception { @Test void unavailableDimensionsAreIndependentAndDoNotPoisonLaterBlocks() throws Exception { - ManifestSidecar.Settings settings = defaults; byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 1); builder.add(null, 10, partition(7, "left")); builder.endBlock(); @@ -211,21 +208,33 @@ void unavailableDimensionsAreIndependentAndDoNotPoisonLaterBlocks() throws Excep .containsExactly(1L); } - private List positions(byte[] data) { + private int partitionCount(byte[] data) throws IOException { ByteBuffer in = ByteBuffer.wrap(data); - in.position(24); - int header = in.getInt(); + in.position(4); + VarLengthIntUtils.decodeLong(in); + int header = (int) VarLengthIntUtils.decodeLong(in); in.position(in.position() + header); - int partitions = in.getInt(); + return (int) VarLengthIntUtils.decodeLong(in); + } + + private List positions(byte[] data) throws IOException { + ByteBuffer in = ByteBuffer.wrap(data); + in.position(4); + VarLengthIntUtils.decodeLong(in); + int header = (int) VarLengthIntUtils.decodeLong(in); + in.position(in.position() + header); + int partitions = (int) VarLengthIntUtils.decodeLong(in); for (int i = 0; i < partitions; i++) { - int length = in.getInt(); + int length = (int) VarLengthIntUtils.decodeLong(in); in.position(in.position() + length); } - int blocks = in.getInt(); + int blocks = (int) VarLengthIntUtils.decodeLong(in); List result = new ArrayList<>(); for (int i = 0; i < blocks; i++) { int block = in.position(); - in.position(block + 24); + VarLengthIntUtils.decodeLong(in); + VarLengthIntUtils.decodeLong(in); + VarLengthIntUtils.decodeLong(in); int partition = skipPayload(in); int row = skipPayload(in); int bucket = skipPayload(in); @@ -234,10 +243,10 @@ private List positions(byte[] data) { return result; } - private int skipPayload(ByteBuffer in) { + private int skipPayload(ByteBuffer in) throws IOException { int position = in.position(); if (in.get() != 0) { - int length = in.getInt(); + int length = (int) VarLengthIntUtils.decodeLong(in); in.position(in.position() + length); } return position; @@ -272,7 +281,7 @@ void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { byte[] header = fixture("avroHeader"); byte[] a = partition(7, "left"); byte[] b = partition(9, null); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 3); builder.add(0L, 10, a, 1, 4); builder.add(5L, 5, a, 1, 4); @@ -318,9 +327,72 @@ void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { } } + @Test + void splitBucketArraysPreserveDecreasingTotalsAndRescaling() throws Exception { + byte[] header = fixture("avroHeader"); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, false, true); + builder.beginBlock(header.length, 100, 5); + for (int[] pair : new int[][] {{3, 4}, {2, 8}, {0, Integer.MAX_VALUE}, {2, 4}, {2, 4}}) { + builder.add(null, 0, partition(7, "left"), pair[0], pair[1]); + } + builder.endBlock(); + byte[] data = builder.serialize(header.length + 100, 5); + ManifestFileMeta meta = meta("m", header.length + 100, 5); + List pairs = new ArrayList<>(); + assertThat( + ManifestSidecar.select( + data, + meta, + null, + null, + type, + (bucket, total) -> { + pairs.add(bucket + ":" + total); + return false; + }) + .blocks()) + .isEmpty(); + assertThat(pairs).containsExactly("0:2147483647", "2:4", "2:8", "3:4"); + assertThat( + ManifestSidecar.select( + data, + meta, + null, + null, + type, + (bucket, total) -> bucket == 2 && total == 4) + .blocks()) + .hasSize(1); + assertThat( + ManifestSidecar.select( + data, + meta, + null, + null, + type, + (bucket, total) -> bucket == 3 && total == 8) + .blocks()) + .isEmpty(); + } + + @Test + void mismatchedBucketCountsAndNegativeTotalsAreRejected() throws Exception { + for (byte[] payload : + Arrays.asList( + new byte[] {2, 1, 0, 1, 8, 8}, // Two buckets, one total, extra bytes. + bucketPayload(2, new long[] {1, 0}, 1, 0), // ZigZag -1 from zero. + bucketPayload(2, new long[] {1, 0}, 8, 0))) { // Repeated pair. + byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, payload); + assertThatThrownBy( + () -> + ManifestSidecar.select( + data, goldenMeta(), null, null, type, bucketFilter(99))) + .isInstanceOf(IOException.class); + } + } + @Test void unknownOrInvalidBucketPayloadIsUnavailable() throws Exception { - ManifestSidecar.Settings settings = defaults; byte[] header = fixture("avroHeader"); for (Integer[] pair : Arrays.asList( @@ -328,7 +400,7 @@ void unknownOrInvalidBucketPayloadIsUnavailable() throws Exception { new Integer[] {-1, 4}, new Integer[] {4, 4}, new Integer[] {0, 0})) { - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 2); builder.add(100L, 10, partition(7, "left"), 1, 4); builder.add(200L, 10, partition(7, "left"), pair[0], pair[1]); @@ -357,7 +429,7 @@ data, meta, query(999), null, type, bucketFilter(99)) @Test void largePayloadsKeepExactCoverage() throws Exception { byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); int blocks = 33; int entriesPerBlock = 4097; int entries = blocks * entriesPerBlock; @@ -391,7 +463,7 @@ data, meta, null, null, type, bucketFilter(entriesPerBlock)) @Test void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Exception { byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); for (int mask = 0; mask < 8; mask++) { builder.beginBlock(header.length + mask * 100L, 100, 1); builder.add( @@ -404,7 +476,7 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti } byte[] data = builder.serialize(header.length + 800, 8); List positions = positions(data); - int[] presentSizes = {10, 25, 10}; + int[] presentSizes = {4, 19, 6}; for (int mask = 0; mask < 8; mask++) { for (int dimension = 0; dimension < 3; dimension++) { int start = positions.get(mask)[dimension + 1]; @@ -436,7 +508,7 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti @Test void deltaVarintsCompressSortedPayloadsWithoutCoarseningRowIds() throws Exception { byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); int count = 10000; builder.beginBlock(header.length, 100, count); for (int i = count - 1; i >= 0; i--) { @@ -445,9 +517,9 @@ void deltaVarintsCompressSortedPayloadsWithoutCoarseningRowIds() throws Exceptio builder.endBlock(); byte[] data = builder.serialize(header.length + 100, count); int[] block = positions(data).get(0); - assertThat(ByteBuffer.wrap(data).getInt(block[1] + 1)).isEqualTo(4 + count); - assertThat(ByteBuffer.wrap(data).getInt(block[2] + 1)).isEqualTo(20 + 2 * (count - 1)); - assertThat(ByteBuffer.wrap(data).getInt(block[3] + 1)).isEqualTo(4 + 2 + 5 * (count - 1)); + assertThat(payloadLength(data, block[1])).isEqualTo(2 + count); + assertThat(payloadLength(data, block[2])).isEqualTo(16 + 3 + 2 * (count - 1)); + assertThat(payloadLength(data, block[3])).isEqualTo(2 * count + 6); ManifestFileMeta meta = meta("m", header.length + 100, count); assertThat(ManifestSidecar.select(data, meta, query(3)).blocks()).isEmpty(); assertThat( @@ -460,14 +532,12 @@ data, meta, query(4), part(9999), type, bucketFilter(9999)) @Test void unpartitionedTablesStillRecordTheEmptyPartition() throws Exception { byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = - new ManifestSidecar.Builder( - new ManifestSidecar.Settings(true, true, false, false), header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, false, false); builder.beginBlock(header.length, 100, 1); builder.add(null, 0, SerializationUtils.serializeBinaryRow(BinaryRow.EMPTY_ROW)); builder.endBlock(); byte[] data = builder.serialize(header.length + 100, 1); - assertThat(ByteBuffer.wrap(data).getInt(28 + header.length)).isEqualTo(1); + assertThat(partitionCount(data)).isEqualTo(1); int[] block = positions(data).get(0); assertThat(data[block[1]]).isEqualTo((byte) 1); assertThat(data[block[2]]).isZero(); @@ -487,7 +557,7 @@ void unpartitionedTablesStillRecordTheEmptyPartition() throws Exception { void rowMissSkipsPartitionAndBucketDecoding() throws Exception { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 999, 1)); - data = replacePayload(data, 0, 3, compressedPayload(2, 0, 0)); + data = replacePayload(data, 0, 3, bucketPayload(2, new long[] {0, 0}, 0, 0)); BiPredicate buckets = mock(BiPredicate.class); assertThat( ManifestSidecar.select( @@ -499,7 +569,12 @@ data, goldenMeta(), query(15), part(7), type, buckets) @Test void partitionMissSkipsBucketDecodingWithOrWithoutRowQuery() throws Exception { - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, compressedPayload(2, 0, 0)); + byte[] data = + replacePayload( + fixture("indexWithBuckets"), + 0, + 3, + bucketPayload(2, new long[] {0, 0}, 0, 0)); for (RowRangeIndex rows : Arrays.asList(null, query(0))) { BiPredicate buckets = mock(BiPredicate.class); assertThat( @@ -545,7 +620,7 @@ partitions, goldenMeta(), query(0), part(99), type)) fixture("indexWithBuckets"), 0, 3, - compressedPayload(2, (1L << 32) | 4, Long.MAX_VALUE)); + bucketPayload(2, new long[] {1, 0}, 8, Long.MAX_VALUE)); assertThat( ManifestSidecar.select( buckets, @@ -568,7 +643,7 @@ partitions, goldenMeta(), query(0), part(99), type)) .isInstanceOf(IOException.class); byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(defaults, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 3); for (long first : new long[] {0, 20, 40}) { builder.add(first, 10); @@ -596,10 +671,10 @@ void malformedCompressedPayloadsFailWhenConsumed() throws Exception { rowPayload(2, 0, 24, 9, 0), // Overlapping intervals. rowPayload(2, 0, 24, Long.MAX_VALUE, 0), // Exceeds the envelope. rowPayload(2, 0, 24, 9, 11, 0), // More values than declared. - rowPayload(1, Long.MAX_VALUE, 1), // Envelope overflows. + rowPayload(1, Long.MAX_VALUE, 1), // Reversed envelope. rowPayload(1, -1, 24), rowPayload(1, 0, -1), - Arrays.copyOf(rowPayload(1, 0, 24), 19), // Truncated fixed-width envelope. + Arrays.copyOf(rowPayload(1, 0, 24), 16), // Missing endpoint count. rowPayload(1, 0, 24, 0)); // Unexpected value for a single interval. for (byte[] payload : badRows) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 2, payload); @@ -617,9 +692,9 @@ data, goldenMeta(), query(0), part(99), type)) } for (byte[] payload : Arrays.asList( - compressedPayload(2, 0, 4), - compressedPayload(2, 1L << 31, 4), - compressedPayload(2, 0, 0))) { + bucketPayload(2, new long[] {0, 4}, 2, 0), + bucketPayload(2, new long[] {1L << 31, 4}, 8, 0), + bucketPayload(2, new long[] {0, 0}, 2, 0))) { byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, payload); assertThatThrownBy( () -> @@ -646,7 +721,12 @@ void malformedDeltaVarintsFailWhenConsumed() throws Exception { overlong)) { for (int dimension = 1; dimension <= 3; dimension++) { ByteArrayOutputStream payload = new ByteArrayOutputStream(); - payload.write(dimension == 2 ? rowPayload(2, 0, 24) : compressedPayload(2)); + payload.write( + dimension == 2 + ? rowPayload(2, 0, 24) + : dimension == 3 + ? bucketPayload(2, new long[] {1, 0}) + : compressedPayload(2)); payload.write(deltas); byte[] data = replacePayload( @@ -671,7 +751,7 @@ void invalidCountFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception { List bad = Arrays.asList( new byte[0], - new byte[3], // Incomplete int count. + new byte[] {(byte) 0x80}, // Incomplete varint count. compressedPayload(-1, 1, 1), compressedPayload(0, 1, 1), compressedPayload(4, 1, 1), @@ -686,19 +766,23 @@ void invalidCountFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception { } byte[] data = fixture("indexWithBuckets"); int position = positions(data).get(0)[dimension]; - ByteBuffer.wrap(data).putInt(position + 1, Integer.MAX_VALUE); + Arrays.fill(data, position + 1, position + 6, (byte) 0xff); checksum(data); assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) .isInstanceOf(IOException.class); } byte[] missingEnvelope = replacePayload( - fixture("indexWithBuckets"), 0, 2, Arrays.copyOf(rowPayload(1, 0, 24), 19)); + fixture("indexWithBuckets"), 0, 2, Arrays.copyOf(rowPayload(1, 0, 24), 16)); assertThatThrownBy(() -> ManifestSidecar.select(missingEnvelope, goldenMeta(), null)) .isInstanceOf(IOException.class); byte[] data = fixture("indexWithBuckets"); int block = positions(data).get(0)[0]; - ByteBuffer.wrap(data).putLong(block + 16, 2); + ByteBuffer directory = ByteBuffer.wrap(data); + directory.position(block); + VarLengthIntUtils.decodeLong(directory); + VarLengthIntUtils.decodeLong(directory); + data[directory.position()] = 2; checksum(data); assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) .isInstanceOf(IOException.class); @@ -733,13 +817,15 @@ private ManifestFileMeta goldenMeta() throws Exception { private byte[] replacePayload(byte[] data, int block, int dimension, byte[] payload) throws Exception { int start = positions(data).get(block)[dimension]; - int end = - data[start] == 0 ? start + 1 : start + 5 + ByteBuffer.wrap(data).getInt(start + 1); + ByteBuffer in = ByteBuffer.wrap(data); + in.position(start); + skipPayload(in); + int end = in.position(); ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); out.write(data, 0, start); out.writeByte(1); - out.writeInt(payload.length); + VarLengthIntUtils.encodeLong(out, payload.length); out.write(payload); out.write(data, end, data.length - end); return checksum(buffer.toByteArray()); @@ -748,22 +834,36 @@ private byte[] replacePayload(byte[] data, int block, int dimension, byte[] payl private static byte[] compressedPayload(int count, long... values) throws IOException { ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); - out.writeInt(count); + out.write(deltaBytes(count)); out.write(deltaBytes(values)); return buffer.toByteArray(); } - private static byte[] rowPayload(int count, long min, long span, long... values) + private static byte[] rowPayload(int count, long min, long max, long... values) throws IOException { ByteArrayOutputStream buffer = new ByteArrayOutputStream(); DataOutputStream out = new DataOutputStream(buffer); - out.writeInt(count); out.writeLong(min); - out.writeLong(span); + out.writeLong(max); + out.write(deltaBytes(2L * (count - 1))); out.write(deltaBytes(values)); return buffer.toByteArray(); } + private static int payloadLength(byte[] data, int position) throws IOException { + ByteBuffer buffer = ByteBuffer.wrap(data); + buffer.position(position + 1); + return (int) VarLengthIntUtils.decodeLong(buffer); + } + + private static byte[] bucketPayload(int count, long[] bucketDeltas, long... totalDeltas) + throws IOException { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + buffer.write(compressedPayload(count, bucketDeltas)); + buffer.write(compressedPayload(count, totalDeltas)); + return buffer.toByteArray(); + } + private static byte[] deltaBytes(long... values) { ByteArrayOutputStream out = new ByteArrayOutputStream(); for (long value : values) { diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index 6b01fa54788a..28f7086626c0 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -74,8 +74,6 @@ class ManifestSidecarTest { @TempDir java.nio.file.Path temp; - private final ManifestSidecar.Settings settings = - new ManifestSidecar.Settings(true, true, true, true); static ManifestFileMeta meta(String name, long size, long entries) { ManifestFileMeta meta = mock(ManifestFileMeta.class); @@ -110,42 +108,12 @@ private ManifestFileMeta goldenMeta() throws IOException { @Test void emptyManifestHasACompleteSidecar() throws Exception { byte[] header = header(); - byte[] data = new ManifestSidecar.Builder(settings, header).serialize(header.length, 0); + byte[] data = new ManifestSidecar.Builder(header, true, true).serialize(header.length, 0); assertThat(ByteBuffer.wrap(data).getInt()).isEqualTo(0x504d5343); assertThat(ManifestSidecar.select(data, meta("empty", header.length, 0), null).blocks()) .isEmpty(); } - @Test - void disabledBuildDoesNotAccessFiles() throws Exception { - FileIO io = mock(FileIO.class); - assertThat( - ManifestSidecar.build( - io, - new Path(temp.toString(), "missing-manifest"), - 100, - 1, - new ManifestSidecar.Settings(false, true, true, true))) - .isNull(); - verifyNoInteractions(io); - } - - @Test - void disabledReadsDoNotAccessMetadataCacheOrFiles() { - FileIO io = mock(FileIO.class); - ManifestFileMeta manifest = mock(ManifestFileMeta.class); - SegmentsCache cache = mock(SegmentsCache.class); - Path path = new Path(temp.toString(), "missing-manifest"); - ManifestSidecar.Settings disabled = new ManifestSidecar.Settings(true, false, true, true); - assertThat(ManifestSidecar.read(io, path, manifest, disabled, null)).isNull(); - assertThat(ManifestSidecar.read(io, path, manifest, disabled, null, null, null)).isNull(); - assertThat( - ManifestSidecar.read( - io, path, manifest, disabled, null, null, null, null, cache)) - .isNull(); - verifyNoInteractions(io, manifest, cache); - } - @Test void buildAndReadSelectedBlocksFromPhysicalManifests() throws Exception { FileIO io = LocalFileIO.create(); @@ -179,7 +147,7 @@ void buildAndReadSelectedBlocksFromPhysicalManifests() throws Exception { for (ManifestFileMeta meta : Arrays.asList(source, rewritten)) { Path path = new Path(temp.toString(), meta.fileName()); byte[] data = - ManifestSidecar.build(io, path, meta.fileSize(), entries.size(), settings); + ManifestSidecar.build(io, path, meta.fileSize(), entries.size(), true, true); ManifestSidecar.Selection selected = ManifestSidecar.select( data, @@ -231,7 +199,7 @@ private ManifestAvroWriter writer(FileIO io, Path path) { @Test void crossLanguageFormatAndBlockOrdinals() throws Exception { byte[] header = header(); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 3); builder.add(0L, 10); builder.add(5L, 5); @@ -287,7 +255,7 @@ void crossLanguageFormatAndBlockOrdinals() throws Exception { @Test void minMaxSkipsExactIntersectionChecksAndHandlesOneInterval() throws Exception { byte[] header = header(); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 2); builder.add(0L, 10); builder.add(20L, 10); @@ -332,7 +300,7 @@ void singleIntervalHandlesBoundariesAndAbsentQueries() throws Exception { new Range(0, 0), new Range(42, 51), new Range(Long.MAX_VALUE, Long.MAX_VALUE))) { - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 1); builder.add(range.from, range.to - range.from + 1); builder.endBlock(); @@ -362,7 +330,7 @@ void singleIntervalHandlesBoundariesAndAbsentQueries() throws Exception { @Test void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception { byte[] header = header(); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 2); builder.add(0L, Long.MAX_VALUE); builder.add(Long.MAX_VALUE, 1); @@ -372,7 +340,7 @@ void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception assertThat(select(data, meta("m", header.length + 100, 2), Long.MAX_VALUE).blocks()) .hasSize(1); for (Long first : Arrays.asList(null, -1L, Long.MAX_VALUE)) { - builder = new ManifestSidecar.Builder(settings, header); + builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 1); builder.add(first, 2); builder.endBlock(); @@ -385,7 +353,7 @@ void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception .hasSize(1); } for (long count : new long[] {0, -1}) { - builder = new ManifestSidecar.Builder(settings, header); + builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 1); builder.add(0L, count); builder.endBlock(); @@ -397,7 +365,7 @@ void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception .blocks()) .hasSize(1); } - builder = new ManifestSidecar.Builder(settings, header); + builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 64); for (int i = 0; i < 64; i++) { builder.add(i * 10L, 1); @@ -491,8 +459,7 @@ void cachedBytesPreservePerQueryCancellation() throws Exception { assertThatThrownBy( () -> ManifestSidecar.read( - io, path, meta, settings, cancelled, null, null, null, - cache)) + io, path, meta, cancelled, null, null, null, cache)) .isInstanceOf(CancellationException.class); assertThat(cache.getIfPresents(sidecar)).isNull(); @@ -500,8 +467,7 @@ void cachedBytesPreservePerQueryCancellation() throws Exception { assertThatThrownBy( () -> ManifestSidecar.read( - io, path, meta, settings, cancelled, null, null, null, - cache)) + io, path, meta, cancelled, null, null, null, cache)) .isInstanceOf(CancellationException.class); assertThat(readCached(io, path, meta, cache).blocks()).hasSize(2); verify(io, times(2)).newInputStream(sidecar); @@ -513,7 +479,6 @@ private ManifestSidecar.Selection readCached( io, path, meta, - settings, RowRangeIndex.create(Collections.singletonList(new Range(20, 20))), null, null, @@ -528,20 +493,18 @@ void corruptMissingIncompleteAndMismatchedIndexesFallback() throws Exception { ManifestFileMeta meta = goldenMeta(); RowRangeIndex query = RowRangeIndex.create(Collections.singletonList(new Range(11, 11))); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, settings, query)) - .isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); byte[] good = golden(); for (int position : new int[] {0, 9, 11, 15, 16, 55, 63, 67, 75, good.length - 1}) { byte[] bad = good.clone(); bad[position] ^= 2; Files.write(index, bad); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, settings, query)) - .isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); } // A valid checksum cannot make an unsupported container version readable. for (int version : new int[] {0, 2, 99}) { byte[] bad = good.clone(); - ByteBuffer.wrap(bad).putInt(4, version); + bad[4] = (byte) version; byte[] hash = MessageDigest.getInstance("SHA-256") .digest(Arrays.copyOf(bad, bad.length - 32)); @@ -550,12 +513,9 @@ void corruptMissingIncompleteAndMismatchedIndexesFallback() throws Exception { .isInstanceOf(IOException.class); } Files.write(index, Arrays.copyOf(good, good.length - 1)); - assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, settings, query)) - .isNull(); + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); Files.write(index, good); - assertThat( - ManifestSidecar.read(LocalFileIO.create(), manifest, meta, settings, query) - .blocks()) + assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query).blocks()) .isEmpty(); // The sidecar is bound to physical coverage, not to a particular file name. assertThat( @@ -593,8 +553,7 @@ void ioFailuresFallBackWithoutInspectingNestedExceptions() throws Exception { suppressed)) { FileIO fileIO = mock(FileIO.class); when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); - assertThat(ManifestSidecar.read(fileIO, path, meta("m", 1, 1), settings, null)) - .isNull(); + assertThat(ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)).isNull(); assertThat(Thread.currentThread().isInterrupted()).isFalse(); } } @@ -607,10 +566,7 @@ void interruptedThreadDoesNotFallBackOnIoFailure() throws Exception { when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); try { Thread.currentThread().interrupt(); - assertThatThrownBy( - () -> - ManifestSidecar.read( - fileIO, path, meta("m", 1, 1), settings, null)) + assertThatThrownBy(() -> ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)) .isInstanceOf(java.io.UncheckedIOException.class) .hasCauseReference(failure); assertThat(Thread.currentThread().isInterrupted()).isTrue(); @@ -630,10 +586,7 @@ void uncheckedFailuresPropagateUnchanged() throws Exception { new AssertionError("error"))) { FileIO fileIO = mock(FileIO.class); when(fileIO.newInputStream(ManifestSidecar.path(path))).thenThrow(failure); - assertThatThrownBy( - () -> - ManifestSidecar.read( - fileIO, path, meta("m", 1, 1), settings, null)) + assertThatThrownBy(() -> ManifestSidecar.read(fileIO, path, meta("m", 1, 1), null)) .isSameAs(failure); } } @@ -642,7 +595,7 @@ fileIO, path, meta("m", 1, 1), settings, null)) void indexReadsUseBoundedBulkRequests() throws Exception { byte[] header = header(); for (int blockCount : new int[] {5000, 25000, 131073}) { - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); for (int blockNumber = 0; blockNumber < blockCount; blockNumber++) { builder.beginBlock(header.length + blockNumber * 100L, 100, 1); builder.add((long) blockNumber, 1); @@ -660,7 +613,6 @@ void indexReadsUseBoundedBulkRequests() throws Exception { io, path, meta, - settings, RowRangeIndex.create(Collections.singletonList(new Range(0, 0)))); assertThat(actual.blocks()).hasSize(1); assertThat(actual.blocks().get(0).offset).isEqualTo(header.length); @@ -678,7 +630,7 @@ void sidecarsLargerThanTheFormerDefaultLimitAreReadCompletely() throws Exception byte[] value = new byte[17 * 1024 * 1024]; rowWriter.writeBinary(0, value, 0, value.length); rowWriter.complete(); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 1); builder.add(0L, 1, SerializationUtils.serializeBinaryRow(partition)); builder.endBlock(); @@ -693,7 +645,6 @@ void sidecarsLargerThanTheFormerDefaultLimitAreReadCompletely() throws Exception io, path, meta("manifest-large", header.length + 100, 1), - settings, null) .blocks()) .hasSize(1); @@ -716,7 +667,6 @@ void indexShortReadsReadTheWholeFile() throws Exception { io, path, goldenMeta(), - settings, RowRangeIndex.create(Collections.singletonList(new Range(20, 20)))); assertThat(actual.blocks()) .extracting(block -> block.firstRecord) @@ -939,7 +889,7 @@ void oversizedBlocksUseBoundedReadsWithoutModifyingPreviouslyCachedBytes() throw byte[] header = header(); int cachedLength = (4 << 20) + 17; int uncachedLength = 2 * (4 << 20) + 31; - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, cachedLength, 1); builder.add(0L, 1); builder.endBlock(); @@ -989,7 +939,7 @@ void oversizedBlocksUseBoundedReadsWithoutModifyingPreviouslyCachedBytes() throw @Test void largeBlockSpansUseBoundedReads() throws Exception { byte[] header = header(); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(settings, header); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); long offset = header.length; for (int length : new int[] {2 << 20, 2 << 20, 2 << 20, 2 << 20, 1 << 20}) { builder.beginBlock(offset, length, 1); diff --git a/paimon-core/src/test/resources/manifest-sidecar.txt b/paimon-core/src/test/resources/manifest-sidecar.txt index c8adbfcef042..0597894b4e0a 100644 --- a/paimon-core/src/test/resources/manifest-sidecar.txt +++ b/paimon-core/src/test/resources/manifest-sidecar.txt @@ -18,6 +18,6 @@ avroHeader=T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAA partitionA=AAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhA== partitionB=AAAAAgACAAAAAAAACQAAAAAAAAAAAAAAAAAAAA== -index=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAADkAAAAAAAAAZAAAAAAAAAADAAEAAAAWAAAAAgAAAAAAAAAAAAAAAAAAABgJCwAAAAAAAAAAnQAAAAAAAADIAAAAAAAAAAIAAQAAABwAAAACAAAAAP////4AAAeAzDhsZwTj2OHhjPABAAAAAAAAAAFlAAAAAAAAAGQAAAAAAAAAAgABAAAAHgAAAAIAAAAAAAAAFH/////////rBOf/////////fwBDmcZxzJVsvy0z3TOchlj6Hm/yYQcj7cvI9GCUkgLTuQ== -indexWithPartitions=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAGAAAAAgABAQAAABYAAAACAAAAAAAAAAAAAAAAAAAAGAkLAAAAAAAAAACdAAAAAAAAAMgAAAAAAAAAAgEAAAAGAAAAAgABAQAAABwAAAACAAAAAP////4AAAeAzDhsZwTj2OHhjPABAAAAAAAAAAFlAAAAAAAAAGQAAAAAAAAAAgEAAAAGAAAAAgABAQAAAB4AAAACAAAAAAAAABR/////////6wTn/////////38ANcXHqVsy9rnZu8lIbURyO1ruCjRBJ6y7D1d+3De2K5c= -indexWithBuckets=UE1TQwAAAAEAAAAAAAAByQAAAAAAAAAHAAAAOU9iagEEFGF2cm8uY29kZWMIbnVsbBZhdnJvLnNjaGVtYQwibG9uZyIAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAcAAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhAAAABwAAAACAAIAAAAAAAAJAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAA5AAAAAAAAAGQAAAAAAAAAAwEAAAAGAAAAAgABAQAAABYAAAACAAAAAAAAAAAAAAAAAAAAGAkLAQAAAAoAAAAChICAgBAEAAAAAAAAAJ0AAAAAAAAAyAAAAAAAAAACAQAAAAYAAAACAAEBAAAAHAAAAAIAAAAA/////gAAB4DMOGxnBOPY4eGM8AEBAAAACgAAAAKEgICAIAQAAAAAAAABZQAAAAAAAABkAAAAAAAAAAIBAAAABgAAAAIAAQEAAAAeAAAAAgAAAAAAAAAUf////////+sE5/////////9/AQAAAAoAAAACAYOAgIAwtetSsOZbZ3HLMKzmCkypZMmlEoQ52HFpuzvnx1F2Ht0= +index=UE1TQwE5T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAAAAM5ZAMAARMAAAAAAAAAAAAAAAAAAAAYAgkLAJ0ByAECAAEZAAAAAP////4AAAeBzDhsZQIE49jh4YzwAQDlAmQCAAEbAAAAAAAAABR//////////wIE5/////////9/AOSxTD8YXVtvciRAimgk5ETdJf+5DDmiuqaBJJiQ1/d3 +indexWithPartitions=UE1TQwE5T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAAAhwAAAACAAAAAAAAAAAHAAAAAAAAAGxlZnQAAACEHAAAAAIAAgAAAAAAAAkAAAAAAAAAAAAAAAAAAAADOWQDAQMCAAEBEwAAAAAAAAAAAAAAAAAAABgCCQsAnQHIAQIBAwIAAQEZAAAAAP////4AAAeBzDhsZQIE49jh4YzwAQDlAmQCAQMCAAEBGwAAAAAAAAAUf/////////8CBOf/////////fwCSLQchM3WaqhcAPeGxNUhGyS2SFk+gI+SVQkxbFyfFrw== +indexWithBuckets=UE1TQwE5T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAAAhwAAAACAAAAAAAAAAAHAAAAAAAAAGxlZnQAAACEHAAAAAIAAgAAAAAAAAkAAAAAAAAAAAAAAAAAAAADOWQDAQMCAAEBEwAAAAAAAAAAAAAAAAAAABgCCQsBBgIBAAIICJ0ByAECAQMCAAEBGQAAAAD////+AAAHgcw4bGUCBOPY4eGM8AEBBgICAAIICOUCZAIBAwIAAQEbAAAAAAAAABR//////////wIE5/////////9/AQYCAAMCAgYv5qxBaT4YRIj+XDWEvw1K7HvPfhYW1Rv5XhKZtzlw/A== From e016a0597f49acf1140275218b67a07b9d5ac63f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Wed, 16 Sep 2026 15:55:00 +0800 Subject: [PATCH 13/14] Fix minus --- docs/docs/concepts/spec/manifest.md | 6 +- .../paimon/utils/VarLengthIntUtils.java | 9 + .../paimon/utils/VarLengthIntUtilsTest.java | 101 +++++++ .../paimon/manifest/ManifestSidecar.java | 54 ++-- .../manifest/ManifestBlockIndexTest.java | 277 +++++++++--------- .../paimon/manifest/ManifestSidecarTest.java | 200 +++++++++---- .../compatibility/manifest-sidecar-v1 | Bin 0 -> 2107 bytes .../src/test/resources/manifest-sidecar.txt | 23 -- 8 files changed, 411 insertions(+), 259 deletions(-) create mode 100644 paimon-common/src/test/java/org/apache/paimon/utils/VarLengthIntUtilsTest.java create mode 100644 paimon-core/src/test/resources/compatibility/manifest-sidecar-v1 delete mode 100644 paimon-core/src/test/resources/manifest-sidecar.txt diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index ceaa8a0e4af8..763d64b1ef86 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -121,7 +121,7 @@ blocks[] // original physical order if bucketEncoding != 0: bucketPayloadLength : varint bucketPayload : bytes -checksum : 32 bytes // SHA-256 of all preceding bytes +checksum : 4 bytes // big-endian CRC32 of all preceding bytes ``` The block ID is its position. Its first entry ordinal is the sum of preceding record counts @@ -189,7 +189,7 @@ represented by its entries: ```text partitionPayload - intsDeltaPayload // N > 0 dictionary IDs, base = 0 + intsDeltaPayload ``` An ID is the zero-based position of a complete tuple in the sidecar's shared dictionary. @@ -213,7 +213,7 @@ sorted and disjoint; they are never expanded into individual row IDs or coarsene rowIdPayload minRowId : long // first interval's start maxRowId : long // last interval's inclusive end - intsDeltaPayload // 2 * (N - 1) sorted interior endpoints, base = minRowId + intsDeltaPayload // stores longs: 2 * (N - 1) sorted interior endpoints, base = minRowId ``` The envelope satisfies `0 <= minRowId <= maxRowId <= Long.MAX_VALUE`. Flatten the intervals diff --git a/paimon-common/src/main/java/org/apache/paimon/utils/VarLengthIntUtils.java b/paimon-common/src/main/java/org/apache/paimon/utils/VarLengthIntUtils.java index e3874a868440..cc7a56e5c526 100644 --- a/paimon-common/src/main/java/org/apache/paimon/utils/VarLengthIntUtils.java +++ b/paimon-common/src/main/java/org/apache/paimon/utils/VarLengthIntUtils.java @@ -178,6 +178,15 @@ public static int decodeInt(DataInput is) throws IOException { throw new Error("Malformed integer."); } + /** Decodes a canonical nonnegative int from the buffer and advances its position. */ + public static int decodeInt(ByteBuffer in) throws IOException { + long value = decodeLong(in); + if (value > Integer.MAX_VALUE) { + throw new IOException("Variable-length integer exceeds Integer.MAX_VALUE"); + } + return (int) value; + } + public static int decodeInt(InputStream is) throws IOException { for (int offset = 0, result = 0; offset < 32; offset += 7) { int b = is.read(); diff --git a/paimon-common/src/test/java/org/apache/paimon/utils/VarLengthIntUtilsTest.java b/paimon-common/src/test/java/org/apache/paimon/utils/VarLengthIntUtilsTest.java new file mode 100644 index 000000000000..76a8f4f8d5ec --- /dev/null +++ b/paimon-common/src/test/java/org/apache/paimon/utils/VarLengthIntUtilsTest.java @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.utils; + +import org.apache.paimon.io.DataOutputSerializer; + +import org.junit.jupiter.api.Test; + +import java.io.EOFException; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Arrays; + +import static org.apache.paimon.utils.VarLengthIntUtils.decodeInt; +import static org.apache.paimon.utils.VarLengthIntUtils.encodeInt; +import static org.apache.paimon.utils.VarLengthIntUtils.encodeLong; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Tests for variable-length integer buffer decoding. */ +class VarLengthIntUtilsTest { + + @Test + void decodeIntBoundariesAndBufferPosition() throws Exception { + int[] values = { + 0, + 1, + 127, + 128, + 16383, + 16384, + (1 << 21) - 1, + 1 << 21, + (1 << 28) - 1, + 1 << 28, + Integer.MAX_VALUE + }; + DataOutputSerializer out = new DataOutputSerializer(64); + for (int value : values) { + encodeInt(out, value); + } + byte[] bytes = out.getCopyOfBuffer(); + for (ByteBuffer buffer : + Arrays.asList( + ByteBuffer.allocate(bytes.length + 4), + ByteBuffer.allocateDirect(bytes.length + 4))) { + buffer.position(2); + buffer.put(bytes).put((byte) 42).flip(); + buffer.position(2); + ByteBuffer in = buffer.slice().asReadOnlyBuffer(); + for (int value : values) { + assertThat(decodeInt(in)).isEqualTo(value); + } + assertThat(in.position()).isEqualTo(bytes.length); + assertThat(in.get()).isEqualTo((byte) 42); + } + } + + @Test + void decodeIntRejectsOverflow() throws Exception { + for (long value : new long[] {(long) Integer.MAX_VALUE + 1, 1L << 32, Long.MAX_VALUE}) { + DataOutputSerializer out = new DataOutputSerializer(16); + encodeLong(out, value); + assertThatThrownBy(() -> decodeInt(ByteBuffer.wrap(out.getCopyOfBuffer()))) + .isInstanceOf(IOException.class) + .hasMessageContaining("Integer.MAX_VALUE"); + } + } + + @Test + void decodeIntRejectsTruncatedAndMalformedValues() { + for (byte[] bytes : + Arrays.asList(new byte[0], new byte[] {(byte) 0x80}, new byte[] {(byte) 0xff})) { + assertThatThrownBy(() -> decodeInt(ByteBuffer.wrap(bytes))) + .isInstanceOf(EOFException.class); + } + byte[] overlong = new byte[10]; + Arrays.fill(overlong, (byte) 0x80); + for (byte[] bytes : + Arrays.asList(new byte[] {(byte) 0x80, 0}, new byte[] {(byte) 0x81, 0}, overlong)) { + assertThatThrownBy(() -> decodeInt(ByteBuffer.wrap(bytes))) + .isInstanceOf(IOException.class); + } + } +} diff --git a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java index 6fc39844f62a..c6ee9e9f5615 100644 --- a/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java +++ b/paimon-core/src/main/java/org/apache/paimon/manifest/ManifestSidecar.java @@ -45,8 +45,6 @@ import java.io.InputStream; import java.io.UncheckedIOException; import java.nio.ByteBuffer; -import java.security.MessageDigest; -import java.security.NoSuchAlgorithmException; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -58,7 +56,9 @@ import java.util.TreeMap; import java.util.TreeSet; import java.util.function.BiPredicate; +import java.util.zip.CRC32; +import static org.apache.paimon.utils.VarLengthIntUtils.decodeInt; import static org.apache.paimon.utils.VarLengthIntUtils.encodeLong; /** Independently usable partition, row-id and bucket coverage for each manifest block. */ @@ -71,7 +71,7 @@ public final class ManifestSidecar { private static final int MIN_HEADER_BYTES = 29; private static final int MIN_BLOCK_BYTES = 6; private static final byte[] EMPTY = new byte[0]; - private static final int DIGEST_BYTES = 32; + private static final int CHECKSUM_BYTES = Integer.BYTES; private static final int SIDECAR_READ_BUFFER_BYTES = 1024 * 1024; private static final int BLOCK_READ_BUFFER_BYTES = 4 * 1024 * 1024; private static final ProjectedManifestEntry.Projection BLOCK_INDEX_PROJECTION = @@ -316,7 +316,9 @@ public byte[] serialize(long fileSize, long entryCount) throws IOException { writePayload(out, block.rowIds); writePayload(out, block.buckets); } - out.write(digest(buffer.toByteArray())); + CRC32 crc = new CRC32(); + crc.update(buffer.toByteArray()); + out.writeInt((int) crc.getValue()); return buffer.toByteArray(); } @@ -428,28 +430,28 @@ public static Selection select( @Nullable RowType partitionType, @Nullable BiPredicate bucketFilter) throws IOException { - require(data.length >= MIN_HEADER_BYTES + DIGEST_BYTES); - int limit = data.length - DIGEST_BYTES; - require( - MessageDigest.isEqual( - digest(data, limit), Arrays.copyOfRange(data, limit, data.length))); + require(data.length >= MIN_HEADER_BYTES + CHECKSUM_BYTES); + int limit = data.length - CHECKSUM_BYTES; + CRC32 crc = new CRC32(); + crc.update(data, 0, limit); + require((int) crc.getValue() == ByteBuffer.wrap(data, limit, CHECKSUM_BYTES).getInt()); ByteBuffer in = ByteBuffer.wrap(data, 0, limit).slice(); require(in.getInt() == MAGIC); - require(readInt(in) == FORMAT_VERSION); + require(decodeInt(in) == FORMAT_VERSION); long entries = Math.addExact(manifest.numAddedFiles(), manifest.numDeletedFiles()); require(entries >= 0); - int headerLength = readInt(in); + int headerLength = decodeInt(in); require(headerLength >= 21 && headerLength <= in.remaining() - 2); require(headerLength <= manifest.fileSize()); byte[] header = new byte[headerLength]; in.get(header); require(header[0] == 'O' && header[1] == 'b' && header[2] == 'j' && header[3] == 1); - int partitions = readInt(in); + int partitions = decodeInt(in); require(partitions <= in.remaining() / 13); boolean[] matches = partitionFilter == null ? null : new boolean[partitions]; Set unique = new java.util.HashSet<>(); for (int id = 0; id < partitions; id++) { - int length = readInt(in); + int length = decodeInt(in); require(length >= 12 && length <= in.remaining()); ByteBuffer encoded = in.slice(); encoded.limit(length); @@ -465,7 +467,7 @@ public static Selection select( } in.position(in.position() + length); } - int count = readInt(in); + int count = decodeInt(in); require(count <= in.remaining() / MIN_BLOCK_BYTES); long nextOffset = headerLength; long firstRecord = 0; @@ -498,7 +500,7 @@ public static Selection select( } if (bucketPayload != null) { ByteBuffer prefix = bucketPayload.duplicate(); - int pairs = readInt(prefix); + int pairs = decodeInt(prefix); require(pairs > 0 && pairs <= records && 2L * pairs + 1 <= prefix.remaining()); } long blockFirstRecord = firstRecord; @@ -592,7 +594,7 @@ private static ByteBuffer payload(ByteBuffer in) throws IOException { if (encoding == 0) { return null; } - int length = readInt(in); + int length = decodeInt(in); require(length <= in.remaining()); ByteBuffer result = in.slice(); result.limit(length); @@ -603,12 +605,6 @@ private static ByteBuffer payload(ByteBuffer in) throws IOException { return result; } - private static int readInt(ByteBuffer in) throws IOException { - long value = VarLengthIntUtils.decodeLong(in); - require(value <= Integer.MAX_VALUE); - return (int) value; - } - /** Reads the complete sidecar. Null means read the original manifest. */ @Nullable public static Selection read( @@ -943,18 +939,4 @@ private static void require(boolean valid) throws IOException { throw new IOException("Invalid, unsupported or mismatched manifest sidecar"); } } - - private static byte[] digest(byte[] bytes) { - return digest(bytes, bytes.length); - } - - private static byte[] digest(byte[] bytes, int length) { - try { - MessageDigest digest = MessageDigest.getInstance("SHA-256"); - digest.update(bytes, 0, length); - return digest.digest(); - } catch (NoSuchAlgorithmException e) { - throw new IllegalStateException(e); - } - } } diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java index f412c14f2f74..77c3cc9c9e69 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestBlockIndexTest.java @@ -19,8 +19,6 @@ package org.apache.paimon.manifest; import org.apache.paimon.data.BinaryRow; -import org.apache.paimon.data.BinaryRowWriter; -import org.apache.paimon.data.BinaryString; import org.apache.paimon.partition.PartitionPredicate; import org.apache.paimon.predicate.PredicateBuilder; import org.apache.paimon.types.DataTypes; @@ -36,16 +34,18 @@ import java.io.DataOutputStream; import java.io.IOException; import java.nio.ByteBuffer; -import java.security.MessageDigest; import java.util.ArrayList; import java.util.Arrays; -import java.util.Base64; import java.util.Collections; import java.util.List; -import java.util.Properties; import java.util.function.BiPredicate; +import java.util.zip.CRC32; +import static org.apache.paimon.manifest.ManifestSidecarTest.header; import static org.apache.paimon.manifest.ManifestSidecarTest.meta; +import static org.apache.paimon.manifest.ManifestSidecarTest.partition; +import static org.apache.paimon.manifest.ManifestSidecarTest.testSidecar; +import static org.apache.paimon.manifest.ManifestSidecarTest.verifyGoldenFile; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.mockito.ArgumentMatchers.any; @@ -61,27 +61,6 @@ class ManifestBlockIndexTest { private final RowType type = RowType.of(DataTypes.INT(), DataTypes.STRING()); - private byte[] fixture(String field) throws IOException { - Properties p = new Properties(); - try (java.io.InputStream in = getClass().getResourceAsStream("/manifest-sidecar.txt")) { - p.load(in); - } - return Base64.getDecoder().decode(p.getProperty(field)); - } - - private byte[] partition(int p, String q) { - BinaryRow row = new BinaryRow(2); - BinaryRowWriter writer = new BinaryRowWriter(row); - writer.writeInt(0, p); - if (q == null) { - writer.setNullAt(1); - } else { - writer.writeString(1, BinaryString.fromString(q)); - } - writer.complete(); - return SerializationUtils.serializeBinaryRow(row); - } - private RowRangeIndex query(long point) { return RowRangeIndex.create(Collections.singletonList(new Range(point, point))); } @@ -99,9 +78,74 @@ private PartitionPredicate part(int value) { return PartitionPredicate.fromPredicate(type, new PredicateBuilder(type).equal(0, value)); } + @Test + void testV1GoldenFile() throws Exception { + int blockCount = 10; + int entriesPerBlock = 16; + int blockLength = 1024 * 1024 + 17; + byte[] header = header(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); + for (int block = 0; block < blockCount; block++) { + builder.beginBlock( + header.length + (long) block * blockLength, blockLength, entriesPerBlock); + long rowBase = (long) block << 40; + int step = block % 4 == 0 ? 16400 : 16; + for (int entry = 0; entry < entriesPerBlock; entry++) { + int p = (block * 7 + entry * 3) % 16; + builder.add( + rowBase + (long) entry * step, + 3, + partition(p, p % 7 == 0 ? null : "partition-" + p), + block + entry * 13, + entry % 2 == 0 ? 512 : 1024); + } + builder.endBlock(); + } + long fileSize = header.length + (long) blockCount * blockLength; + long entryCount = (long) blockCount * entriesPerBlock; + byte[] current = builder.serialize(fileSize, entryCount); + assertThat(current.length).isBetween(2 * 1024, 2 * 1024 + 256); + byte[] data = verifyGoldenFile(current); + ManifestFileMeta meta = meta("manifest-golden-v1", fileSize, entryCount); + assertThat(partitionCount(data)).isEqualTo(16); + List blocks = ManifestSidecar.select(data, meta, null).blocks(); + assertThat(blocks).hasSize(blockCount); + for (int block = 0; block < blockCount; block++) { + ManifestSidecar.Block expected = blocks.get(block); + assertThat(expected.offset).isEqualTo(header.length + (long) block * blockLength); + assertThat(expected.length).isEqualTo(blockLength); + assertThat(expected.firstRecord).isEqualTo((long) block * entriesPerBlock); + assertThat(expected.recordCount).isEqualTo(entriesPerBlock); + int step = block % 4 == 0 ? 16400 : 16; + for (int entry = 0; entry < entriesPerBlock; entry++) { + long rowId = ((long) block << 40) + (long) entry * step; + int p = (block * 7 + entry * 3) % 16; + int bucket = block + entry * 13; + int total = entry % 2 == 0 ? 512 : 1024; + assertThat( + ManifestSidecar.select( + data, + meta, + query(rowId), + part(p), + type, + (b, t) -> b == bucket && t == total) + .blocks()) + .extracting(b -> b.firstRecord) + .containsExactly((long) block * entriesPerBlock); + assertThat(ManifestSidecar.select(data, meta, query(rowId + 3)).blocks()).isEmpty(); + } + } + assertThat(ManifestSidecar.select(data, meta, null, part(99), type).blocks()).isEmpty(); + assertThat( + ManifestSidecar.select(data, meta, null, null, type, (b, t) -> t == 4096) + .blocks()) + .isEmpty(); + } + @Test void partitionCoverageIsAlwaysGenerated() throws Exception { - byte[] header = fixture("avroHeader"); + byte[] header = header(); for (int mask = 0; mask < 4; mask++) { boolean rowIdEnabled = (mask & 1) != 0; boolean bucketEnabled = (mask & 2) != 0; @@ -128,7 +172,7 @@ void partitionCoverageIsAlwaysGenerated() throws Exception { .hasSize(bucketEnabled ? 0 : 2); // Generation settings do not disable payloads already stored in a sidecar. - byte[] existing = fixture("indexWithBuckets"); + byte[] existing = testSidecar(); ManifestFileMeta existingMeta = meta("manifest-golden", header.length + 400, 7); assertThat(ManifestSidecar.select(existing, existingMeta, query(999)).blocks()) .isEmpty(); @@ -144,14 +188,11 @@ void partitionCoverageIsAlwaysGenerated() throws Exception { } } - @Test - void jointGoldenPreservesTuplesNullsAndDerivedOrdinals() throws Exception { - byte[] a = partition(7, "left"); - byte[] b = partition(9, null); - assertThat(a).isEqualTo(fixture("partitionA")); - assertThat(b).isEqualTo(fixture("partitionB")); - byte[] header = fixture("avroHeader"); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); + private byte[] sidecarWithoutBuckets(boolean partitioned) throws IOException { + byte[] a = partitioned ? partition(7, "left") : null; + byte[] b = partitioned ? partition(9, null) : null; + byte[] header = header(); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, false); builder.beginBlock(header.length, 100, 3); builder.add(0L, 10, a); builder.add(5L, 5, a); @@ -165,9 +206,13 @@ void jointGoldenPreservesTuplesNullsAndDerivedOrdinals() throws Exception { builder.add(20L, 5, a); builder.add(Long.MAX_VALUE, 1, b); builder.endBlock(); - byte[] data = builder.serialize(header.length + 400, 7); - assertThat(data).isEqualTo(fixture("indexWithPartitions")); - ManifestFileMeta meta = meta("manifest-golden", header.length + 400, 7); + return builder.serialize(header.length + 400, 7); + } + + @Test + void partitionTuplesNullsAndDerivedOrdinals() throws Exception { + byte[] data = testSidecar(); + ManifestFileMeta meta = testMeta(); PartitionPredicate filter = spy(part(7)); assertThat(ManifestSidecar.select(data, meta, query(20), filter, type).blocks()) .extracting(block -> block.firstRecord) @@ -178,13 +223,16 @@ void jointGoldenPreservesTuplesNullsAndDerivedOrdinals() throws Exception { assertThat(ManifestSidecar.select(data, meta, null, nullFilter, type).blocks()).hasSize(3); assertThat(ManifestSidecar.select(data, meta, null, part(99), type).blocks()).isEmpty(); // Missing partition payloads cannot be pruned by dictionary misses. - assertThat(ManifestSidecar.select(fixture("index"), meta, null, part(99), type).blocks()) + assertThat( + ManifestSidecar.select( + sidecarWithoutBuckets(false), meta, null, part(99), type) + .blocks()) .hasSize(3); } @Test void unavailableDimensionsAreIndependentAndDoNotPoisonLaterBlocks() throws Exception { - byte[] header = fixture("avroHeader"); + byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 1); builder.add(null, 10, partition(7, "left")); @@ -252,17 +300,18 @@ private int skipPayload(ByteBuffer in) throws IOException { return position; } - private byte[] checksum(byte[] data) throws Exception { - byte[] hash = - MessageDigest.getInstance("SHA-256").digest(Arrays.copyOf(data, data.length - 32)); - System.arraycopy(hash, 0, data, data.length - 32, 32); + private byte[] checksum(byte[] data) { + int limit = data.length - Integer.BYTES; + CRC32 crc = new CRC32(); + crc.update(data, 0, limit); + ByteBuffer.wrap(data, limit, Integer.BYTES).putInt((int) crc.getValue()); return data; } @Test void absentRowOrBucketFiltersKeepRemainingDimensions() throws Exception { - byte[] data = fixture("indexWithBuckets"); - ManifestFileMeta meta = meta("manifest-golden", fixture("avroHeader").length + 400, 7); + byte[] data = testSidecar(); + ManifestFileMeta meta = meta("manifest-golden", header().length + 400, 7); assertThat( ManifestSidecar.select(data, meta, null, part(7), type, bucketFilter(1)) .blocks()) @@ -277,26 +326,9 @@ void absentRowOrBucketFiltersKeepRemainingDimensions() throws Exception { } @Test - void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { - byte[] header = fixture("avroHeader"); - byte[] a = partition(7, "left"); - byte[] b = partition(9, null); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); - builder.beginBlock(header.length, 100, 3); - builder.add(0L, 10, a, 1, 4); - builder.add(5L, 5, a, 1, 4); - builder.add(20L, 5, b, 1, 8); - builder.endBlock(); - builder.beginBlock(header.length + 100, 200, 2); - builder.add((1L << 32) - 2, 5, b, 2, 4); - builder.add(8254058425445L, 1, a, 2, 8); - builder.endBlock(); - builder.beginBlock(header.length + 300, 100, 2); - builder.add(20L, 5, a, 0, 1); - builder.add(Long.MAX_VALUE, 1, b, 3, 4); - builder.endBlock(); - byte[] data = builder.serialize(header.length + 400, 7); - assertThat(data).isEqualTo(fixture("indexWithBuckets")); + void bucketPayloadAndTotalBucketsArePreserved() throws Exception { + byte[] header = header(); + byte[] data = testSidecar(); ManifestFileMeta meta = meta("manifest-golden", header.length + 400, 7); BiPredicate bucket = bucketFilter(1); assertThat(ManifestSidecar.select(data, meta, null, null, type, bucket).blocks()) @@ -313,10 +345,10 @@ void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { .containsExactly(3L); // The caller can omit a bucket filter which requires an actual entry partition. assertThat(ManifestSidecar.select(data, meta, null, null, type, null).blocks()).hasSize(3); - for (String unavailable : new String[] {"index", "indexWithPartitions"}) { + for (boolean partitioned : new boolean[] {false, true}) { assertThat( ManifestSidecar.select( - fixture(unavailable), + sidecarWithoutBuckets(partitioned), meta, null, null, @@ -329,7 +361,7 @@ void bucketPayloadGoldenAndTotalBucketsArePreserved() throws Exception { @Test void splitBucketArraysPreserveDecreasingTotalsAndRescaling() throws Exception { - byte[] header = fixture("avroHeader"); + byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, false, true); builder.beginBlock(header.length, 100, 5); for (int[] pair : new int[][] {{3, 4}, {2, 8}, {0, Integer.MAX_VALUE}, {2, 4}, {2, 4}}) { @@ -382,18 +414,18 @@ void mismatchedBucketCountsAndNegativeTotalsAreRejected() throws Exception { new byte[] {2, 1, 0, 1, 8, 8}, // Two buckets, one total, extra bytes. bucketPayload(2, new long[] {1, 0}, 1, 0), // ZigZag -1 from zero. bucketPayload(2, new long[] {1, 0}, 8, 0))) { // Repeated pair. - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, payload); + byte[] data = replacePayload(testSidecar(), 0, 3, payload); assertThatThrownBy( () -> ManifestSidecar.select( - data, goldenMeta(), null, null, type, bucketFilter(99))) + data, testMeta(), null, null, type, bucketFilter(99))) .isInstanceOf(IOException.class); } } @Test void unknownOrInvalidBucketPayloadIsUnavailable() throws Exception { - byte[] header = fixture("avroHeader"); + byte[] header = header(); for (Integer[] pair : Arrays.asList( new Integer[] {null, null}, @@ -428,7 +460,7 @@ data, meta, query(999), null, type, bucketFilter(99)) @Test void largePayloadsKeepExactCoverage() throws Exception { - byte[] header = fixture("avroHeader"); + byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); int blocks = 33; int entriesPerBlock = 4097; @@ -462,7 +494,7 @@ data, meta, null, null, type, bucketFilter(entriesPerBlock)) @Test void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Exception { - byte[] header = fixture("avroHeader"); + byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); for (int mask = 0; mask < 8; mask++) { builder.beginBlock(header.length + mask * 100L, 100, 1); @@ -483,7 +515,9 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti int end = dimension < 2 ? positions.get(mask)[dimension + 2] - : mask < 7 ? positions.get(mask + 1)[0] : data.length - 32; + : mask < 7 + ? positions.get(mask + 1)[0] + : data.length - Integer.BYTES; boolean present = (mask & (1 << dimension)) != 0; assertThat(data[start]).isEqualTo((byte) (present ? 1 : 0)); assertThat(end - start).isEqualTo(present ? presentSizes[dimension] : 1); @@ -507,7 +541,7 @@ void absentPayloadsOmitLengthFieldsForEveryDimensionCombination() throws Excepti @Test void deltaVarintsCompressSortedPayloadsWithoutCoarseningRowIds() throws Exception { - byte[] header = fixture("avroHeader"); + byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); int count = 10000; builder.beginBlock(header.length, 100, count); @@ -531,7 +565,7 @@ data, meta, query(4), part(9999), type, bucketFilter(9999)) @Test void unpartitionedTablesStillRecordTheEmptyPartition() throws Exception { - byte[] header = fixture("avroHeader"); + byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, false, false); builder.beginBlock(header.length, 100, 1); builder.add(null, 0, SerializationUtils.serializeBinaryRow(BinaryRow.EMPTY_ROW)); @@ -555,13 +589,11 @@ void unpartitionedTablesStillRecordTheEmptyPartition() throws Exception { @Test void rowMissSkipsPartitionAndBucketDecoding() throws Exception { - byte[] data = - replacePayload(fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 999, 1)); + byte[] data = replacePayload(testSidecar(), 0, 1, compressedPayload(2, 999, 1)); data = replacePayload(data, 0, 3, bucketPayload(2, new long[] {0, 0}, 0, 0)); BiPredicate buckets = mock(BiPredicate.class); assertThat( - ManifestSidecar.select( - data, goldenMeta(), query(15), part(7), type, buckets) + ManifestSidecar.select(data, testMeta(), query(15), part(7), type, buckets) .blocks()) .isEmpty(); verifyNoInteractions(buckets); @@ -570,16 +602,11 @@ data, goldenMeta(), query(15), part(7), type, buckets) @Test void partitionMissSkipsBucketDecodingWithOrWithoutRowQuery() throws Exception { byte[] data = - replacePayload( - fixture("indexWithBuckets"), - 0, - 3, - bucketPayload(2, new long[] {0, 0}, 0, 0)); + replacePayload(testSidecar(), 0, 3, bucketPayload(2, new long[] {0, 0}, 0, 0)); for (RowRangeIndex rows : Arrays.asList(null, query(0))) { BiPredicate buckets = mock(BiPredicate.class); assertThat( - ManifestSidecar.select( - data, goldenMeta(), rows, part(99), type, buckets) + ManifestSidecar.select(data, testMeta(), rows, part(99), type, buckets) .blocks()) .isEmpty(); verifyNoInteractions(buckets); @@ -588,11 +615,10 @@ data, goldenMeta(), rows, part(99), type, buckets) @Test void absentPartitionFilterDoesNotDecodePartitionIds() throws Exception { - byte[] data = - replacePayload(fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 999, 1)); + byte[] data = replacePayload(testSidecar(), 0, 1, compressedPayload(2, 999, 1)); BiPredicate buckets = spy(bucketFilter(1)); assertThat( - ManifestSidecar.select(data, goldenMeta(), query(20), null, type, buckets) + ManifestSidecar.select(data, testMeta(), query(20), null, type, buckets) .blocks()) .extracting(block -> block.firstRecord) .containsExactly(0L); @@ -604,45 +630,37 @@ void absentPartitionFilterDoesNotDecodePartitionIds() throws Exception { @Test void matchesSkipUnusedDeltas() throws Exception { - byte[] partitions = - replacePayload(fixture("indexWithBuckets"), 0, 1, compressedPayload(2, 0, 999)); - assertThat( - ManifestSidecar.select(partitions, goldenMeta(), query(0), part(7), type) - .blocks()) + byte[] partitions = replacePayload(testSidecar(), 0, 1, compressedPayload(2, 0, 999)); + assertThat(ManifestSidecar.select(partitions, testMeta(), query(0), part(7), type).blocks()) .hasSize(1); assertThatThrownBy( () -> ManifestSidecar.select( - partitions, goldenMeta(), query(0), part(99), type)) + partitions, testMeta(), query(0), part(99), type)) .isInstanceOf(IOException.class); byte[] buckets = replacePayload( - fixture("indexWithBuckets"), + testSidecar(), 0, 3, bucketPayload(2, new long[] {1, 0}, 8, Long.MAX_VALUE)); assertThat( ManifestSidecar.select( - buckets, - goldenMeta(), - query(0), - null, - type, - bucketFilter(1)) + buckets, testMeta(), query(0), null, type, bucketFilter(1)) .blocks()) .hasSize(1); assertThatThrownBy( () -> ManifestSidecar.select( buckets, - goldenMeta(), + testMeta(), query(0), null, type, bucketFilter(99))) .isInstanceOf(IOException.class); - byte[] header = fixture("avroHeader"); + byte[] header = header(); ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); builder.beginBlock(header.length, 100, 3); for (long first : new long[] {0, 20, 40}) { @@ -677,17 +695,17 @@ void malformedCompressedPayloadsFailWhenConsumed() throws Exception { Arrays.copyOf(rowPayload(1, 0, 24), 16), // Missing endpoint count. rowPayload(1, 0, 24, 0)); // Unexpected value for a single interval. for (byte[] payload : badRows) { - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 2, payload); - assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(15))) + byte[] data = replacePayload(testSidecar(), 0, 2, payload); + assertThatThrownBy(() -> ManifestSidecar.select(data, testMeta(), query(15))) .isInstanceOf(IOException.class); } for (byte[] payload : Arrays.asList(compressedPayload(2, 999, 0), compressedPayload(2, 0, 0))) { - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 1, payload); + byte[] data = replacePayload(testSidecar(), 0, 1, payload); assertThatThrownBy( () -> ManifestSidecar.select( - data, goldenMeta(), query(0), part(99), type)) + data, testMeta(), query(0), part(99), type)) .isInstanceOf(IOException.class); } for (byte[] payload : @@ -695,12 +713,12 @@ data, goldenMeta(), query(0), part(99), type)) bucketPayload(2, new long[] {0, 4}, 2, 0), bucketPayload(2, new long[] {1L << 31, 4}, 8, 0), bucketPayload(2, new long[] {0, 0}, 2, 0))) { - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, 3, payload); + byte[] data = replacePayload(testSidecar(), 0, 3, payload); assertThatThrownBy( () -> ManifestSidecar.select( data, - goldenMeta(), + testMeta(), query(0), null, type, @@ -728,15 +746,13 @@ void malformedDeltaVarintsFailWhenConsumed() throws Exception { ? bucketPayload(2, new long[] {1, 0}) : compressedPayload(2)); payload.write(deltas); - byte[] data = - replacePayload( - fixture("indexWithBuckets"), 0, dimension, payload.toByteArray()); + byte[] data = replacePayload(testSidecar(), 0, dimension, payload.toByteArray()); int dim = dimension; assertThatThrownBy( () -> ManifestSidecar.select( data, - goldenMeta(), + testMeta(), query(0), dim == 1 ? part(99) : null, type, @@ -760,23 +776,22 @@ void invalidCountFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception { compressedPayload(2, 1)); for (int dimension = 1; dimension <= 3; dimension++) { for (byte[] payload : bad) { - byte[] data = replacePayload(fixture("indexWithBuckets"), 0, dimension, payload); - assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) + byte[] data = replacePayload(testSidecar(), 0, dimension, payload); + assertThatThrownBy(() -> ManifestSidecar.select(data, testMeta(), query(999))) .isInstanceOf(IOException.class); } - byte[] data = fixture("indexWithBuckets"); + byte[] data = testSidecar(); int position = positions(data).get(0)[dimension]; Arrays.fill(data, position + 1, position + 6, (byte) 0xff); checksum(data); - assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) + assertThatThrownBy(() -> ManifestSidecar.select(data, testMeta(), query(999))) .isInstanceOf(IOException.class); } byte[] missingEnvelope = - replacePayload( - fixture("indexWithBuckets"), 0, 2, Arrays.copyOf(rowPayload(1, 0, 24), 16)); - assertThatThrownBy(() -> ManifestSidecar.select(missingEnvelope, goldenMeta(), null)) + replacePayload(testSidecar(), 0, 2, Arrays.copyOf(rowPayload(1, 0, 24), 16)); + assertThatThrownBy(() -> ManifestSidecar.select(missingEnvelope, testMeta(), null)) .isInstanceOf(IOException.class); - byte[] data = fixture("indexWithBuckets"); + byte[] data = testSidecar(); int block = positions(data).get(0)[0]; ByteBuffer directory = ByteBuffer.wrap(data); directory.position(block); @@ -784,22 +799,20 @@ void invalidCountFramingAndDirectoryFailEvenWhenFiltersMiss() throws Exception { VarLengthIntUtils.decodeLong(directory); data[directory.position()] = 2; checksum(data); - assertThatThrownBy(() -> ManifestSidecar.select(data, goldenMeta(), query(999))) + assertThatThrownBy(() -> ManifestSidecar.select(data, testMeta(), query(999))) .isInstanceOf(IOException.class); } @Test void unknownEncodingsRemainIndependent() throws Exception { for (int dimension = 1; dimension <= 3; dimension++) { - byte[] data = - replacePayload( - fixture("indexWithBuckets"), 0, dimension, new byte[] {(byte) 0x80}); + byte[] data = replacePayload(testSidecar(), 0, dimension, new byte[] {(byte) 0x80}); data[positions(data).get(0)[dimension]] = (byte) 202; checksum(data); assertThat( ManifestSidecar.select( data, - goldenMeta(), + testMeta(), query(dimension == 2 ? 999 : 0), part(dimension == 1 ? 99 : 7), type, @@ -810,8 +823,8 @@ void unknownEncodingsRemainIndependent() throws Exception { } } - private ManifestFileMeta goldenMeta() throws Exception { - return meta("manifest-golden", fixture("avroHeader").length + 400, 7); + private ManifestFileMeta testMeta() throws Exception { + return meta("manifest-golden", header().length + 400, 7); } private byte[] replacePayload(byte[] data, int block, int dimension, byte[] payload) diff --git a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java index 28f7086626c0..b4a7b3e29bf3 100644 --- a/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/manifest/ManifestSidecarTest.java @@ -20,6 +20,7 @@ import org.apache.paimon.data.BinaryRow; import org.apache.paimon.data.BinaryRowWriter; +import org.apache.paimon.data.BinaryString; import org.apache.paimon.data.InternalRow; import org.apache.paimon.data.SingleSegments; import org.apache.paimon.format.FileFormat; @@ -34,6 +35,7 @@ import org.apache.paimon.options.Options; import org.apache.paimon.schema.FileSystemSchemaManager; import org.apache.paimon.utils.CloseableIterator; +import org.apache.paimon.utils.CompatibilityUtils; import org.apache.paimon.utils.IOUtils; import org.apache.paimon.utils.PathFactory; import org.apache.paimon.utils.Range; @@ -41,6 +43,8 @@ import org.apache.paimon.utils.SegmentsCache; import org.apache.paimon.utils.SerializationUtils; +import org.apache.avro.io.BinaryEncoder; +import org.apache.avro.io.EncoderFactory; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; @@ -49,15 +53,14 @@ import java.io.IOException; import java.io.InputStream; import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; -import java.security.MessageDigest; import java.util.ArrayList; import java.util.Arrays; -import java.util.Base64; import java.util.Collections; import java.util.List; -import java.util.Properties; import java.util.concurrent.CancellationException; +import java.util.zip.CRC32; import static org.apache.paimon.TestKeyValueGenerator.DEFAULT_PART_TYPE; import static org.assertj.core.api.Assertions.assertThat; @@ -70,9 +73,12 @@ import static org.mockito.Mockito.verifyNoInteractions; import static org.mockito.Mockito.when; -/** Cross-language format, physical block positions, completeness and allocation bounds. */ +/** Golden file format, physical block positions, completeness and allocation bounds. */ class ManifestSidecarTest { + private static final String GENERATE_GOLDEN_FILES_PROPERTY = + "generateManifestSidecarGoldenFiles"; + @TempDir java.nio.file.Path temp; static ManifestFileMeta meta(String name, long size, long entries) { @@ -85,23 +91,79 @@ static ManifestFileMeta meta(String name, long size, long entries) { return meta; } - private Properties fixture() throws IOException { - Properties properties = new Properties(); - try (java.io.InputStream input = getClass().getResourceAsStream("/manifest-sidecar.txt")) { - properties.load(input); + static byte[] readGoldenFile() throws IOException { + String resource = "/compatibility/manifest-sidecar-v1"; + try (InputStream input = ManifestSidecarTest.class.getResourceAsStream(resource)) { + assertThat(input).as("Golden file %s", resource).isNotNull(); + return IOUtils.readFully(input, false); } - return properties; } - private byte[] header() throws IOException { - return Base64.getDecoder().decode(fixture().getProperty("avroHeader")); + static byte[] verifyGoldenFile(byte[] current) throws IOException { + if (Boolean.parseBoolean( + System.getProperties().getProperty(GENERATE_GOLDEN_FILES_PROPERTY))) { + CompatibilityUtils.writeCompatibilityFile("manifest-sidecar-v1", current); + return current; + } + byte[] golden = readGoldenFile(); + assertThat(current).isEqualTo(golden); + return golden; } - private byte[] golden() throws IOException { - return Base64.getDecoder().decode(fixture().getProperty("index")); + static byte[] header() throws IOException { + ByteArrayOutputStream output = new ByteArrayOutputStream(); + BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(output, null); + encoder.writeFixed(new byte[] {'O', 'b', 'j', 1}); + // Fix both metadata order and sync marker for deterministic golden files. + encoder.writeMapStart(); + encoder.setItemCount(2); + encoder.startItem(); + encoder.writeString("avro.codec"); + encoder.writeBytes("null".getBytes(StandardCharsets.UTF_8)); + encoder.startItem(); + encoder.writeString("avro.schema"); + encoder.writeBytes("\"long\"".getBytes(StandardCharsets.UTF_8)); + encoder.writeMapEnd(); + encoder.writeFixed(new byte[16]); + encoder.flush(); + return output.toByteArray(); + } + + static byte[] partition(int p, String q) { + BinaryRow row = new BinaryRow(2); + BinaryRowWriter writer = new BinaryRowWriter(row); + writer.writeInt(0, p); + if (q == null) { + writer.setNullAt(1); + } else { + writer.writeString(1, BinaryString.fromString(q)); + } + writer.complete(); + return SerializationUtils.serializeBinaryRow(row); } - private ManifestFileMeta goldenMeta() throws IOException { + static byte[] testSidecar() throws IOException { + byte[] header = header(); + byte[] a = partition(7, "left"); + byte[] b = partition(9, null); + ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); + builder.beginBlock(header.length, 100, 3); + builder.add(0L, 10, a, 1, 4); + builder.add(5L, 5, a, 1, 4); + builder.add(20L, 5, b, 1, 8); + builder.endBlock(); + builder.beginBlock(header.length + 100, 200, 2); + builder.add((1L << 32) - 2, 5, b, 2, 4); + builder.add(8254058425445L, 1, a, 2, 8); + builder.endBlock(); + builder.beginBlock(header.length + 300, 100, 2); + builder.add(20L, 5, a, 0, 1); + builder.add(Long.MAX_VALUE, 1, b, 3, 4); + builder.endBlock(); + return builder.serialize(header.length + 400, 7); + } + + private ManifestFileMeta testMeta() throws IOException { return meta("manifest-golden", header().length + 400, 7); } @@ -114,6 +176,27 @@ void emptyManifestHasACompleteSidecar() throws Exception { .isEmpty(); } + @Test + void crc32FooterAndTruncatedSidecars() throws Exception { + byte[] good = testSidecar(); + ManifestFileMeta meta = testMeta(); + int limit = good.length - Integer.BYTES; + // Independently generated CRC32, stored in big-endian byte order. + assertThat(ByteBuffer.wrap(good, limit, Integer.BYTES).getInt()).isEqualTo(0xdf82cd30); + assertThat(ManifestSidecar.select(good, meta, null).blocks()).hasSize(3); + for (int position = limit; position < good.length; position++) { + byte[] bad = good.clone(); + bad[position] ^= 1; + assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, null)) + .isInstanceOf(IOException.class); + } + for (int length = 0; length < good.length; length++) { + byte[] truncated = Arrays.copyOf(good, length); + assertThatThrownBy(() -> ManifestSidecar.select(truncated, meta, null)) + .isInstanceOf(IOException.class); + } + } + @Test void buildAndReadSelectedBlocksFromPhysicalManifests() throws Exception { FileIO io = LocalFileIO.create(); @@ -197,25 +280,10 @@ private ManifestAvroWriter writer(FileIO io, Path path) { } @Test - void crossLanguageFormatAndBlockOrdinals() throws Exception { + void rowIdCoverageAndBlockOrdinals() throws Exception { byte[] header = header(); - ManifestSidecar.Builder builder = new ManifestSidecar.Builder(header, true, true); - builder.beginBlock(header.length, 100, 3); - builder.add(0L, 10); - builder.add(5L, 5); - builder.add(20L, 5); - builder.endBlock(); - builder.beginBlock(header.length + 100, 200, 2); - builder.add((1L << 32) - 2, 5); - builder.add(8254058425445L, 1); - builder.endBlock(); - builder.beginBlock(header.length + 300, 100, 2); - builder.add(20L, 5); - builder.add(Long.MAX_VALUE, 1); - builder.endBlock(); - byte[] data = builder.serialize(header.length + 400, 7); - assertThat(data).isEqualTo(golden()); - ManifestFileMeta meta = goldenMeta(); + byte[] data = testSidecar(); + ManifestFileMeta meta = testMeta(); for (long point : new long[] { 0, @@ -382,11 +450,11 @@ void hugeRangesAreNotExpandedAndInvalidCoverageRetainsBlocks() throws Exception @Test void cacheRespectsElementThreshold() throws Exception { - byte[] data = golden(); + byte[] data = testSidecar(); Path path = new Path(temp.toString(), "manifest-golden"); Path sidecar = ManifestSidecar.path(path); Files.write(temp.resolve(sidecar.getName()), data); - ManifestFileMeta meta = goldenMeta(); + ManifestFileMeta meta = testMeta(); FileIO io = spy(LocalFileIO.create()); SegmentsCache tooSmall = new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), data.length - 1L, null, false); @@ -404,7 +472,7 @@ void cacheRespectsElementThreshold() throws Exception { @Test void cachedSegmentsRequireSidecarType() throws Exception { - byte[] data = golden(); + byte[] data = testSidecar(); Path path = new Path(temp.toString(), "manifest-golden"); Path sidecar = ManifestSidecar.path(path); Files.write(temp.resolve(sidecar.getName()), data); @@ -413,21 +481,21 @@ void cachedSegmentsRequireSidecarType() throws Exception { cache.put(sidecar, new SingleSegments(MemorySegment.wrap(data), data.length)); FileIO io = spy(LocalFileIO.create()); - assertThat(readCached(io, path, goldenMeta(), cache).blocks()).hasSize(2); + assertThat(readCached(io, path, testMeta(), cache).blocks()).hasSize(2); assertThat(cache.getIfPresents(sidecar)).isInstanceOf(ManifestSidecarSegment.class); ManifestSidecarSegment cached = (ManifestSidecarSegment) cache.getIfPresents(sidecar); assertThat(cached.bytes()).containsExactly(data); assertThat(cached.totalMemorySize()).isEqualTo(data.length); - assertThat(readCached(io, path, goldenMeta(), cache).blocks()).hasSize(2); + assertThat(readCached(io, path, testMeta(), cache).blocks()).hasSize(2); verify(io, times(1)).newInputStream(sidecar); } @Test void missingAndInvalidSidecarsAreNotCached() throws Exception { - byte[] data = golden(); + byte[] data = testSidecar(); Path path = new Path(temp.toString(), "manifest-golden"); Path sidecar = ManifestSidecar.path(path); - ManifestFileMeta meta = goldenMeta(); + ManifestFileMeta meta = testMeta(); FileIO io = spy(LocalFileIO.create()); SegmentsCache cache = new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), Long.MAX_VALUE, null, false); @@ -448,8 +516,8 @@ void missingAndInvalidSidecarsAreNotCached() throws Exception { void cachedBytesPreservePerQueryCancellation() throws Exception { Path path = new Path(temp.toString(), "manifest-golden"); Path sidecar = ManifestSidecar.path(path); - Files.write(temp.resolve(sidecar.getName()), golden()); - ManifestFileMeta meta = goldenMeta(); + Files.write(temp.resolve(sidecar.getName()), testSidecar()); + ManifestFileMeta meta = testMeta(); FileIO io = spy(LocalFileIO.create()); SegmentsCache cache = new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), Long.MAX_VALUE, null, false); @@ -490,11 +558,11 @@ private ManifestSidecar.Selection readCached( void corruptMissingIncompleteAndMismatchedIndexesFallback() throws Exception { Path manifest = new Path(temp.toString(), "manifest-golden"); java.nio.file.Path index = temp.resolve("manifest-golden" + ManifestSidecar.SUFFIX); - ManifestFileMeta meta = goldenMeta(); + ManifestFileMeta meta = testMeta(); RowRangeIndex query = RowRangeIndex.create(Collections.singletonList(new Range(11, 11))); assertThat(ManifestSidecar.read(LocalFileIO.create(), manifest, meta, query)).isNull(); - byte[] good = golden(); + byte[] good = testSidecar(); for (int position : new int[] {0, 9, 11, 15, 16, 55, 63, 67, 75, good.length - 1}) { byte[] bad = good.clone(); bad[position] ^= 2; @@ -505,10 +573,10 @@ void corruptMissingIncompleteAndMismatchedIndexesFallback() throws Exception { for (int version : new int[] {0, 2, 99}) { byte[] bad = good.clone(); bad[4] = (byte) version; - byte[] hash = - MessageDigest.getInstance("SHA-256") - .digest(Arrays.copyOf(bad, bad.length - 32)); - System.arraycopy(hash, 0, bad, bad.length - 32, 32); + int limit = bad.length - Integer.BYTES; + CRC32 crc = new CRC32(); + crc.update(bad, 0, limit); + ByteBuffer.wrap(bad, limit, Integer.BYTES).putInt((int) crc.getValue()); assertThatThrownBy(() -> ManifestSidecar.select(bad, meta, query)) .isInstanceOf(IOException.class); } @@ -656,7 +724,7 @@ void sidecarsLargerThanTheFormerDefaultLimitAreReadCompletely() throws Exception @Test void indexShortReadsReadTheWholeFile() throws Exception { - byte[] data = golden(); + byte[] data = testSidecar(); Path path = new Path(temp.toString(), "manifest-golden"); for (int maxRead : new int[] {Integer.MAX_VALUE, 7}) { CountingInput stream = new CountingInput(data, maxRead); @@ -666,7 +734,7 @@ void indexShortReadsReadTheWholeFile() throws Exception { ManifestSidecar.read( io, path, - goldenMeta(), + testMeta(), RowRangeIndex.create(Collections.singletonList(new Range(20, 20)))); assertThat(actual.blocks()) .extracting(block -> block.firstRecord) @@ -684,8 +752,8 @@ void adjacentBlocksShareReadsForSingleByteConsumers() throws Exception { } ManifestSidecar.Selection selected = ManifestSidecar.select( - golden(), - goldenMeta(), + testSidecar(), + testMeta(), RowRangeIndex.create( Arrays.asList( new Range(0, 0), @@ -722,7 +790,8 @@ void blockReadsSkipGapsAndEmptySelections() throws Exception { when(io.newInputStream(path)).thenReturn(stream); byte[] actual; try (InputStream input = - ManifestSidecar.openManifest(io, path, select(golden(), goldenMeta(), point))) { + ManifestSidecar.openManifest( + io, path, select(testSidecar(), testMeta(), point))) { actual = IOUtils.readFully(input, false); } if (point == 20) { @@ -758,8 +827,8 @@ void cachedBlocksAreSharedByDifferentSelectionsWithoutOpeningTheManifest() throw when(io.newInputStream(path)).thenReturn(stream); ManifestSidecar.Selection all = ManifestSidecar.select( - golden(), - goldenMeta(), + testSidecar(), + testMeta(), RowRangeIndex.create( Collections.singletonList(new Range(0, Long.MAX_VALUE)))); try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { @@ -771,7 +840,7 @@ void cachedBlocksAreSharedByDifferentSelectionsWithoutOpeningTheManifest() throw assertThat(cache.getIfPresents(path)).isNull(); when(io.newInputStream(path)).thenThrow(new IOException("Must use cached blocks")); for (long point : new long[] {20, 8254058425445L, 0}) { - ManifestSidecar.Selection selected = select(golden(), goldenMeta(), point); + ManifestSidecar.Selection selected = select(testSidecar(), testMeta(), point); ByteArrayOutputStream expected = new ByteArrayOutputStream(); expected.write(header); for (ManifestSidecar.Block block : selected.blocks()) { @@ -788,7 +857,8 @@ void cachedBlocksAreSharedByDifferentSelectionsWithoutOpeningTheManifest() throw otherBytes[header.length] ^= 1; when(io.newInputStream(other)).thenReturn(new CountingInput(otherBytes, Integer.MAX_VALUE)); try (InputStream in = - ManifestSidecar.openManifest(io, other, select(golden(), goldenMeta(), 0), cache)) { + ManifestSidecar.openManifest( + io, other, select(testSidecar(), testMeta(), 0), cache)) { assertThat(IOUtils.readFully(in, false)) .isEqualTo(Arrays.copyOf(otherBytes, header.length + 100)); } @@ -808,13 +878,13 @@ void mixedHitsAndMissesReadOnlyUncachedBlocks() throws Exception { new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), 400, null, false); try (InputStream in = ManifestSidecar.openManifest( - io, path, select(golden(), goldenMeta(), 8254058425445L), cache)) { + io, path, select(testSidecar(), testMeta(), 8254058425445L), cache)) { IOUtils.readFully(in, false); } ManifestSidecar.Selection all = ManifestSidecar.select( - golden(), - goldenMeta(), + testSidecar(), + testMeta(), RowRangeIndex.create( Collections.singletonList(new Range(0, Long.MAX_VALUE)))); try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { @@ -842,8 +912,8 @@ void truncatedCoalescedReadsDoNotPopulateTheBlockCache() throws Exception { new SegmentsCache<>(1024, MemorySize.ofMebiBytes(1), 400, null, false); ManifestSidecar.Selection all = ManifestSidecar.select( - golden(), - goldenMeta(), + testSidecar(), + testMeta(), RowRangeIndex.create( Collections.singletonList(new Range(0, Long.MAX_VALUE)))); try (InputStream in = ManifestSidecar.openManifest(io, path, all, cache)) { @@ -870,8 +940,8 @@ void evictedBlocksAreReadAgainWithinTheSharedBudget() throws Exception { new SegmentsCache<>(1024, MemorySize.ofBytes(1300), 400, null, false); ManifestSidecar.Selection all = ManifestSidecar.select( - golden(), - goldenMeta(), + testSidecar(), + testMeta(), RowRangeIndex.create( Collections.singletonList(new Range(0, Long.MAX_VALUE)))); for (int round = 0; round < 2; round++) { @@ -985,8 +1055,8 @@ void blockShortReadsAndTruncation() throws Exception { Path path = new Path(temp.toString(), "manifest-golden"); ManifestSidecar.Selection selected = ManifestSidecar.select( - golden(), - goldenMeta(), + testSidecar(), + testMeta(), RowRangeIndex.create( Collections.singletonList(new Range(0, Long.MAX_VALUE)))); for (int bodyLength : new int[] {400, 399}) { diff --git a/paimon-core/src/test/resources/compatibility/manifest-sidecar-v1 b/paimon-core/src/test/resources/compatibility/manifest-sidecar-v1 new file mode 100644 index 0000000000000000000000000000000000000000..53bf4c421e3f40e628d87ea1605a54fe1f56bc8c GIT binary patch literal 2107 zcmWIW4R&U<^iRrSWD!X$E6UeP&QD2A=Ey6}$q@tdi<2`_a}#-#a`N-il^D>0fDBNC z2?((Q9k3)+iWy3C18E5$E=VjY$t=mt&(k#q%b+M`!>QN|Q!yt_#g>?gd7z3xG^#TU zjX+W;&fv$b*c3xCED$g~2sRkiWLBujn2Ny$qblaWsn`N6gW_h8i6DT*gN7h+lo;a! zi68+KlNg~Am~I3cgz82XoQh4rGAP!vBUuSzqFBoXmBiu!0}REyxD|tw6_@421_uE~ zK>-FvG~f%>389z|e3E18YhWZ7Ga3mn@S*~N295@f|LnN&sRjpJj?-sgfQ3Go20D%p z9|T&@h-y18LF;cfG%(_^AC{cKG!gbQk!t^&h6DKQhm{v#nh5)uNwxoPLjw~o{{s^T zEI|WlDkKgTQeD^AcmSvCfT;sket>BrLWh-9`{y?{Fyjgxp#87{8cY*mKO3p`Z)iM# z&wf}50;Y+upPf|u4>mTi;PO8(b-;>bAWenT!9l9)E;Sy&={jKSz-mM=O+@H$l4}3G H^vkXQhZ{F? literal 0 HcmV?d00001 diff --git a/paimon-core/src/test/resources/manifest-sidecar.txt b/paimon-core/src/test/resources/manifest-sidecar.txt deleted file mode 100644 index 0597894b4e0a..000000000000 --- a/paimon-core/src/test/resources/manifest-sidecar.txt +++ /dev/null @@ -1,23 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -avroHeader=T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAA -partitionA=AAAAAgAAAAAAAAAABwAAAAAAAABsZWZ0AAAAhA== -partitionB=AAAAAgACAAAAAAAACQAAAAAAAAAAAAAAAAAAAA== -index=UE1TQwE5T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAAAAM5ZAMAARMAAAAAAAAAAAAAAAAAAAAYAgkLAJ0ByAECAAEZAAAAAP////4AAAeBzDhsZQIE49jh4YzwAQDlAmQCAAEbAAAAAAAAABR//////////wIE5/////////9/AOSxTD8YXVtvciRAimgk5ETdJf+5DDmiuqaBJJiQ1/d3 -indexWithPartitions=UE1TQwE5T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAAAhwAAAACAAAAAAAAAAAHAAAAAAAAAGxlZnQAAACEHAAAAAIAAgAAAAAAAAkAAAAAAAAAAAAAAAAAAAADOWQDAQMCAAEBEwAAAAAAAAAAAAAAAAAAABgCCQsAnQHIAQIBAwIAAQEZAAAAAP////4AAAeBzDhsZQIE49jh4YzwAQDlAmQCAQMCAAEBGwAAAAAAAAAUf/////////8CBOf/////////fwCSLQchM3WaqhcAPeGxNUhGyS2SFk+gI+SVQkxbFyfFrw== -indexWithBuckets=UE1TQwE5T2JqAQQUYXZyby5jb2RlYwhudWxsFmF2cm8uc2NoZW1hDCJsb25nIgAAAAAAAAAAAAAAAAAAAAAAAhwAAAACAAAAAAAAAAAHAAAAAAAAAGxlZnQAAACEHAAAAAIAAgAAAAAAAAkAAAAAAAAAAAAAAAAAAAADOWQDAQMCAAEBEwAAAAAAAAAAAAAAAAAAABgCCQsBBgIBAAIICJ0ByAECAQMCAAEBGQAAAAD////+AAAHgcw4bGUCBOPY4eGM8AEBBgICAAIICOUCZAIBAwIAAQEbAAAAAAAAABR//////////wIE5/////////9/AQYCAAMCAgYv5qxBaT4YRIj+XDWEvw1K7HvPfhYW1Rv5XhKZtzlw/A== From 1cf280cca081c101e57927bc0749337a7540d245 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BB=9F=E5=BC=8B?= Date: Wed, 16 Sep 2026 16:08:02 +0800 Subject: [PATCH 14/14] Fix minus --- docs/docs/concepts/spec/manifest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md index 763d64b1ef86..ada9baf1cee1 100644 --- a/docs/docs/concepts/spec/manifest.md +++ b/docs/docs/concepts/spec/manifest.md @@ -189,7 +189,7 @@ represented by its entries: ```text partitionPayload - intsDeltaPayload + intsDeltaPayload // dictionary IDs, base = 0 ``` An ID is the zero-based position of a complete tuple in the sidecar's shared dictionary.