Skip to content

Core: Add UTs for ManifestEntriesTableTaskParser class - #18125

Open
pbajpai21 wants to merge 2 commits into
apache:mainfrom
pbajpai21:test-manifest-entries-table-task-parser
Open

pbajpai21 wants to merge 2 commits into
apache:mainfrom
pbajpai21:test-manifest-entries-table-task-parser

Conversation

@pbajpai21

Copy link
Copy Markdown

Added UTs for ManifestEntriesTableTaskParser, which previously had no test coverage.

The test file TestManifestEntriesTableTaskParser mirrors the existing test
TestFilesTableTaskParser:

  • nullCheck — validates the toJson/fromJson precondition guards.
  • invalidJsonNode — validates rejection of non-object JSON nodes.
  • testParser — full round-trip through ScanTaskParser (task → JSON → task),
    asserting both the serialized JSON and field-by-field task equality.

@github-actions github-actions Bot added the core label Sep 15, 2026
@pbajpai21

Copy link
Copy Markdown
Author

@ebyhr @szehon-ho Please review the PR when you have time. Thank you.

@mkroll-db mkroll-db left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest 2 minor improvements.

Comment on lines +81 to +114
private BaseEntriesTable.ManifestReadTask createTask() {
Schema schema = TestBase.SCHEMA;
HadoopFileIO fileIO = new HadoopFileIO();
fileIO.initialize(ImmutableMap.of("k1", "v1", "k2", "v2"));
Map<Integer, PartitionSpec> specsById =
PartitionUtil.indexSpecs(
Arrays.asList(PartitionSpec.builderFor(schema).bucket("data", 16).build()));
ManifestFile manifestFile = TestManifestFileParser.createManifestFile();
return new BaseEntriesTable.ManifestReadTask(
schema, fileIO, specsById, manifestFile, schema, Expressions.equal("id", 1));
}

private String taskJson() {
return "{\"task-type\":\"manifest-entries-task\","
+ "\"schema\":{\"type\":\"struct\",\"schema-id\":0,\"fields\":[{"
+ "\"id\":3,\"name\":\"id\",\"required\":true,\"type\":\"int\"},"
+ "{\"id\":4,\"name\":\"data\",\"required\":true,\"type\":\"string\"}]},"
+ "\"file-io\":{\"io-impl\":\"org.apache.iceberg.hadoop.HadoopFileIO\","
+ "\"properties\":{\"k1\":\"v1\",\"k2\":\"v2\"}},"
+ "\"partition-specs\":[{\"spec-id\":0,\"fields\":[{"
+ "\"name\":\"data_bucket\",\"transform\":\"bucket[16]\",\"source-id\":4,\"field-id\":1000}]}],"
+ "\"manifest-file\":{\"path\":\"/path/input.m0.avro\","
+ "\"length\":5878,\"partition-spec-id\":0,\"content\":0,\"sequence-number\":1,\"min-sequence-number\":2,"
+ "\"added-snapshot-id\":12345678901234567,"
+ "\"added-files-count\":1,\"existing-files-count\":3,\"deleted-files-count\":0,"
+ "\"added-rows-count\":10,\"existing-rows-count\":30,\"deleted-rows-count\":0,"
+ "\"partition-field-summary\":[{\"contains-null\":true,\"contains-nan\":false,"
+ "\"lower-bound\":\"0A000000\",\"upper-bound\":\"64000000\"}],\"key-metadata\":\"DB030000\","
+ "\"first-row-id\":10},"
+ "\"projection\":{\"type\":\"struct\",\"schema-id\":0,\"fields\":[{"
+ "\"id\":3,\"name\":\"id\",\"required\":true,\"type\":\"int\"},"
+ "{\"id\":4,\"name\":\"data\",\"required\":true,\"type\":\"string\"}]},"
+ "\"residual-filter\":{\"type\":\"eq\",\"term\":\"id\",\"value\":1}}";
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two suggestions to strengthen the round-trip test:

  1. Use a projection that differs from the table schema.
  2. Include two partition specs to verify collection serialization and deserialization.
Suggested change
private BaseEntriesTable.ManifestReadTask createTask() {
Schema schema = TestBase.SCHEMA;
HadoopFileIO fileIO = new HadoopFileIO();
fileIO.initialize(ImmutableMap.of("k1", "v1", "k2", "v2"));
Map<Integer, PartitionSpec> specsById =
PartitionUtil.indexSpecs(
Arrays.asList(PartitionSpec.builderFor(schema).bucket("data", 16).build()));
ManifestFile manifestFile = TestManifestFileParser.createManifestFile();
return new BaseEntriesTable.ManifestReadTask(
schema, fileIO, specsById, manifestFile, schema, Expressions.equal("id", 1));
}
private String taskJson() {
return "{\"task-type\":\"manifest-entries-task\","
+ "\"schema\":{\"type\":\"struct\",\"schema-id\":0,\"fields\":[{"
+ "\"id\":3,\"name\":\"id\",\"required\":true,\"type\":\"int\"},"
+ "{\"id\":4,\"name\":\"data\",\"required\":true,\"type\":\"string\"}]},"
+ "\"file-io\":{\"io-impl\":\"org.apache.iceberg.hadoop.HadoopFileIO\","
+ "\"properties\":{\"k1\":\"v1\",\"k2\":\"v2\"}},"
+ "\"partition-specs\":[{\"spec-id\":0,\"fields\":[{"
+ "\"name\":\"data_bucket\",\"transform\":\"bucket[16]\",\"source-id\":4,\"field-id\":1000}]}],"
+ "\"manifest-file\":{\"path\":\"/path/input.m0.avro\","
+ "\"length\":5878,\"partition-spec-id\":0,\"content\":0,\"sequence-number\":1,\"min-sequence-number\":2,"
+ "\"added-snapshot-id\":12345678901234567,"
+ "\"added-files-count\":1,\"existing-files-count\":3,\"deleted-files-count\":0,"
+ "\"added-rows-count\":10,\"existing-rows-count\":30,\"deleted-rows-count\":0,"
+ "\"partition-field-summary\":[{\"contains-null\":true,\"contains-nan\":false,"
+ "\"lower-bound\":\"0A000000\",\"upper-bound\":\"64000000\"}],\"key-metadata\":\"DB030000\","
+ "\"first-row-id\":10},"
+ "\"projection\":{\"type\":\"struct\",\"schema-id\":0,\"fields\":[{"
+ "\"id\":3,\"name\":\"id\",\"required\":true,\"type\":\"int\"},"
+ "{\"id\":4,\"name\":\"data\",\"required\":true,\"type\":\"string\"}]},"
+ "\"residual-filter\":{\"type\":\"eq\",\"term\":\"id\",\"value\":1}}";
}
private BaseEntriesTable.ManifestReadTask createTask() {
Schema schema = TestBase.SCHEMA;
Schema projection = schema.select("data");
HadoopFileIO fileIO = new HadoopFileIO();
fileIO.initialize(ImmutableMap.of("k1", "v1", "k2", "v2"));
Map<Integer, PartitionSpec> specsById =
PartitionUtil.indexSpecs(
Arrays.asList(
PartitionSpec.builderFor(schema).bucket("data", 16).build(),
PartitionSpec.builderFor(schema).withSpecId(1).bucket("id", 4).build()));
ManifestFile manifestFile = TestManifestFileParser.createManifestFile();
return new BaseEntriesTable.ManifestReadTask(
schema, fileIO, specsById, manifestFile, projection, Expressions.equal("id", 1));
}
private String taskJson() {
return "{\"task-type\":\"manifest-entries-task\","
+ "\"schema\":{\"type\":\"struct\",\"schema-id\":0,\"fields\":[{"
+ "\"id\":3,\"name\":\"id\",\"required\":true,\"type\":\"int\"},"
+ "{\"id\":4,\"name\":\"data\",\"required\":true,\"type\":\"string\"}]},"
+ "\"file-io\":{\"io-impl\":\"org.apache.iceberg.hadoop.HadoopFileIO\","
+ "\"properties\":{\"k1\":\"v1\",\"k2\":\"v2\"}},"
+ "\"partition-specs\":[{\"spec-id\":0,\"fields\":[{"
+ "\"name\":\"data_bucket\",\"transform\":\"bucket[16]\",\"source-id\":4,\"field-id\":1000}]},"
+ "{\"spec-id\":1,\"fields\":[{"
+ "\"name\":\"id_bucket\",\"transform\":\"bucket[4]\",\"source-id\":3,\"field-id\":1000}]}],"
+ "\"manifest-file\":{\"path\":\"/path/input.m0.avro\","
+ "\"length\":5878,\"partition-spec-id\":0,\"content\":0,\"sequence-number\":1,\"min-sequence-number\":2,"
+ "\"added-snapshot-id\":12345678901234567,"
+ "\"added-files-count\":1,\"existing-files-count\":3,\"deleted-files-count\":0,"
+ "\"added-rows-count\":10,\"existing-rows-count\":30,\"deleted-rows-count\":0,"
+ "\"partition-field-summary\":[{\"contains-null\":true,\"contains-nan\":false,"
+ "\"lower-bound\":\"0A000000\",\"upper-bound\":\"64000000\"}],\"key-metadata\":\"DB030000\","
+ "\"first-row-id\":10},"
+ "\"projection\":{\"type\":\"struct\",\"schema-id\":0,\"fields\":[{"
+ "\"id\":4,\"name\":\"data\",\"required\":true,\"type\":\"string\"}]},"
+ "\"residual-filter\":{\"type\":\"eq\",\"term\":\"id\",\"value\":1}}";
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mkroll-db, Thank you, great suggestions - applied both. Using a projection that differs from the table schema (data only) and adding a second partition spec makes the round-trip actually verify that the projection and the spec collection are serialized/deserialized independently, rather than passing coincidentally.

Verified locally: test green, spotless and checkstyle clean. Appreciate the review...

@mkroll-db mkroll-db left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No more suggestions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants