What would you like to happen?
TableRowToStorageApiProto.messageFromMap and messageFromTableRow take an allowMissingRequiredFields parameter. When it is true, the conversion already skips the per-field missing-required check. It then calls DynamicMessage.Builder.build(), which runs a full recursive isInitialized() walk over the message tree and throws UninitializedMessageException if any required field is unset.
That has two consequences:
- Cost. The walk visits every field of every nested message on every record. On a production Dataflow pipeline writing rows with 1,059 schema fields and nesting depth 10 through
BigQueryIO.write() with STORAGE_WRITE_API and withAutoSchemaUpdate(true), the destination descriptor is built with no required fields, so the walk can never fail, yet it still runs. JFR attributed a large share of the conversion step's CPU and allocation (descriptor list wrappers and iterators) to it.
- Semantics. The parameter's promise is kept during conversion and broken at build time. A caller that passes
true with a descriptor that still has required fields gets the missing-field exception from build() anyway.
Proposal
At both build() sites:
return allowMissingRequiredFields ? builder.buildPartial() : builder.build();
No new method, no new parameter, no change to the false path. Both in-tree callers that pass true already use descriptors built without required fields (StorageApiDynamicDestinationsTableRow.TableRowConverter when autoSchemaUpdates is on, and AppendClientInfo.encodeUnknownFields), so for them this is a pure optimization with byte-identical output.
Behavior change to declare: an out-of-tree caller passing true with a descriptor that still has required fields would receive a partial message instead of a conversion-time UninitializedMessageException. The BigQuery service still rejects such a row. Given the parameter's name, this is the documented behavior being honored, but it should be called out in CHANGES.md. If maintainers prefer, the change can be guarded with a one-time check that the descriptor has no required fields.
Measurements
On the production-shaped workload above, measured locally over a 280-record corpus with production options (five JVM forks, census-weighted): this change alone reduced whole-record CPU by 12.3%. Combined with the descriptor lookup change in #40188, 35.9% locally. The combined pair on x86 Dataflow at equal, fixed capacity (35 × n2-standard-4, Streaming Engine, backlogged input): 33.6% lower CPU per record and 46% higher drain throughput, with exact offset-set parity.
Related
#40188 addresses the other independent cost in the same conversion (per-field Descriptor.findFieldByName lookups). The two changes touch different regions of TableRowToStorageApiProto.java and can land in either order.
A PR implementing this is ready.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
What would you like to happen?
TableRowToStorageApiProto.messageFromMapandmessageFromTableRowtake anallowMissingRequiredFieldsparameter. When it istrue, the conversion already skips the per-field missing-required check. It then callsDynamicMessage.Builder.build(), which runs a full recursiveisInitialized()walk over the message tree and throwsUninitializedMessageExceptionif any required field is unset.That has two consequences:
BigQueryIO.write()withSTORAGE_WRITE_APIandwithAutoSchemaUpdate(true), the destination descriptor is built with no required fields, so the walk can never fail, yet it still runs. JFR attributed a large share of the conversion step's CPU and allocation (descriptor list wrappers and iterators) to it.truewith a descriptor that still has required fields gets the missing-field exception frombuild()anyway.Proposal
At both
build()sites:No new method, no new parameter, no change to the
falsepath. Both in-tree callers that passtruealready use descriptors built without required fields (StorageApiDynamicDestinationsTableRow.TableRowConverterwhenautoSchemaUpdatesis on, andAppendClientInfo.encodeUnknownFields), so for them this is a pure optimization with byte-identical output.Behavior change to declare: an out-of-tree caller passing
truewith a descriptor that still has required fields would receive a partial message instead of a conversion-timeUninitializedMessageException. The BigQuery service still rejects such a row. Given the parameter's name, this is the documented behavior being honored, but it should be called out inCHANGES.md. If maintainers prefer, the change can be guarded with a one-time check that the descriptor has no required fields.Measurements
On the production-shaped workload above, measured locally over a 280-record corpus with production options (five JVM forks, census-weighted): this change alone reduced whole-record CPU by 12.3%. Combined with the descriptor lookup change in #40188, 35.9% locally. The combined pair on x86 Dataflow at equal, fixed capacity (35 × n2-standard-4, Streaming Engine, backlogged input): 33.6% lower CPU per record and 46% higher drain throughput, with exact offset-set parity.
Related
#40188 addresses the other independent cost in the same conversion (per-field
Descriptor.findFieldByNamelookups). The two changes touch different regions ofTableRowToStorageApiProto.javaand can land in either order.A PR implementing this is ready.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components