What happened?
The Go SDK reflection row coder encodes int32 struct fields with EncodeVarInt(rv.Int()), the signed 64 bit varint, so a negative value takes 10 bytes: -1 becomes ff ff ff ff ff ff ff ff ff 01 (sdks/go/pkg/beam/core/graph/coder/row_encoder.go, the reflect.Int32 case shared with the other int kinds).
The other SDKs encode the INT32 schema type as the varint of the unsigned 32 bit form of the value, 5 bytes at most:
- Java
VarIntCoder calls VarInt.encode(int), which masks with 0xFFFFFFFFL, so -1 is ff ff ff ff 0f. VarInt.decodeInt throws varint overflow for the Go bytes because they decode to a negative long.
- Python
VarInt32Coder writes v & 0xFFFFFFFF (write_var_int32 in slow_stream.py) and read_var_int32 packs the decoded value with struct.pack('<I', v), which raises for the negative value produced by the Go bytes.
So a Go row with a negative int32 field cannot be read by the Java or Python SDK. Positive values are unaffected, and Go can already read the 5 byte form because reflect.Value.SetInt truncates to 32 bits.
This is the INT32 counterpart of #40151 (INT16). Found while working on #39684.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
The Go SDK reflection row coder encodes
int32struct fields withEncodeVarInt(rv.Int()), the signed 64 bit varint, so a negative value takes 10 bytes:-1becomesff ff ff ff ff ff ff ff ff 01(sdks/go/pkg/beam/core/graph/coder/row_encoder.go, thereflect.Int32case shared with the other int kinds).The other SDKs encode the INT32 schema type as the varint of the unsigned 32 bit form of the value, 5 bytes at most:
VarIntCodercallsVarInt.encode(int), which masks with0xFFFFFFFFL, so-1isff ff ff ff 0f.VarInt.decodeIntthrowsvarint overflowfor the Go bytes because they decode to a negative long.VarInt32Coderwritesv & 0xFFFFFFFF(write_var_int32inslow_stream.py) andread_var_int32packs the decoded value withstruct.pack('<I', v), which raises for the negative value produced by the Go bytes.So a Go row with a negative
int32field cannot be read by the Java or Python SDK. Positive values are unaffected, and Go can already read the 5 byte form becausereflect.Value.SetInttruncates to 32 bits.This is the INT32 counterpart of #40151 (INT16). Found while working on #39684.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components