[Bug] PyPaimon nested projection returns incorrect MAP values and nullability

[wangzhigang1999]

Search before asking

• I searched existing issues for these projection failures and found no matching report.

Paimon version

Master at bb5498ff3, using paimon-python from the source checkout.

Compute Engine

PyPaimon, Python 3.12.6, PyArrow 19.0.1, macOS. The example uses a local filesystem table with Parquet files.

Minimal reproduce step

Read two distinct MAP keys, foo and .foo, from a one-row table:

import tempfile

import pyarrow as pa

from pypaimon import CatalogFactory, Schema


with tempfile.TemporaryDirectory() as warehouse:
    catalog = CatalogFactory.create({"warehouse": warehouse})
    catalog.create_database("default", False)
    data = pa.table(
        {
            "attrs": pa.array(
                [[("foo", 100), (".foo", 107)]],
                type=pa.map_(pa.string(), pa.int64()),
            )
        }
    )
    catalog.create_table(
        "default.repro",
        Schema.from_pyarrow_schema(
            data.schema, options={"bucket": "-1", "file.format": "parquet"}
        ),
        False,
    )
    table = catalog.get_table("default.repro")
    wb = table.new_batch_write_builder()
    writer = wb.new_write()
    try:
        writer.write_arrow(data)
        wb.new_commit().commit(writer.prepare_commit())
    finally:
        writer.close()

    rb = table.new_read_builder().with_projection(["attrs['foo']", "attrs['.foo']"])
    result = rb.new_read().to_arrow(rb.new_scan().plan().splits()).to_pydict()
    print(result)
    assert result == {"attrs_foo": [100], "attrs__foo": [107]}

What doesn't meet your expectations?

Expected:

{'attrs_foo': [100], 'attrs__foo': [107]}

Actual:

{'attrs_foo': [100], 'attrs__foo': [100]}

The read returns the value for foo in both columns without reporting an error. Reading only attrs['.foo'] raises ArrowInvalid. I reproduced both cases with Parquet and row files.

Anything else?

I also found two projection failures in the same checkout:

• MAP selector prefix matching: with MAP columns named attrs and attrs['x, and attrs containing ('x[0]', 42), attrs['x[0]'] drops the requested column, while attrs["x[0]"] returns 42. A longer field-name prefix wins before the remaining selector is validated.
• ROW nullability: for a nullable r: ROW<x BIGINT NOT NULL>, writing r = NULL and r = {x: 7} then projecting r.x returns [None, 7] with a not null output field. Writing this Arrow result to Parquet raises Column 'r_x' is declared non-nullable but contains nulls.

For the first example, the selected MAP keys become struct fields during reading. Passing .foo as a string to pyarrow.compute.struct_field interprets it as a field path. The projection reader needs to preserve the literal key name.

Are you willing to submit a PR?

• I'm willing to submit a PR!