Search before asking
Paimon version
2.2-SNAPSHOT, commit 10cf6ebc7c4f0ba6066d32d8aed2b0b85e0b8349.
Compute Engine
PyPaimon: Python 3.11.13, PyArrow 19.0.1. Java API: JDK 8, same Paimon revision.
Minimal reproduce step
Write one row with the default table options:
from datetime import date
from pathlib import Path
from tempfile import TemporaryDirectory
import pyarrow as pa
from pypaimon import CatalogFactory, Schema
with TemporaryDirectory() as warehouse:
catalog = CatalogFactory.create({"warehouse": warehouse})
catalog.create_database("default", False)
arrow_schema = pa.schema([("id", pa.int64()), ("day", pa.date32())])
catalog.create_table(
"default.t",
Schema.from_pyarrow_schema(arrow_schema, partition_keys=["day"]),
False,
)
table = catalog.get_table("default.t")
builder = table.new_batch_write_builder()
writer, commit = builder.new_write(), builder.new_commit()
try:
writer.write_arrow(
pa.table({"id": [1], "day": [date(1970, 1, 2)]}, schema=arrow_schema)
)
commit.commit(writer.prepare_commit())
finally:
writer.close()
commit.close()
partitions = sorted(p.name for p in Path(table.table_path).glob("day=*"))
print(partitions)
assert partitions == ["day=1"], partitions
The assertion fails:
AssertionError: ['day=1970-01-02']
What doesn't meet your expectations?
With partition.legacy-name=true (the default), Java uses day=1, while Python writes and reads day=1970-01-02. Cross-language reads fail. Python should honor the same option.
Anything else?
A strict reader fix would break existing Python-written tables after upgrading. Could we correct new writes while retaining a file-not-found fallback to historical Python paths?
Are you willing to submit a PR?
Search before asking
Paimon version
2.2-SNAPSHOT, commit
10cf6ebc7c4f0ba6066d32d8aed2b0b85e0b8349.Compute Engine
PyPaimon: Python 3.11.13, PyArrow 19.0.1. Java API: JDK 8, same Paimon revision.
Minimal reproduce step
Write one row with the default table options:
The assertion fails:
What doesn't meet your expectations?
With
partition.legacy-name=true(the default), Java usesday=1, while Python writes and readsday=1970-01-02. Cross-language reads fail. Python should honor the same option.Anything else?
A strict reader fix would break existing Python-written tables after upgrading. Could we correct new writes while retaining a file-not-found fallback to historical Python paths?
Are you willing to submit a PR?