Search before asking
Paimon version
master @ bbae07d
Compute Engine
Engine-agnostic. Any catalog reading an object table on S3/OSS/OBS through the plugin loader jars (PluginFileIO), through the REST catalog (ResolvingFileIO), or with local-cache.enabled (CachingFileIO).
Minimal reproduce step
- Create an object table on
s3://bucket/prefix whose prefix has D sub-prefixes.
- Refresh it.
ObjectTableImpl calls fileIO.listFilesIterative(location, true).
HadoopCompliantFileIO.listFilesIterative (paimon-s3-impl, oss-impl, obs-impl, jindo) overrides the method with one flat paginated listing. PluginFileIO, ResolvingFileIO and CachingFileIO do not forward it, so the FileIO interface default runs on the wrapper: D+1 listStatus calls, one per directory, each materialized in memory.
What doesn't meet your expectations?
The wrappers should forward listFilesIterative to the inner FileIO, as RESTTokenFileIO does since #9071 and as all four wrappers do for tryToWriteAtomic since #9034. Expected one flat listing per refresh.
Anything else?
The plugin path also sits beneath RESTTokenFileIO, so #9071's forward lands on a PluginFileIO that still runs the default.
Are you willing to submit a PR?
Search before asking
Paimon version
master @ bbae07d
Compute Engine
Engine-agnostic. Any catalog reading an object table on S3/OSS/OBS through the plugin loader jars (
PluginFileIO), through the REST catalog (ResolvingFileIO), or withlocal-cache.enabled(CachingFileIO).Minimal reproduce step
s3://bucket/prefixwhose prefix has D sub-prefixes.ObjectTableImplcallsfileIO.listFilesIterative(location, true).HadoopCompliantFileIO.listFilesIterative(paimon-s3-impl, oss-impl, obs-impl, jindo) overrides the method with one flat paginated listing.PluginFileIO,ResolvingFileIOandCachingFileIOdo not forward it, so theFileIOinterface default runs on the wrapper: D+1listStatuscalls, one per directory, each materialized in memory.What doesn't meet your expectations?
The wrappers should forward
listFilesIterativeto the inner FileIO, asRESTTokenFileIOdoes since #9071 and as all four wrappers do fortryToWriteAtomicsince #9034. Expected one flat listing per refresh.Anything else?
The plugin path also sits beneath
RESTTokenFileIO, so #9071's forward lands on aPluginFileIOthat still runs the default.Are you willing to submit a PR?