fix: honor the S3 profile name and file and Hadoop's addressing mode for custom endpoints

[dwsmith1983]

Which issue does this PR close?

Closes #4245, closes #2802.

Rationale for this change

Two gaps in how the native S3 store is configured from fs.s3a.* settings.

The profile credentials provider ignored fs.s3a.auth.profile.name and fs.s3a.auth.profile.file, so a job that selects a named profile or a non-default credentials file on the Hadoop side got the SDK defaults on the native side.

fs.s3a.path.style.access was applied inverted: true set object_store's virtual_hosted_style_request to true and then appended /bucket to the endpoint, which object_store, treating a virtual-hosted endpoint as already containing the bucket, sent as a path-style URL anyway. The net effect was that every custom endpoint was addressed path-style whatever the flag said, and virtual-hosted addressing (bucket.host) was never produced.

What changes are included in this PR?

• CredentialProviderMetadata::Profile carries name and file, read through the existing per-bucket config lookup with blanks treated as unset, and passed to the SDK builder. The file is loaded in credentials-file format, which is what Hadoop's ProfileAWSCredentialsProvider does; with no file the SDK default applies as before. aws-runtime becomes a direct dependency because the file-kind types re-exported by aws_config are deprecated aliases; it was already in the tree.
• path.style.access is parsed the way Hadoop's Configuration.getBoolean parses it (default false, non-boolean text falls back to the default), virtual_hosted_style_request is its negation and is always passed, and normalize_endpoint returns the endpoint together with the effective mode so the two cannot disagree: virtual-hosted rebuilds scheme://bucket.host[:port][/path], path-style leaves the endpoint alone for object_store to append the bucket, and an IP-literal host forces path-style the way the AWS SDK's endpoint rules do, so http://127.0.0.1:9000 keeps working without the flag. localhost is not special-cased, matching Hadoop. The s3.amazonaws.com skip is unchanged.
• The data sources page lists the two profile keys and explains the addressing rule.

Behavior change: a custom fs.s3a.endpoint with fs.s3a.path.style.access unset is now addressed virtual-hosted, as Hadoop S3A addresses it. Deployments on MinIO, Ceph RGW or similar services behind a hostname that relied on the previous always-path-style behavior need fs.s3a.path.style.access=true, which Hadoop already requires for those services; IP-address endpoints keep working either way. Vendor alias schemes are unaffected because the JVM side already synthesizes the flag for them.

How are these changes tested?

52 unit tests in the S3 module, 11 of them written first and failing on the previous code (the profile metadata carried no name or file; path.style.access unset produced no flag; a hostname endpoint was never rewritten; an IP endpoint was rewritten to bucket.127.0.0.1). Coverage: the flag unset, true, false, mixed case with whitespace, and an invalid value; per-bucket override of the flag and of the endpoint, each against a global value set the other way; the s3.amazonaws.com skip in both modes; scheme-less, http://, port, trailing slash, path suffix, an AWS regional host and a dotted bucket name; IPv4 and IPv6 hosts with and without a port; create_store called with a custom endpoint in each mode and with an IP endpoint; profile name only, file only, both, neither, blank, trimmed and per-bucket, plus the provider chain building with the profile provider among others. Two existing tests that had encoded the inverted flag were replaced; one that asserted an empty config now asserts the endpoint key is absent, since the flag is always present.

The create_store calls show object_store accepts each flag and endpoint pair but do not issue a request, since object_store parses the endpoint on first use. The four Scala tests that set an endpoint either set path-style access or are pure config-translation tests, so none needed changing. Full core crate suite passes, clippy and fmt clean.

[sunchao]

Reviewed 1917adca against base db790673. One verified P2 finding: the new default addressing mode breaks HTTPS buckets whose names contain dots.

Correctness

Previously, the native S3 configuration ignored the two profile override keys and inverted fs.s3a.path.style.access. This change carries the selected profile name/file through provider metadata, preserves bucket-specific precedence, and returns an endpoint together with its effective addressing mode. The maintained Spark 3.5/4.0 sources pass spark.hadoop.* values into Hadoop configuration. Comet's existing prefix-based extraction carries these S3 options through to native code. The new boolean parsing matches Hadoop's trimmed, case-insensitive true/false handling and default-false behavior.

The addressing change needs one correction before merge. Setting virtual_hosted_style_request = !path_style_access forces virtual hosting for dotted bucket names over HTTPS, including ordinary AWS endpoints with no override. For review.dotted.bucket, the resulting host is review.dotted.bucket.s3.us-east-1.amazonaws.com. The AWS SDK chooses the path-style URL instead, because the dotted hostname does not match S3's wildcard certificate. The inline P2 requests the same eligibility check for default and custom HTTPS endpoints.

I reproduced the endpoint difference without credentials or storage requests. A Rust harness using the HEAD configuration expressions and the local object_store 0.13.2 endpoint expression produced the dotted hostname. The actual offline AWS Java SDK 2.29.52 endpoint resolver, the version declared by Hadoop 3.4.2, selected path style. Normal bucket names, explicit path style and synthetic profile/bucket precedence checks behaved as expected. These are isolated configuration checks, not native/JNI or live S3 tests.

At 2026-09-12 20:03:22 UTC, only the label check succeeded. CI, CodeQL, the Delta gate and title validation were awaiting workflow approval. The cached merge has the assigned base/head parents and HEAD's tree, but no product CI execution can be credited. The author's reported 52 S3 tests/full-core pass was not rerun locally. Exact locked aws-config 1.12.0 and aws-runtime 1.9.2 source was unavailable locally, so SDK profile-file loading and refresh remain unverified. Maintained Spark 3.4/4.1 source gaps also remain. The final publication check confirmed that the head, base and discussion were unchanged after the temporary API rate limit cleared.

Performance

The added profile string handling and URL parsing occur during store/provider construction. The existing store cache includes the full configuration hash, so changing a profile name or file selects a different cached store. This PR does not add work to the object-read loop or alter credential expiry caching.

The endpoint result keeps normalization to one pass, and the profile description allocates only during construction-time logging. I found no separate verified performance issue. No benchmark or speedup claim is established by this review, and an expression microbenchmark is not applicable to this configuration change.

Design

Returning the normalized endpoint and addressing mode together is a useful safeguard against the original disagreement between those two values. The missing piece is deciding whether a bucket is eligible for virtual hosting before producing either result. That decision must also run when the endpoint is omitted, where object_store constructs the normal AWS URL.

Profile name and file are independently optional, with bucket values taking precedence over global values. The PR preserves the existing provider chain and expiry wrapper. Its metadata tests demonstrate option selection but do not establish actual SDK file precedence or credential renewal. The review therefore keeps those validation boundaries explicit.

Abstraction & complexity

The small NormalizedEndpoint type and blank-filtering helper are proportionate to the change. Configuration lookup remains centralized, and the new direct aws-runtime dependency supplies file-kind types already present in the dependency graph.

No broader provider or endpoint framework is needed. The actionable change is to extend the addressing decision with HTTPS bucket eligibility and test the final URL, rather than relying on a configuration-map assertion or a builder that has not issued a request.

[sunchao]

Correctness

[P2] Preserve path-style addressing for dotted HTTPS buckets

Could we apply the AWS SDK's virtual-host eligibility rules before enabling this flag? With fs.s3a.endpoint.region=us-east-1 and path.style.access unset or false, a bucket such as review.dotted.bucket now becomes https://review.dotted.bucket.s3.us-east-1.amazonaws.com. BASE used path-style addressing, and the AWS SDK endpoint resolver still selects https://s3.us-east-1.amazonaws.com/review.dotted.bucket for this case. The dotted host does not match S3's wildcard TLS certificate, so this breaks native reads of otherwise valid buckets. The same eligibility issue exists for custom HTTPS endpoints. Please retain path-style addressing for dotted HTTPS buckets, including when no custom endpoint is configured, and add assertions on the resulting request URL. The current dotted-bucket test asserts the virtual-hosted string, which misses this regression.

[dwsmith1983]

Please retain path-style addressing for dotted HTTPS buckets, including when no custom endpoint is configured, and add assertions on the resulting request URL.

In 0c76e6c. A bucket name containing a dot is addressed path-style whenever the endpoint is HTTPS, the default AWS endpoint included, and stays virtual-hosted over HTTP as the SDK does. The tests assert the endpoint and flag pair for the default endpoint, an explicit HTTPS endpoint, an HTTP endpoint and a plain bucket.

[andygrove]

The two keys being read here are PROFILE_NAME and PROFILE_FILE on Hadoop's org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider, and Hadoop only applies them when fs.s3a.aws.credentials.provider names that class. When the provider is spelled software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider, which is the spelling this arm matches, Hadoop instantiates it through S3AUtils.getInstanceFromReflection, which finds no (URI, Configuration) constructor and falls through to the SDK's static create(). The Hadoop config never reaches it. So the JVM side of the job resolves the SDK default profile while the native side now resolves the configured one, and the two halves of one job authenticate as different identities.

At the same time build_aws_credential_provider_metadata has no arm for org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider, so the one provider spelling that does honor these keys on the Hadoop side hits the _ => arm and fails the native scan with Unsupported credential provider. Would it make sense to accept that FQCN as a third alias here and add it to the datasources.md row, so the configuration Hadoop actually documents works on both sides?

On the addressing change, the new paragraph in datasources.md explains the rule well but does not mention that the default is changing. Today a custom fs.s3a.endpoint is addressed path-style whatever fs.s3a.path.style.access says, so a MinIO or Ceph RGW deployment that never set the flag works right now and will start sending requests to http://<bucket>.<host> once this lands. All the user sees is a DNS failure with nothing tying it back to this config. Could we add a sentence saying deployments that relied on the previous always-path-style behavior need to set fs.s3a.path.style.access=true?

For what it is worth, the object store cache key is fine. hash_object_store_configs hashes the whole forwarded map and extractObjectStoreOptions forwards every fs.s3a.* key by prefix, so two stores differing only in profile name or file get distinct hashes. Nothing logs credentials either.

[dwsmith1983]

Would it make sense to accept that FQCN as a third alias here and add it to the datasources.md row, so the configuration Hadoop actually documents works on both sides?

Added org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider as the provider that reads the profile keys, with its own row. The SDK spellings now resolve the SDK default profile and ignore the keys, since Hadoop constructs those without its configuration, so both halves of a job resolve the same identity. A test pins that for both SDK spellings and the chain test uses the Hadoop provider.

Could we add a sentence saying deployments that relied on the previous always-path-style behavior need to set fs.s3a.path.style.access=true?

Added, right after the addressing paragraph.

[sunchao]

Re-reviewed 3f66db90 against 1d0ce5fe. The Hadoop profile-provider spelling now reads the profile keys, the SDK spellings ignore them, and the custom-endpoint migration note is present.

The existing P2 dotted HTTPS bucket finding remains unresolved. The addressing code is unchanged, and rerunning the offline SDK endpoint resolver and configuration probe confirms the mismatch.

One new P2 is inline: with the Hadoop provider and no profile-file override, native code still merges the SDK config and credentials files, whereas Hadoop reads only the credentials file. A same-name role profile in the config file can therefore change native credential resolution.

The exact aws-config 1.12.0 and aws-runtime 1.9.2 source gaps from the first review are now closed using lockfile-checksummed archives. Validation included source tracing and an isolated probe of the SDK file-selection code. No account files, credential resolution, storage requests, full native/JNI tests, or benchmarks were used. The merge tree matches HEAD. At September 15, 05:56 UTC, CI, CodeQL, and the Delta gate awaited approval with zero jobs; only labeling passed.

[sunchao]

Correctness

[P2] Keep Hadoop's default profile source credentials-only

Could we preserve Hadoop's file selection when fs.s3a.auth.profile.file is unset too? The newly supported HADOOP_PROFILE arm reaches this branch with file: None, so it leaves the Rust SDK defaults in place. In the locked SDK those defaults merge ~/.aws/config with ~/.aws/credentials. Hadoop's provider instead selects only AWS_SHARED_CREDENTIALS_FILE or ~/.aws/credentials in this case.

For example, with auth.profile.name=analytics, static credentials in the credentials file and a same-name config profile containing role_arn plus source_profile=analytics, Hadoop uses the static identity while the native SDK merges in the role and assumes it. That can change the identity or fail native reads that Hadoop can perform. Please retain the Hadoop/SDK provider distinction and select a credentials-only default for the Hadoop spelling, while preserving normal SDK defaults for the SDK spellings. A test of the selected file set when the override is absent would cover this case.

[dwsmith1983]

Could we preserve Hadoop's file selection when fs.s3a.auth.profile.file is unset too?

In 0c76e6c. The Hadoop profile provider arm reads only the credentials file, from fs.s3a.auth.profile.file, then AWS_SHARED_CREDENTIALS_FILE, then the default path, and never merges the config file, so the static identity wins in your example on both sides.

[dwsmith1983]

Please retain path-style addressing for dotted HTTPS buckets, including when no custom endpoint is configured, and add assertions on the resulting request URL.

Done. A bucket whose name contains a dot is addressed path-style whenever the endpoint is HTTPS, which covers the default AWS endpoint and a custom https:// one; over http:// it stays virtual-hosted, as the SDK does. The tests assert the endpoint and flag pair object_store receives for the default endpoint, an explicit HTTPS endpoint, an HTTP endpoint, and a plain bucket as the control. The addressing paragraph in datasources.md says the same. Apologies for the delay, this note and your earlier ones slipped past my sweep.

Please retain the Hadoop/SDK provider distinction and select a credentials-only default for the Hadoop spelling, while preserving normal SDK defaults for the SDK spellings.

Done. The profile metadata carries whether the provider is credentials-only; Hadoop's spelling is, the SDK spellings are not. With no file override, Hadoop's spelling now reads AWS_SHARED_CREDENTIALS_FILE or ~/.aws/credentials as a credentials-format file and nothing else, so a same-name role profile in the config file never applies; the SDK spellings keep the SDK defaults. A test covers the selected file with and without the override.

[sunchao]

Re-reviewed f9b1338a against a8e8157e. Both prior findings are addressed: dotted buckets now use path-style addressing over HTTPS, and Hadoop's profile provider now selects a credentials-only file without merging the SDK config.

Two new P2 findings are inline. The dotted-bucket guard also forces path style for custom HTTP endpoints because it runs before the scheme check. The credentials-only fallback uses native HOME instead of Hadoop's JVM user.home, so the two sides can select different files.

Validation used the current configuration functions in an isolated Rust probe, the offline AWS Java SDK 2.29.52 endpoint resolver, the locked SDK file-selection code, and Hadoop's declared Commons Lang 3.17.0 home-directory behavior. No account files, credential resolution, storage requests, full native/JNI tests, or benchmarks were used. Maintained Spark 3.5/4.0 configuration forwarding was checked. The 3.4/4.1 maintained-source gaps remain.

At September 15, 16:11 UTC, CI and CodeQL awaited approval with zero jobs. Only labeling passed. The current merge has the assigned base/head parents and the same tree as HEAD.

[sunchao]

Correctness

[P2] Apply the dotted-bucket guard after choosing the endpoint scheme

Could we limit this initial dotted-bucket fallback to the default HTTPS endpoint? With fs.s3a.endpoint=http://storage.example.test, bucket review.dotted.bucket, and path.style.access unset or false, this expression already sets the flag to false. normalize_endpoint then returns at its first path-style branch, before it can apply the scheme-sensitive rule. The native configuration produces http://storage.example.test/review.dotted.bucket, while Hadoop's AWS SDK resolver selects http://review.dotted.bucket.storage.example.test. This breaks a custom HTTP service that routes buckets by hostname. The new HTTP test calls normalize_endpoint(..., true) directly, bypassing the caller that supplies false. Please preserve virtual hosting for this HTTP case and cover it through extract_s3_config_options, including the resulting URL.

[dwsmith1983]

Please preserve virtual hosting for this HTTP case and cover it through extract_s3_config_options, including the resulting URL.

In 31fafaf. The dotted-bucket rule runs in extract_s3_config_options only when no custom endpoint is configured, since the default endpoint is HTTPS; a custom endpoint decides by its own scheme inside normalize_endpoint. The test goes through extract_s3_config_options and asserts http://review.dotted.bucket.storage.example.test for the HTTP case beside the HTTPS and default cases.

[sunchao]

Correctness

[P2] Resolve Hadoop's default credentials path from JVM user.home

Could we pass Hadoop's resolved default file into this branch instead of deriving it from the native process's HOME? When both fs.s3a.auth.profile.file and AWS_SHARED_CREDENTIALS_FILE are unset, Hadoop's provider uses SystemUtils.getUserHome(), which reads the JVM user.home property. For an executor launched with -Duser.home=/synthetic/jvm-home while HOME=/synthetic/env-home, Hadoop selects /synthetic/jvm-home/.aws/credentials but this code selects /synthetic/env-home/.aws/credentials. If HOME is absent, it selects /.aws/credentials even when the JVM has a valid home. A job can therefore load a different profile or fail native reads after Hadoop successfully loads its credentials. Please retain credentials-only loading while using the same resolved file on both sides, with a case where HOME and user.home differ.

[dwsmith1983]

Could we pass Hadoop's resolved default file into this branch instead of deriving it from the native process's HOME?

In f3758e5. The executor resolves fs.s3a.comet.default.profile.file against its own user.home or AWS_SHARED_CREDENTIALS_FILE when a native plan is created, native keeps it as a session extension and overlays it onto each scan's options, and the Hadoop provider arm takes a configured fs.s3a.auth.profile.file first, then that path. A driver and executor with different homes resolve the executor's.

[dwsmith1983]

Please preserve virtual hosting for this HTTP case and cover it through extract_s3_config_options, including the resulting URL.

Done. The dotted-bucket rule now runs in extract_s3_config_options only when no custom endpoint is configured, since the default AWS endpoint is HTTPS; a custom endpoint decides by its own scheme inside normalize_endpoint. The test now goes through extract_s3_config_options and asserts http://storage.example.test with review.dotted.bucket yields virtual hosting and the endpoint http://review.dotted.bucket.storage.example.test, next to the HTTPS and default-endpoint cases.

Could we pass Hadoop's resolved default file into this branch instead of deriving it from the native process's HOME?

Done. NativeConfig.extractObjectStoreOptions now forwards fs.s3a.comet.default.profile.file, resolved on the JVM as AWS_SHARED_CREDENTIALS_FILE or the user.home .aws/credentials, for every scheme on the fs.s3a.* surface. Native's Hadoop-provider arm takes a configured fs.s3a.auth.profile.file first, then that forwarded path; the native HOME fallback only remains for callers that forward nothing. Tests on both sides: the Scala one checks the forwarded value and that a blank env override falls back to user.home rather than HOME, and the native one checks the forwarded file is taken, a configured file wins, and the SDK spellings ignore both. Merged with main at a8e8157 first.

[sunchao]

Re-reviewed 31fafaff against 4479e722. The dotted HTTP bucket finding is fixed: the extraction-caller probe now matches the scheme-aware rule, including the original failing case.

The existing P2 profile-path finding remains for distributed execution. extractObjectStoreOptions runs during driver-side scan planning, so the new key freezes the driver's user.home or AWS_SHARED_CREDENTIALS_FILE into the plan. The executor's native provider consumes that path unchanged. With different driver/executor homes, the isolated JVM checks select /synthetic/driver-home/.aws/credentials for native code where executor-side Hadoop selects /synthetic/executor-home/.aws/credentials. Could we resolve the omitted default in the executor JVM before creating the native provider, while preserving explicit file overrides?

Validation used the full NativeConfig object in isolated JVMs, exact-source Rust configuration probes with a bounded URL test double, the actual Hadoop path selector, and the offline AWS SDK endpoint resolver. No full Spark/JNI tests, credential loading, storage requests or benchmarks ran. Maintained Spark 3.4/4.1 source gaps remain. At September 15, 18:28 UTC, CI and CodeQL awaited approval with zero jobs. Only labeling passed.

[dwsmith1983]

Could we resolve the omitted default in the executor JVM before creating the native provider, while preserving explicit file overrides?

Done. The driver no longer forwards anything for it. CometExecIterator.serializeCometSQLConfs, which runs on the executor when a native plan is created (the shuffle writer path included), now carries fs.s3a.comet.default.profile.file resolved against that executor's user.home or AWS_SHARED_CREDENTIALS_FILE; native keeps it as a session extension and overlays it onto each scan's object store options, and the Hadoop-provider arm takes a configured fs.s3a.auth.profile.file first, then that path. With different driver and executor homes the executor's wins. Tests: the serialized executor config carries the resolved path, planning on the driver forwards nothing, the overlay replaces a driver-serialized value and is a no-op without the extension, and the provider arm prefers an explicit file. Merged with main at 3868760.

[sunchao]

Rechecked f3758e59 against 4479e722, including the change since 31fafaff.

The remaining profile-path finding is addressed. The implicit file is now resolved in the executor JVM before native plan creation, then applied to Parquet and CSV scan options. Explicit global and per-bucket profile files still take precedence, and the SDK profile aliases remain separate. The prior endpoint/addressing fixes are unchanged. I found no new or remaining P1/P2 issues.

The focused reproduction now selects the executor's file for both differing user.home values and differing AWS_SHARED_CREDENTIALS_FILE values. The previous head selects the driver's file. This used the full NativeConfig, the exact serializer method with small Spark/protobuf test doubles, isolated native overlay/provider-selection code, and Hadoop's actual path selector. It did not run a full Spark/JNI workload or authenticate to storage.

CI and CodeQL for this head are still awaiting approval with zero jobs. Only labeling has passed. Maintained Spark 3.5/4.0 sources were checked. Maintained 3.4/4.1 sources remain unavailable, so this review does not claim coverage for those versions.

[sunchao]

Rechecked 629d913b against 8c229a70. This is a base-only merge: all 11 PR files and the full authored diff are unchanged from f3758e59, and all 61 changed paths match the new base. The executor-side profile-path fix, explicit-file precedence, provider distinction and endpoint/addressing fixes remain intact. No new or remaining P1/P2 findings. Keeping the existing approval.

Validation checked source and dependency equivalence, including 14 complete files across the configuration path. The prior synthetic probes were not rerun. CI and CodeQL still await approval with zero jobs. Only labeling passed. No full Spark/JNI or cloud validation is claimed. Maintained Spark 3.4/4.1 source gaps remain.

[sunchao]

Follow-up on unchanged 629d913b / 8c229a70: attempt 2 Preflight now fails Markdown formatting in datasources.md. One new P2 is inline. Cached Prettier 3.9.6 passes the base file and fails the head file, with the difference confined to the changed credentials table. CI does not record its formatter version, so I am not claiming an exact version match.

The executed merge d89e12fa has the reviewed head's exact tree. Required Checks fails downstream of Preflight, and no product build or test jobs ran in this attempt. The newer advertised merge 4c39c21d is a different tree and gets no validation credit from these jobs.

The authored diff and prior source fixes are unchanged. Keeping the existing approval while the formatting blocker is corrected. No new runtime validation was performed.

[sunchao]

Correctness

[P2] Format the changed credentials table to unblock Preflight

Could you run prettier --write docs/source/user-guide/latest/datasources.md for this table change? Attempt 2 Preflight fails Check markdown formatting on this file. Its checked-out merge d89e12fa has the reviewed head's exact tree. A read-only check with cached Prettier 3.9.6 passes the base file but fails the head/CI file, and the only formatting diff adjusts this credentials table after the longer profile description. The downstream Required Checks failure comes from Preflight, and product build/test jobs were skipped. This needs formatting before CI can validate the implementation.

[dwsmith1983]

Could you run prettier --write docs/source/user-guide/latest/datasources.md for this table change?

Done in 7b95c8f. prettier --check "**/*.md" with prettier 3.9.7 now passes on the full tree, and the branch is merged with main.

[sunchao]

Rechecked 7b95c8fbb602cdfd39e859ed9faf80f4a989e768 against 36146a87bf9ca9ca9e211b4372628ed2f9d8c8c6. The Markdown-formatting finding is fixed: cached Prettier 3.9.6 reproduces the prior-head failure, while the base, current head, and advertised merge pass for datasources.md. The current document is byte-identical to the formatter output expected for the previous head.

The new authored change is table formatting. All ten non-documentation authored changes are unchanged from the previous review. The inherited base changes preserve executor-side profile-file resolution, JNI transport, scan-option overlays, and the earlier provider/addressing fixes. No remaining P1/P2 findings in this follow-up.

CI and CodeQL currently require approval and each has zero jobs. Only the label workflow passed. This approval covers the source review and focused local formatter check. I did not run full-tree formatting, native/JNI tests, or live object-store authentication at this head.

[sunchao]

Rechecked 1f2858eb against 58ab5f61 after the main merge. All 11 authored files are byte-identical to 7b95c8fb, and the six inherited file changes match the base update. The executor-local profile-file resolution, Hadoop credentials-only loading and dotted-bucket endpoint fixes remain intact. No new or remaining P1/P2 findings.

The document still matches the previously verified Prettier output. This round used source comparisons, with no product tests or formatter rerun. CI and CodeQL require approval and have run zero jobs. The successful label workflow checked out base/main and provides no product test evidence. My existing approval remains unchanged.

[sunchao]

Summary

• Prior state and problem: Native S3 configuration inverted path.style.access and lacked Hadoop profile-name/file support.
• Design approach: Pair normalized endpoints with their addressing mode, carry profile options in provider metadata, and resolve default credentials paths on executors.
• Correctness / compatibility analysis: One introduced P2 remains: legacy mixed-case bucket names are incorrectly virtual-hosted. The previously discussed issues are addressed, although their threads remain open. Configuration forwarding was checked against Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 sources, with Hadoop and AWS SDK semantics compared.
• Key design decisions: Separating Hadoop’s credentials-only provider from SDK aliases preserves file-selection behavior. NormalizedEndpoint keeps endpoint and mode consistent without a broader abstraction.
• Implementation sketch: Executor configuration crosses JNI through a session extension and is applied to Parquet and CSV scans. Added work occurs during plan/store construction. No separate P1/P2 performance issue was established.
• Behavioral changes worth calling out: Custom hostname endpoints now default to virtual hosting. Existing MinIO/Ceph deployments may require path.style.access=true. IP endpoints and dotted HTTPS buckets retain path-style addressing.
• Suggested improvements: Apply the SDK’s full bucket-name eligibility check before enabling virtual hosting, with a regression test asserting the URL for a mixed-case legacy bucket.

Reviewed the full 11-file diff at 4d3752702bcce2bfe4890357e87411a4fe5b9360 against 88a1f48cc8a5c8f017016737c86f0912bbaefbe9. Routed skills: review-comet-pr and review-comet-ffi-pr. The PR remains non-draft.

Exact-head CI: Comet CI and CodeQL require approval and have zero jobs. Only labeling passed.

Validation: Rust formatting and git diff --check passed. Three synthetic profile-loading checks passed, and 32 offline base/head URL outputs were compared using the locked dependencies. The native suite failed to build because jni.h is absent. A retry without HDFS support reached the 180-second build limit before tests ran. No Spark/JNI integration suite, live storage requests or benchmarks ran.

[sunchao]

[P2] Preserve path-style addressing for legacy mixed-case bucket names. For an existing US East bucket named LegacyBucket, scanning s3a://LegacyBucket/object with fs.s3a.endpoint.region=us-east-1, no custom endpoint and path.style.access unset now enables virtual hosting. The later guard only rejects dotted names. Base and Hadoop’s AWS SDK retain https://s3.us-east-1.amazonaws.com/LegacyBucket/object, but head produces https://legacybucket.s3.us-east-1.amazonaws.com/object. Hostname canonicalization changes the bucket being addressed, breaking previously valid reads. AWS supports these pre-March-2018 bucket names. Could we apply the SDK’s DNS bucket-name eligibility rules independently of the HTTPS dotted-name rule and add a final-URL regression test?

Evidence: An isolated Rust probe extracted the configuration functions verbatim from the requested base and head, then passed their output through locked object_store 0.13.2 and url 2.5.8. Offline signing with synthetic credentials produced base=https://s3.us-east-1.amazonaws.com/LegacyBucket/object and head=https://legacybucket.s3.us-east-1.amazonaws.com/object. No storage requests were sent. Probe and output: /tmp/comet-5872-probe/src/main.rs, /tmp/comet-5872-probe.log. An offline AWS Java SDK 1.12.780 check retained the uppercase path. Source inspection confirmed the same eligibility rejection in Hadoop 3.3.4’s SDK 1.12.262 S3RequestEndpointResolver/BucketNameUtils and SDK 2.29.52’s IsVirtualHostableS3Bucket. Hadoop’s URI handling preserves the bucket’s case.

[dwsmith1983]

Could we apply the SDK's DNS bucket-name eligibility rules independently of the HTTPS dotted-name rule and add a final-URL regression test?

Yes. is_virtual_hostable_bucket now follows SDK 2.29.52's isVirtualHostableS3Bucket: without dots the name must match [a-z\d][a-z\d\-]{1,61}[a-z\d], and over plain HTTP, where the rules pass allowSubdomains=true, dots are allowed but an IPv4-shaped name and adjacent ./- pairs are not. A name that fails goes path-style whatever fs.s3a.path.style.access says. The HTTPS dotted-name rule is now the allow_dots=false case of the same check, and it applies to both the default AWS endpoint and a custom one.

test_bucket_the_sdk_cannot_virtual_host_stays_path_style reads the final URL from an object_store presigned GET. s3a://LegacyBucket/object with fs.s3a.endpoint.region=us-east-1, no endpoint and the flag unset now resolves to https://s3.us-east-1.amazonaws.com/LegacyBucket/object, the same as base. The test also covers an underscore, a leading or trailing hyphen, a 64-character name, and, over an HTTP endpoint, an IPv4-shaped name, a dot next to a hyphen and a mixed-case name. A lowercase legacy-bucket still resolves to https://legacy-bucket.s3.us-east-1.amazonaws.com/object. The user guide now describes the name rule.

[dwsmith1983]

@andygrove could you approve a CI run on e9bd183f0? sunchao approved it, and the legacy bucket name fix from his last review is in d31577a8d. The last CI run was on 09-16, and it failed only the markdown check, which passes now.

[dwsmith1983]

@andygrove could you approve a CI run on b636fcbb8? The last CI run was on 09-16, on an earlier head.

[sunchao]

Summary

• Prior state and problem: Native S3 configuration inverted path.style.access and ignored Hadoop’s profile-name/file settings.
• Design approach: Pair normalized endpoints with their addressing mode, carry profile settings explicitly, and resolve default credentials paths on executors.
• Correctness / compatibility analysis: No introduced P1/P2 issues found within this review. Earlier concerns are addressed, including legacy mixed-case buckets. Checked configuration forwarding against Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 sources, plus relevant Hadoop and AWS SDK behavior.
• Key design decisions: Hadoop’s profile provider reads credentials-only files while SDK aliases retain their defaults. The small endpoint and session-default types are proportionate to the change.
• Implementation sketch: Executor configuration crosses JNI through a session extension and overlays Parquet/CSV scan options. Added work stays in plan/store construction. Arrow ownership and the object-read loop are unchanged. No separate P1/P2 performance issue was established.
• Behavioral changes worth calling out: Custom hostname endpoints now default to virtual hosting. The documented path.style.access=true setting preserves prior behavior where required. IP endpoints, dotted HTTPS buckets and ineligible legacy names retain path-style addressing.
• Suggested improvements: No additional P1/P2 changes requested.

Reviewed the full 11-file diff at 82472bf3fd39a12d5bfedcb1eb9fb3da5b317aaa against ce455f32d948355e073638e81009ecc3e5dea349. Routed skills: review-comet-pr and review-comet-ffi-pr. The PR remains non-draft.

Exact-head CI: Comet CI and CodeQL require approval and have zero jobs. Only labeling passed.

Validation: All 57 native S3 tests passed. Three synthetic profile-loading checks passed, and 44 head request URLs matched the checked addressing rules using locked dependencies. Rust formatting and git diff --check passed. Native tests used --no-default-features because JNI headers are unavailable. Full Spark/JNI and Spark SQL suites, live storage requests and benchmarks were not run.

[sunchao]

Summary

• Prior state and problem: Native S3 configuration inverted path.style.access and ignored Hadoop’s profile-name/file settings.
• Design approach: Pair normalized endpoints with their addressing mode, carry profile settings explicitly, and resolve default credentials paths on executors.
• Correctness / compatibility analysis: No introduced P1/P2 issues found within this review. Previously reported concerns are addressed. Checked configuration forwarding against Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 sources, plus relevant Hadoop and AWS SDK behavior.
• Key design decisions: Hadoop’s profile provider reads credentials-only files while SDK aliases retain their defaults. The endpoint and session-default types keep the changes localized without unnecessary abstraction.
• Implementation sketch: Executor configuration crosses JNI through a session extension and overlays Parquet/CSV scan options. Added work occurs during plan/store construction. Arrow ownership and the object-read loop are unchanged. No separate P1/P2 performance issue was established.
• Behavioral changes worth calling out: Custom hostname endpoints now default to virtual hosting. The documented path.style.access=true setting preserves prior behavior where required. IP endpoints, dotted HTTPS buckets and ineligible legacy names retain path-style addressing.
• Suggested improvements: No additional P1/P2 changes requested.

Reviewed the full 11-file diff at 167506ca58426e52436748233f7630a8cf1acf1b against 65a0cda1cf62877a37ec1a1f5f9ebc9dd1405ebd. Routed skills: review-comet-pr and review-comet-ffi-pr. The PR remains non-draft.

Exact-head CI: Comet CI and CodeQL require approval and have zero jobs. Only labeling passed.

Validation: All 57 native S3 tests passed. Three synthetic profile-loading checks passed, and 44 offline request URLs matched the checked SDK addressing rules using locked dependencies and verified current-source excerpts. Rust formatting and git diff --check passed. Native tests used --no-default-features because JNI headers are unavailable. Full Spark/JNI and Spark SQL suites, live storage requests and benchmarks were not run.

[andygrove]

This is a light fully automated review since there are so many PRs open.

I think the addressing change needs an entry in the upgrade guide as well as the note in datasources.md. docs/source/user-guide/latest/migration-guide.md now says each release's section also lists changes that need no legacy key but can still change what an existing deployment does, "such as a fix that makes Comet apply a setting as documented", and the 1.1.0 section has a "Settings That Now Take Effect as Documented" subsection for exactly that kind of fix. This PR is one of them. With fs.s3a.endpoint=http://minio.internal:9000 and fs.s3a.path.style.access unset or false, native reads move from http://minio.internal:9000/bucket/key to http://bucket.minio.internal:9000/key. The upgrade guide is the page that tells people to read every section between their old and new release, and right now the only mention is the paragraph at docs/source/user-guide/latest/datasources.md:260. Could you add an entry there for the release this lands in, naming fs.s3a.path.style.access=true as the setting that keeps the old addressing?

On the new row at docs/source/user-guide/latest/datasources.md:234, org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider first shipped in Hadoop 3.4.2. Spark 3.4 and 3.5 bundle Hadoop 3.3.4 and Spark 4.0 bundles 3.4.1, and the matching hadoop-aws jars do not contain the class. On those versions S3A cannot create the provider, so the query fails on the JVM side before the native scan runs. Since the next row says the SDK spellings ignore the profile keys, users on those Spark versions have no way to use fs.s3a.auth.profile.name or fs.s3a.auth.profile.file at all. Would it make sense to say in the row that this provider needs Hadoop 3.4.2 or later, which Spark 4.1 bundles?

[sunchao]

Summary

• Prior state and problem: Native S3 configuration inverted path.style.access and ignored Hadoop’s profile-name/file settings.
• Design approach: Normalize endpoints together with their addressing mode and resolve default credentials paths on executors.
• Correctness / compatibility analysis: No additional introduced P1/P2 issues found within this review. Earlier runtime findings are fixed. Checked Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 configuration forwarding against upstream sources and relevant Hadoop/AWS SDK behavior.
• Key design decisions: Hadoop’s profile provider uses credentials-only files. SDK aliases retain their defaults. The endpoint and session-default types keep the implementation localized.
• Implementation sketch: Executor configuration crosses JNI and overlays Parquet/CSV scan options before store lookup. Added work occurs during construction. Arrow ownership and the object-read loop are unchanged. No P1/P2 performance issue was established.
• Behavioral changes worth calling out: Custom hostname endpoints now default to virtual hosting. fs.s3a.path.style.access=true preserves previous addressing. IP endpoints, dotted HTTPS buckets and ineligible legacy names retain path style.
• Suggested improvements: Address the existing documentation concerns. The migration guide still omits this deployment-breaking addressing change. The provider table still omits the Hadoop 3.4.2+ prerequisite. Spark 3.4/3.5 bundle Hadoop 3.3.4 and Spark 4.0 bundles 3.4.1, where selecting the documented Hadoop provider fails before native execution. These remain existing blockers, with no duplicate inline findings added.

Reviewed the full 11-file diff at 167506ca58426e52436748233f7630a8cf1acf1b against 65a0cda1cf62877a37ec1a1f5f9ebc9dd1405ebd. Routed skills: review-comet-pr and review-comet-ffi-pr. The PR remains non-draft.

Exact-head CI: Comet CI and CodeQL require approval and have zero jobs. Only labeling passed.

Validation: All 57 native S3 tests passed. Three synthetic profile-loading checks passed, and 44 offline request URLs matched the checked SDK addressing rules using locked dependencies and verified current-source excerpts. Rust formatting and git diff --check passed. Native tests used --no-default-features because JNI headers are unavailable. Full Spark/JNI and Spark SQL suites, live storage requests and benchmarks were not run.

[dwsmith1983]

Could you add an entry there for the release this lands in, naming fs.s3a.path.style.access=true as the setting that keeps the old addressing?

Added in 78dd798, under "Settings That Now Take Effect as Documented" in the 1.1.0 section, with the MinIO example and fs.s3a.path.style.access=true as the way to keep the old addressing.

Would it make sense to say in the row that this provider needs Hadoop 3.4.2 or later, which Spark 4.1 bundles?

Yes. The row now says the class needs Hadoop 3.4.2 or later, that Spark 4.1 bundles it, and that older Spark releases lack it.

[sunchao]

Summary

• Prior state and problem: Native S3 configuration inverted path.style.access and ignored Hadoop’s profile-name/file settings.
• Design approach: Normalize endpoints together with their addressing mode, carry profile settings explicitly, and resolve default credentials paths on executors.
• Correctness / compatibility analysis: No introduced P1/P2 issues found within this review. Previously reported concerns are addressed, including the migration warning and Hadoop prerequisite. Checked configuration forwarding against Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 sources, plus relevant Hadoop/AWS SDK behavior.
• Key design decisions: Hadoop’s profile provider loads credentials-only files while SDK aliases retain their defaults. The endpoint and session-default types keep the implementation localized without unnecessary abstraction.
• Implementation sketch: Executor configuration crosses JNI and overlays Parquet/CSV scan options before store lookup. Added work occurs during construction. Arrow ownership and the object-read loop are unchanged. No P1/P2 performance issue was established.
• Behavioral changes worth calling out: Custom hostname endpoints now default to virtual hosting. fs.s3a.path.style.access=true preserves previous addressing. IP endpoints, dotted HTTPS buckets and ineligible legacy names retain path style. Hadoop’s profile provider requires Hadoop 3.4.2 or later.
• Suggested improvements: No additional P1/P2 changes requested.

Reviewed the full 12-file diff at 78dd79859cb80f7a0e862ae89aa1a8aaa9588bad against 22a07067bda272ed68adc67733cbaa7462a13c85. Routed skills: review-comet-pr and review-comet-ffi-pr. The PR remains open and non-draft.

Exact-head CI: Comet CI and CodeQL require approval and have zero jobs. Only labeling passed.

Validation: Three synthetic profile-loading checks passed, and 44 offline request URLs matched the checked SDK addressing rules. The harness uses verified current-source excerpts and matching locked dependencies. Rust formatting and git diff --check passed. Earlier evidence records 57 passing S3 tests, and that module remains unchanged. The current native-suite attempt could not start because the dependency mirror lacks locked hdfs-sys 0.3.1, even with --no-default-features. No full Spark/JNI or Spark SQL suite, live storage requests, benchmarks, or Markdown formatter ran.