aws.XXX options are not recognized in SQL API for CREATE EXTERNAL TABLE with AWS S3 location

Describe the bug
Hi

I was trying to create external table from AWS S3 using examples from Datafusion CLI docs and SQL DDL docs, see To Reproduce section below. But looks like AWS-specific options for external tables doesn't work with SQL API in the same way as it described in CLI docs

Exception: DataFusion error: Configuration("Could not find config namespace \"aws\"")

I can successfully register and read the same parquet with the same credentials via register_object_store + register_parquet functions, so there's no problem with the data or S3 connection

To Reproduce

import datafusion

ctx = datafusion.SessionContext()

raw_data_df = ctx.sql(f"""
    create external table raw_data
    stored as parquet
    options
    (
        'aws.access_key_id' '***',
        'aws.secret_access_key' '***',
        'aws.region' 'eu-central-1'
    ) 
    location 's3://some-bucket/some-folder/'
""")

raw_data_df.show()

Expected behavior
create external table command works with aws.xxx options

Additional context
v 42.0.0

I know this is a very old issue, but I've looked into the root cause of it. The datafusion-cli adds in a special configuration option for AWS credentials. In datafusion-python the more canonical way to add these credentials is documented at https://datafusion.apache.org/python/user-guide/data-sources.html#object-store

If you have a specific need to be able to add credentials via configuration we could go the route of adding a configuration extension as shown here in datafusion-cli but for right now I suspect using the credentials from the OS is the better path forward.

If you need that configuration option you could either add it as a user defined configuration option and expose it to python or we could consider adding it to this project. For the latter, we'd probably want more justification since the canonical support method already exists. Please reopen this issue if you need further asssistance!

Navigation Menu

Repository navigation

aws.XXX options are not recognized in SQL API for CREATE EXTERNAL TABLE with AWS S3 location #970

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

aws.XXX options are not recognized in SQL API for CREATE EXTERNAL TABLE with AWS S3 location #970

Description

Activity

timsaucer commented on Sep 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions