Skip to content

Add apache-datafusion provider skelton#64998

Open
gopidesupavan wants to merge 4 commits into
apache:mainfrom
gopidesupavan:apache-datafusion-provider
Open

Add apache-datafusion provider skelton#64998
gopidesupavan wants to merge 4 commits into
apache:mainfrom
gopidesupavan:apache-datafusion-provider

Conversation

@gopidesupavan

Copy link
Copy Markdown
Member

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@Srabasti

Copy link
Copy Markdown
Contributor

Thanks for adding this new provider @gopidesupavan!!
Looks like the static checks are failing since the the new provider apache.datafusion is not listed in path below.
https://github.com/apache/airflow/blob/main/airflow-core/docs/extra-packages-ref.rst

Direct link:
https://github.com/apache/airflow/blob/main/airflow-core/docs/extra-packages-ref.rst#apache-software-extras

Suggest to add this to the relevant reference.

| apache.datafusion | pip install apache-airflow[apache.datafusion] | Apache.datafusion hooks and operators |

@Srabasti Srabasti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Link "https://airflow.apache.org/docs/apache-airflow-providers-apache-datafusion/0.1.0" gives 404 Error, as below.

Image

I would be happy to update the link if you permit Sir! Was curious to learn more about this new provider, hence was checking out the links.

@gopidesupavan

Copy link
Copy Markdown
Member Author

Link "https://airflow.apache.org/docs/apache-airflow-providers-apache-datafusion/0.1.0" gives 404 Error, as below.

Image I would be happy to update the link if you permit Sir! Was curious to learn more about this new provider, hence was checking out the links.

these will be published part of the release for now not required..

@gopidesupavan

Copy link
Copy Markdown
Member Author

Link "https://airflow.apache.org/docs/apache-airflow-providers-apache-datafusion/0.1.0" gives 404 Error, as below.

Image I would be happy to update the link if you permit Sir! Was curious to learn more about this new provider, hence was checking out the links.

yes look at this about https://datafusion.apache.org/python/ functionalities it provides

+---------------------+-----------------------------------------------------+------------------------------------------------+
| apache-beam | ``pip install 'apache-airflow[apache-beam]'`` | Apache Beam operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
| apache-datafusion | ``pip install 'apache-airflow[apache-datafusion]'`` | Apache DataFusion provider package |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alphabetical order: apache-cassandra should come before apache-datafusion here. The entry was inserted between apache-beam and apache-cassandra, which breaks the alphabetical sort the rest of the table follows. Move it down two rows so the order reads beam, cassandra, datafusion, drill.

Comment thread .github/boring-cyborg.yml Outdated
Comment thread providers/apache/datafusion/README.rst Outdated
Comment thread providers/apache/datafusion/provider.yaml Outdated
Comment thread providers/apache/datafusion/pyproject.toml
Comment thread providers/apache/datafusion/tests/unit/apache/datafusion/test_example.py Outdated
@github-actions

github-actions Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

uv.lock on main just moved via #68642 ("Prepare provider documentation 2026-06-16"), commit 0236c57 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

@gopidesupavan gopidesupavan force-pushed the apache-datafusion-provider branch from d477b07 to 567bfb4 Compare June 20, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants