Repository navigation
Make dbt-core an optional dependency so edr can run against dbt v2 without a second dbt #2354
Description
Activity
I ran into this too. Looking at the code, the runtime side seems to already be there — the blocker looks like packaging alone.
factory.pyimportsAPIDbtRunnerlazily,Dbt2RunnerandSubprocessDbtRunnershell out to the binary, andget_dbt_core_version()already returnsNonewhen the package is absent. Acrosselementary/, the only module-levelfrom dbtimport isapi_dbt_runner.py— the other hits are Jinja macros in.sql. So dbt-core appears to be needed only by the API runner, which is the one path a Fusion user never takes.That would make the change roughly:
- move
dbt-core = ">=1.8,<3.0.0"out of[tool.poetry.dependencies]into an optional extra - leave the adapter extras alone, since each already pulls dbt-core 1.x transitively
- add one clear error when neither dbt-core nor a dbt 2 binary is found, since today that path ends in a confusing subprocess failure
The part that's yours to decide:
pip install elementary-datacurrently gives a working 1.x setup, and this would end that. Acceptable in a minor, or would you rather stage it?Happy to put up the PR if you're open to it.
- move
This is already addressed in an open PR — linking here so others don't duplicate the work.
Code: #2360 — Remove the CLI's hard dbt-core dependency (dbt v2 ships as dbt/dbt-oss) (CORE-1477, follow-up to #2333)
Docs: #2361 — two-path CLI install (dbt v2 vs dbt Core 1.x with dbt1-* extras)
#2360 makes
dbt-coreoptional: plainpip install elementary-datano longer pulls in dbt-core 1.x, so it can coexist withpip install dbt/ Fusion in the same environment. dbt-core 1.x still comes in via the existing warehouse extras (now also documented asdbt1-<warehouse>), withdbt-corepinned<2.0in those extras.CI on #2360 is green across the warehouse matrix and fusion/dbt2 targets. Happy to help review or land small follow-ups (e.g. removing the obsolete
pip uninstall dbt-corestep in Fusion CI once packaging lands) rather than open a competing PR.@haritamar — if you agree this closes this issue, adding
Closes #2354to the #2360 description would auto-close on merge.
Is your feature request related to a problem? Please describe.
elementary-datadeclaresdbt-core<3.0.0,>=1.8as a hard dependency, and every adapter extra pulls a 1.x adapter that depends on it too. Installed alongsidedbt==2.0.1(the Fusion engine), dbt-core wins the install: both ship adbtconsole script and adbtPython package, anddbt --versionafterwards reports 1.12.4. Everydbt runin that environment silently drops to dbt 1.x.The CLI imports dbt-core in one place,
from dbt.cli.main import dbtRunner, dbtRunnerResultinclients/dbt/api_dbt_runner.py, which is the dbt 1.x runner.Dbt2Runnerfrom #2333 drives Fusion over a subprocess and doesn't need it, and edr never opens a warehouse connection of its own, so under dbt v2 neither dbt-core nor a Python adapter is doing anything.Describe the solution you'd like
Move
dbt-coreout of the base dependencies into an extra, the way the adapters already are, sopip install elementary-dataon its own works against a Fusion binary found throughDBT_FUSION_PATH. #2333 already detects "no dbt-core, Fusion binary present", so the runtime side exists and this is packaging.Describe alternatives you've considered
Keeping edr in its own virtualenv, separate from the dbt project's. That's what we've done, though it means a second lockfile and a second dependabot entry for one CLI.
Additional context
elementary-data 0.26.0, dbt 2.0.1 on Snowflake, Python 3.14.