Repository navigation
Simplified filter expression has a Null type instead of Int64 type across the FFI layer #1551
Description
Activity
Thanks! A quick look and I found that disabling the pushdown filters removes the problem, but I still haven't root caused why that's creating a problem with the optimizer.
I also found that there is no change (i.e. same error) from leaving the filter in place but replacing
TableProviderFilterPushDown::InexactwithTableProviderFilterPushDown::Exact; not that I suspected this, just hunting for load bearing parts of the repro.Ok, it looks like this is happening in the provider not in datafusion-python library as I had originally expected. What is happening is that we have a
Columnphysical expression that is failing to simplify becausesimplify_const_expr_immediatecannot correctly downcast it toColumnsince it's foreign at that point.This was an unintended side effect of apache/datafusion#18916
Here is what is happening:
- The table provider is getting a push down filter expression (logical) and a '&dyn Session' that is the
datafusion-pythonsession context. - The
ParquetSourceit is using under the hood takes aPhysicalExpras it's predicate. ThePhysicalExpris created by thedatafusion-pythonsession context during the call tocreate_physical_expr. This PhysicalExpr originates in thedatafusion-pythonlibrary so it is foreign in terms of the user library. That is, from the user library perspective it will be aForeignPhysicalExpr - On the user library we are getting calls to
simplify. It looks like this happens both in open and in row group filter (which is where we're hitting it here). simplifychecks to see if it is aColumnduringsimplify_const_expr_immediatehowever it cannot downcast toColumnbecause we are in the user library NOT the main datafusion-python library when this simplify gets called.
Here is a work around I have tested with this code
let execution_props = ExecutionProps::new(); let predicate = predicate .map(|predicate| { datafusion::physical_expr::create_physical_expr( &predicate, &df_schema, &execution_props, ) }) .transpose()? // if there are no filters, use a literal true to have a predicate // that always evaluates to true we can pass to the index .unwrap_or_else(|| datafusion::physical_expr::expressions::lit(true));- The table provider is getting a push down filter expression (logical) and a '&dyn Session' that is the
I am working on an upstream issue that will resolve this in a more complete manner, but I hope that unblocks you for 53.
Reacted by Jack WimberleyI confirmed the workaround in the MRE and am now implementing and testing it in the more complex provider from which that was derived. What are the implications of creating the physical expression using the default execution properties instead of the session state? The docs say that the result of the
SessionState::create_physical_exprcall isis not simplified or otherwise optimized
so I'd guess there's no custom optimizer rules that could come into play in it?
create_physical_exprdoes not do any simplification but the parquet provider is doing simplification.Reacted by Jack WimberleyJust confirming (unsurprisingly) that the workaround also works in the original context of the problem
Cool. I have some proper fixes going in upstream but probably won't land in time for 54. I'm sorry about the troubles.
- added a commit that references this issue
on May 20, 2026 Not at all, thanks for the quick workaround and resolution!
Describe the bug
A custom table provider for a ParquetSource with a trivial catalog and an Int64 column yields some errors when a SQL query has a filter with a literal limit on that column of the form
The error does not occur when using datafusion-python 52; it also does not occur when running the query purely in a Rust SessionContext; the backtrace for the above error shows it coming from datafusion-ffi code as well.
This may of course not be a bug, but instead some bad practice that the version 52 set of crates tolerates but which is now invalid. A MRE of this custom table provider can be found in the public repo https://github.com/jwimberl/datafusion_python_53_int64filter_repro, which contains
and a canned dummy dataset. The README.md of this repo has more details.
To Reproduce
In the
mainbranch, build thepy_repro_providercrate withmaturin developand runpython repro.py. This loads the dummy dataset as a tabledummy_tableand runs two queriesSELECT * FROM dummy_table LIMIT 1, which is successfulSELECT * FROM dummy_table LIMIT 5, which panicsItss output should be something like
followed by backtrace information.
Expected behavior
In the
datafusion52branch, build thepy_repro_providerwithmaturin developand runpython repro.py. It runs the same two queries, and its output should beAdditional context
In either the
mainbranch ordatafusion52branch, the Rust code for the table provider is in the directoryrepro_provider, and there is a corresponding cargo test that runsSELECT * FROM dummy_table WHERE a < 5. Without the FFI layer, this is successful with both datafusion 52 and 53.