[Python] libc++ Compatibility Issue - Crash

[dylanetaft]

Describe the bug, including details regarding any error messages, version, and platform.

System-wide clang libc++ use has compatibility issues with pyarrow.

Disclosure - I investigated this with Grok 4.5, but made an effort to understand the issue and limited changes it wanted to make. I will submit a PR shortly that addresses this. It is very small, two lines, moves array dtor to cpp file.

python -c "import pyarrow as pa
import pyarrow.parquet as pq
arr = pa.array([1], type=pa.int64())
table = pa.table([arr], names=['a'])
pq.write_table(table, '/tmp/t1.parquet')
print('1 OK')
"
libc++abi: terminating due to uncaught exception of type std::bad_cast: std::bad_cast
Aborted (core dumped) python -c "import pyarrow as pa

tldr; Python RTLD_LOCAL + Arrow’s header-inline virtual destructor results in duplicate array typeinfo in several DSOs; without NDEBUG, parquet’s dynamic_cast fails under libc++ when used in pyarrow.

Python defaults to loading shared libraries with RTLD_LOCAL. Several Apache Arrow headers contain inline defined virtuals in the headers instead of the source files, which leads to technically separate class footprints for each translation unit that includes the header, RTTI is not unified across DSOs. libc++ is very particular with dynamic_cast, where libstdc++ is more relaxed. When apache arrow is compiled without DNDEBUG, a code path in VisitArrayInline is enabled that uses c++ dynamic casts. Pyarrow has multiple shared objects loaded that use arrow data types, combined with RTLD_LOCAL - this leads to objects that actually cannot dynamic_cast under clang's libc++ - dynamic_cast will see different type identities.

This is reproducible under a Gentoo system compiled entirely with clang + libc++, plus all libraries compiled against libc++, with pyarrow.

Other workarounds:
This will basically put libarrow symbols in the global namespace circumventing the issue as a workaround
LD_PRELOAD=/usr/lib64/libarrow.so.2500:/usr/lib64/libparquet.so.2500 python yourapp.py

Component(s)

Python

[pitrou]

Several Apache Arrow headers contain inline defined virtuals in the headers instead of the source files, which leads to technically separate class footprints for each translation unit that includes the header, RTTI is not unified across DSOs.

Is there a reference to this C++/libc++ behavior?

[dylanetaft]

https://github.com/llvm/llvm-project/blob/main/libcxxabi/src/private_typeinfo.cpp
_LIBCXXABI_FORGIVING_DYNAMIC_CAST

https://github.com/aalexand/sharedlib_typeinfo

https://www.qt.io/quality-assurance/blog/one-way-dynamic_cast-across-library-boundaries-can-fail-and-how-to-fix-it

I can just find references of people saying libc++ and libstdc++ behave a bit different.
libstdc++ relies on symbol names, libc++ relies on a hard ptr comparison.
https://en.cppreference.com/cpp/language/dynamic_cast

"Some forms of dynamic_cast rely on run-time type identification (RTTI), that is, information about each polymorphic class in the compiled program. Compilers typically have options to disable the inclusion of this information."

It seems like a C++ annoyance...I am not sure it is a widespread compatibility issue with pyarrow though.

I have done some local testing with LD_LIBRARY_PATH with the fix in, I'm installing git head pyarrow\arrow globally to make sure the fix works without any linker overrides before I send the PR, it is a tiny fix.

[dylanetaft]

So the patch does not work. The workaround of preloading the library seems to be viable. Or compiling apache arrow with NDEBUG, which has guards in the code to disable dynamic_cast, only use static.

Forgiving dynamic cast option for libc++abi lib isn't enough either. Compiling libc++ with -DLIBCXX_TYPEINFO_COMPARISON_IMPLEMENTATION=2 might work, but that's not default.

I think this is just a general incompatibility with systems using clang\libc++ for everything. It doesn't crop up often as it's not a common setup...most envs are pip install and walk away.

Should this be closed - people can use the workaround(s)?
Otherwise a lot of syntax sugar is needed to get the typeinfos to align for libc++.

Other workaround is
import sys
import ctypes
sys.setdlopenflags(sys.getdlopenflags() | ctypes.RTLD_GLOBAL)
BEFORE you import pyarrow - I think this pollutes pythons' process wide symbol table which may or may not matter in various scenarios