I see the scope of the dataframe api mentions "Memory Ownership and Lifetimes" as In Scope, but I couldn't find anything in else in the document that actually enforces this.
While trying to use the dataframe api with pandas, I ran into an issue where the current pandas implementation to write string data makes a temporary buffer whose lifetime has nothing to do with the dataframe itself. I found this because I wrote an extension that assumed the lifetime of a buffer in the dataframe API was the same as the producing dataframe. That held while I was using primitive types but once I tried to use strings I started seeing segfaults.
My first thought was that pandas was doing something wrong, but without any guidance from the document it is hard to make that judgment.
By contrast, the Arrow C Data specification (which I ultimately ended up using) has well defined semantics for memory management:
https://arrow.apache.org/docs/format/CDataInterface.html#memory-management
I see the scope of the dataframe api mentions "Memory Ownership and Lifetimes" as In Scope, but I couldn't find anything in else in the document that actually enforces this.
While trying to use the dataframe api with pandas, I ran into an issue where the current pandas implementation to write string data makes a temporary buffer whose lifetime has nothing to do with the dataframe itself. I found this because I wrote an extension that assumed the lifetime of a buffer in the dataframe API was the same as the producing dataframe. That held while I was using primitive types but once I tried to use strings I started seeing segfaults.
My first thought was that pandas was doing something wrong, but without any guidance from the document it is hard to make that judgment.
By contrast, the Arrow C Data specification (which I ultimately ended up using) has well defined semantics for memory management:
https://arrow.apache.org/docs/format/CDataInterface.html#memory-management