DataFrames - Pandas or an alternative?

[cmutel]

Here are some Pandas alternatives:

• Polars: Written in Rust for better speed. Mostly API compatible but not always.
• Dask: Scales to multiple machines. Can also run distributed tasks across clusters.
• Modin: Drop-in replacement for Pandas with multiple cores. Uses Dask, Ray, or MPI for dispatch.
• xarray: Not a dataframe per se, but allows for label across multiple dimensions and otherwise follows Numpy.

There are also Big Data alternatives like Spark, Snowflake, Databricks, and dbt.

We don't have great criteria for choosing between these alternatives. This might be one case where we postpone making a decision.

[romainsacchi]

Note that xarray isn't very good with storing sparse data. You quickly run into memory issues I noticed. I believe it is possible to initialize it with a scipy.sparse matrix, but this is not really meant to be used that way (and working directly with scipy and dictionaries is often more straightforward). Storing non-null exchanges in a dataframe makes a lot more sense IMO (but since I do not really have a good mental picture of the prototype in question, I may be wrong).

[michaelweinold]

I investigated this a few weeks ago. It seems that xarray does now support sparse matrices (through scipy.sparse, as you noted). See also: https://github.com/sustainableaviation/EcoPyLot/issues/8
Generally, I am not convinced additional dimensions are justified in most cases - simply adding another label column to a two-dimensional tabular data structure introduces less overhead.
I found that moving from Pandas to Apache Spark was rather straighforward - most of the PySpark API is similar. So postponing "big data" infrastructure decisions might be prudent, @cmutel.

[romainsacchi]

Other candidate: datatable. Their pitch: "a toolkit for performing big data (up to 100GB) operations on a single-node machine, at the maximum speed possible". It seems faster than polars on some operations.

[cmutel]

This feels like premature optimization. Let's build a data API which allows data to be returned as Numpy arrays (both normal and recordarrays, DataFrames, and plain Python or Javascript objects), and build something more complicated only when we really need it.