[Trying Polars] performance lacking when parallel processing

[zain-sohail]

I spent a bit of time trying to implement polars to read the dataframes. Here are some things I learnt:

• The API is similar but also significantly different from pandas so there's a learning curve
• Not as well documented as pandas
• The single file read performance is 5x faster than with pandas
• The performance is fast for reading a single file as it utilizes multiple cores. Hence while reading multiple files in parallel, there's no improvement
• No h5 reader available. If it were, the polars optimized queries might be useful.

Since our application has no cpu constraints, polars becomes a poor fit. Might still be useful for when we need to do serial reading. And especially if we want to stream data.

[rettigl]

Thanks, I was also wondering if that could help. Good that you tested this.
As for performace, I actually tested and noticed that the direct h5 reading of the mpes loader is ~20-30% faster compared to reading from converted parquet files, even though there are lots of additional dask layers.
It could be helpful to also test this for the fel/flash loader.

[zain-sohail]

We would do it but we use the multiindex of pandas and that is not supported by dask yet.
Either ways, after first read, the buffer files are much faster to use.

[rettigl]

Either ways, after first read, the buffer files are much faster to use.

My point is that this is not true (at least for our date with the mpes reader).