How to use Vaex on-the-fly data with Researchpy or Pingouin or any Statisctic-related Python module

[josemiguelregoterol]

Hi, all of you,
So I have several questions that can be resumed in one sentence: How can I apply functions from other packages like Researchpy or Pingouin to on-the-fly data created by Vaex? Is it possible to manipulate data that is not "written" in the memory? I guess it is possible since Vaex is doing it (e.g. data.mean(name_data_column)).
I tried to convert on-the-fly Vaex data to panda DataFrame, but it took a lot of time.

A practical example I want to do is the following:

import vaex
import scipy.stats as sp

df_remote = vaex.open('s3://vaex/taxi/yellow_taxi_2015_f32s.hdf5?anon=true')
%time t,p=sp.shapiro(df_remote['tip_amount'])

I want to calculate the normality of one on-the-fly column using the module scipy.stats

One option is to integrate the Stat Vaex module with the functions I want to use. Do you know which files contain the stat function from Vaex?

Thanks

[JovanVeljanoski]

Hi,

The formal answer is "it depends", but in practice it is more like "does not look good". Let me explain.

When using vaex, all the operations that vaex can do on columns (mean, stdev, skew, value_counts, etc..) we have implemented ourselves in C++ using streaming algorithms so you can use apply them on data of arbitrary size. (I say "we", but it is all work of @maartenbreddels ).

So when talking about applying functions / methods etc.. from other packages to vaex data, they fall broadly in 2 categories:

1. Transformation that you want to do on an individual sample (row). Simple example: add +1 to column 'x'.
2. Aggregations that you want to do that you get a single answer from the entire column. Simple example: compute the mean of 'x'.

For 1):
This is generally possible. Vaex supports the most common operations, and most of the numpy operations are supported. Even if they are not supported, as an escape hatch you can use the apply method. This might not be as efficient, since if you are using some function from a 3rd party library you are basically outsourcing the performance burden on that library. Although vaex's apply is executed in parallel by default. Relevant parts of the documentation are: apply and maybe "extending vaex".

For 2):
If the aggregation you are trying to do can be decomposed in aggregations that vaex supports, then sure - no problem. A simple example of this is mean(x) / std(x). If the aggregation requires some mathematical transformation that vaex does not support then, i am afraid the answer is no. That is generally (but not always) true for stuff in the scipy package. There are 2 main reasons for this:

• we may not know how to implement some operation in a streaming way that would work out of core efficiently on billions of samples
  • some things are simply not possible (or researchers have not found a way yet)
• we may know how to do something, but it is very hard or very niche, and we are unwilling to take that work without dedicated funding / support / motivation.

Looking at the source of sp.shapiro i see some operations that are not supported in vaex and thus this can't work out of the box.

There is also no need to export data from vaex to pandas is chunks in order to do some operations on them (on those chunk), as that will indeed include some overhead. Better is to use apply or go via the register_function route (see documentation links above).

Having said all of this, if you do know how to make more things out-of-core, PRs are always welcome! :)

Finally, as vaex is fully compatible and support Apache Arrow, if there is something in that project that would be useful but we do not support it yet, let us know and we will look into it.

I hope this answers the question.