How does vaex handle RNGs?

[dclong]

Hi everyone,

I wonder how does vaex (or more generally, distributed computing engines) handle RNGs? High level speaking, there are 2 ways.

1. Use a single RNG in the main process (master node, driver, etc.) and workers make request to the single RNG to generating random numbers.
2. Use a separate RNG in each worker (node, process, etc).

I read the source code of a few methods in vaex.dataframe, and it seems to me that the first approach is used. I wonder is this consistent across all implementations?

If in anywhere the 2nd approach is used, how does vaex seed RNGs to ensure good quality of overall random number sampling?

[JovanVeljanoski]

Vaex is not (currently) a distributed computing engine. We might be able to go that way if there is a need/interest. So far, single node works just fine.

To the best of my knowledge, we do not really use random number generators anywhere.. Vaex works best with fixed data that lives on disk. If you want to generate random data, that is fine, but that is a process happening outside of Vaex.

[dclong]

I see the RNG in numpy is used in some implementation of Vaex. Does Vaex leverage multiprocessing? If so, is there any possibility that vaex use separate RNGs in different processes?