How to perform multiple .where operations with the same condition efficiently

[xeladsn]

Hi all,

I have a code that currently performs .where operations one after another to update a DataFrame row-wise with the same condition my_condition, such as:

df['col_1'] = df.func.where(my_condition, value_col_1, df['col_1'])
df['col_2'] = df.func.where(my_condition, value_col_2, df['col_2'])
...
df['col_n'] = df.func.where(my_condition, value_col_n, df['col_n'])

As the .where method is eagerly evaluated, all the operations below are performed subsequently, but I have a sense this might be greatly optimized by using another implementation, or parallelization.

Thanks in advance for your insights !

Alexandre

[maartenbreddels]

Hi,

it is not eager, so this should only evaluate my_condition once if you do a particular operation. If my_condition is expensive, you could consider a materialize on it.
Does that help?

Regards,

Maarten

[xeladsn]

Performance significantly improved (4 times faster now) by using .materialize.
I have tried to use .execute and .extract before, but it did not bring such a performance increase as .materialize.

Anyway, thanks a lot ! :)