Interpolation of values in vaex

[nilsleh]

Hi,

I am working with geolocation time series climate data. Data exploration has already been much easier thanks to vaex. Now I would like to interpolate NaN values for each grid cell representing a time series. The dataframe is ~210 million rows by 5 columns and the interpolation needs to happen per individual "geo_idx" which is a series of about 400 time steps. Currently, I am attempting to iterate over the dataframe to filter per "geo_idx" and do the interpolation in pandas and save the result to vaex hdf5 format, i.e.:

for geo_idx in tqdm(unique_ids):
   grid_cell_df = df[df.geo_idx==geo_idx]

   grid_pd = grid_cell_df.to_pandas_df()
   grid_pd["measure"] = grid_pd["measure"].interpolate(method="linear") # interpolate time series

   vaex_grid = vaex.from_pandas(df=grid_pd, copy_index=False)
   vaex_grid.export_hdf5(os.path.join(path, f"{geo_idx}.hdf5"))

However, this is prohibitively slow to call to_pandas_df() and from_pandas() again so I was wondering if there are any suggestions about how to do this interpolation. Thanks in advance!