Output npy of hdf5 file using the processor

[ico1036]

Hi, I'm using Coffea in my physics analysis.

I'm very curious about how to write npy or hdf5 files in a processor.
I'm using run_uproot_job with multi workers.

my processor link

I understand that the accumulator only can stack histograms and write histo output.
Is there any method to output "arrays" with npy, hdf5, or other types?

Thanks

[nsmith-]

@kondratyevd has contributed a feature in #368 to optionally output a dask dataframe from run_uproot_job, after which one could write the dask dataframe to a file or set of files in a variety of formats e.g. to_hdf

[kondratyevd]

Hi @ico1036,
if you can convert the outputs of your processor to Pandas DataFrames, then you should be able to use Dask executor with argument use_dataframes=True.

The output will be a distributed Dask dataframe.
If you want to continue working with it, or print out as a single dataframe, you will also need to call output.compute() after you retrieve the outputs from run_uproot_job. Otherwise, you can directly save chunks of the output dataframe as Parquet files using dd.to_parquet(df=output).

Please let me know if you run into any issues, I will be happy to help.

[kondratyevd]

@ico1036 I just remembered - I've added a test example showing how to use this setup, see here: https://github.com/CoffeaTeam/coffea/blob/master/tests/test_dask_pandas.py

[ico1036]

@nsmith @kondratyevd
Thank you very much for helps!
I will test it and let you know if there is another issue.