How can I filter the entries of big root file before converting it to a panda data frame?

[SaharGholipour]

My root file includes a big data set and it would take forever to convert it to a panda data frame after opening the root file, so I need to remove some of the entries I don't want to have before converting it to a panda data frame. I would appreciate it if you could help me how to remove those entries.

[Moelf]

do you have example code of what you're currently doing (that is slow but shows what you're trying to do)? that makes it easier to help you

[SaharGholipour]

This is how I work with a root file in Python:
import uproot
file = uproot.open('file.root')
branches = file['tree_name'].arrays(library='pd')
So, basically, my data set will be a table and the columns are my variables (or branches) and the rows are the different entries. then the rest is all in numpy and panda not uproot anymore. Generally, after converting it to a table I use .query to define my cuts and remove the rows I don't want to.
But now the size of my root file is almost 1GB which can't be converted to the panda data frame, so I need to remove some of my entries to make the data set smaller.
I saw in uproot documentation there's something uproot.iteration(step_size=n) which will take small batches from the data set but I don't know how to define my cuts and at the end having a smaller data set being able to open in panda.

[Moelf]

as a side note, 1GB isn't that large, even after decompression it should comfortably sit in your RAM. Do you know what are the branch types? any chance they are not friendly to pandas?

One possibility is to not do library='pd' first, and use Awkard to obtain a "mask", which should be faster, then mask the branches and only then convert to pandas

[jpivarski]

1 GB can be large in Pandas. :) The syntax for filtering is the same in NumPy, Awkward, and Pandas:

all_data = file["tree_name"].arrays()
some_data = all_data[(all_data["variable1"] > 123) & (all_data["variable2"] > 321)]

although

• NumPy is the fastest and most memory-efficient if the data are purely numerical (i.e. no jagged arrays—no per-particle quantities, only per-event quantities)
• Awkward is the best suited for jagged arrays, including cuts on both per-event and per-particle quantities
• Pandas is popular, has the most documentation online, makes plotting and statistical routines easier, but can be slow or a memory hog sometimes.

@Moelf's suggestion of going through Awkward (or NumPy) first, then converting to Pandas is a good one. You can use the pd.DataFrame constructor, ak.to_dataframe, awkward-pandas, etc. Uproot's library="pd" is not the only way.

Also, you can use entry_start=123, entry_stop=321 to reduce how much Uproot reads (as opposed to reading everything into memory and then applying cuts). See uproot.TTree.arrays. This is essentially what uproot.TTree.iterate does in each step (except that that function is a little more efficient in dealing with granularity within the ROOT file and sharing data between steps).

Also, also, if you're not going to use all of the TBranches, you can specify a subset to read as the first argument to uproot.TTree.arrays and uproot.TTree.iterate. The filter_name argument can take wildcards.