How to avoid using for loop in these kind of situations

[ZhenxuanZhang-Jensen]

Hi experts,
Currently, I am trying to use uproot to load a root file, and change it as a numpy array and do the cut, and calculate the cut efficiency. I keep this in mind for a long time, and I'm always thinking about this kind of situation which I met a lots of times. Can we don't use for loop, but use awkward array to do the exact same thing? such as creating an awkward array to save all the efficiency and then using slice to do the cut?
Best regards,
Zhenxuan

bkg_xgb_roc = []
for i in np.linspace(-1,1,1000):
    sig_xgb_roc.append(event_signal_xgb_weights[event_signal_xgb['DiphotonMVA_self'].array(library = 'np') > i].sum() / event_signal_xgb_weights.sum())
    bkg_xgb_roc.append((event_bkg_pp_xgb_weights[event_bkg_pp_xgb['DiphotonMVA_self'].array(library = 'np') > i].sum() +  event_bkg_pfff_xgb_weights[event_bkg_pfff_xgb['DiphotonMVA_self'].array(library = 'np') > i].sum()) / ( event_bkg_pp_xgb_weights.sum() + event_bkg_pfff_xgb_weights.sum())) ```

[jpivarski]

Well, first thing: the uproot.TBranch.array method reads data and you don't want to read and re-read the same data over and over. You could pass a cache to the array method so that it would check the cache for an already-read value instead of getting it from the file, but it's probably easier to just move the

signal_xgb = event_signal_xgb['DiphotonMVA_self'].array(library = 'np')
bkg_pp_xgb = event_bkg_pp_xgb['DiphotonMVA_self'].array(library = 'np')
bkg_pfff_xgb = event_bkg_pfff_xgb['DiphotonMVA_self'].array(library = 'np')

read commands out of the loop and just use those arrays, rather than reading them, in the loop.

Secondly, you can remove the sums that don't depend on i outside the loop on i, so that they don't get recomputed each time. That is,

signal_xgb_weights_sum = event_signal_xgb_weights.sum()
bkg_pp_xgb_weights_sum = event_bkg_pp_xgb_weights.sum()
bkg_pfff_xgb_weights_sum = event_bkg_pfff_xgb_weights.sum()

What remains is a for loop over 1000 values of i, and generally pure Python is acceptable for loops of only 1000. That is, the fact that you have to step over Python statements 1000 times is not going to be a pain point: the Python overhead will be milliseconds. But, what you have in that loop is a few sums, and if those sums are over a large amount of data, then doing 1000 of those might be painful.

So first, try the fixes described above and if it's still a problem, you might want to try using np.sort, np.cumsum and/or np.searchsorted. You want to add up everything in an array that's greater than a given i. Instead of repeatedly cutting and summing, sort the data so that all the elements greater than i would be to the right of i, do one cumulative sum, so that the sum of everything greater than i is the value at i, and that might be fast enough already because 1000 tests would just be 1000 array-lookups (reading values from the cumulative sum 1000 times). If it really, really matters after that, you can even vectorize the lookups with searchsorted, but I doubt it will matter—unless this example with 1000 was a small-scale test and you'll actually be doing it with millions or billions of i values.

All while you are making these incremental improvements, check the results of the optimized code against the results you already have with the slow code. Don't try to do all the improvements at once—small steps with correctness-checking at each step (i.e. "test driven development")—will likely involve less debugging.

[ZhenxuanZhang-Jensen]

I have tried numba before and it seems I can't use TensorFlow in numba, so I can't do the prediction in numba, so as the Rdataframe.

[agoose77]

I don't fully understand what you're doing, but at a high level, you wouldn't use Numba / RDataFrame to perform your model fitting. Instead, you'd use it to manipulate/transform your arrays, treating it like a black box:

flowchart LR
data[Data Source] --> select[Jagged Selection] --> tensor[TensorFlow]

Loading

[ZhenxuanZhang-Jensen]

yes, What I want to do is to use Numba or RDataFrame to accelerate the DNN model evaluation and get the score of each given dataset that needs to be produced in the for loop. So I need to predict the score one by one in the for loop inside Numba/RDataFrame which means I need to use tensorflow or more specific model.predict(array) inside Numba/RDataFrame. But I tried, and it seems that doesn't work, so it just gets me back to the very beginning of my problem, how to accelerate the whole running process...

[jpivarski]

The trick is to use these tools to build up what you're calling "array" here. That's what would use Numba or RDataFrame, the formatting of that data. Numba and RDataFrame don't recognize TensorFlow objects or functions as types (it would have to have an explicit Numba or C++ interface; it's possible there's a C++ interface, but I doubt you want to use it).

The model.predict function surely takes large, multi-event datasets as possible inputs, perhaps as a two-dimensional NumPy array.

If your TensorFlow model takes jagged data, the interface it probably wants is RaggedTensor. We don't have an explicit function to convert an Awkward Array into a RaggedTensor (issue #1466), but it wouldn't be hard to do that conversion, zero-copy. RaggedTensor.from_row_lengths needs the values that ak.num provides (high-level route), and RaggedTensor.from_row_splits takes the offsets of a ListOffsetArray (low-level route, fewer computations). If you go the low-level route, you can find the ListOffsetArray inside an array's layout, and if your jagged array is based on a ListArray or if the offsets doesn't start with 0 (which TensorFlow needs), there's a toListOffsetArray64(start_at_zero=True) method that you can call on it.

[ZhenxuanZhang-Jensen]

Okay. Thanks for all the answers, I think I know where I need to take a look now.