DNN Indexing Bug Fix + Normalization Improvements

[GNiendorf]

Fixes a bug related to the hit indexing in the DNN train script. Also adds other hits to write_lst_ntuple. Will add more to this PR soon.

[GNiendorf]

/run all

[github-actions]

The PR was built and ran successfully in standalone mode. Here are some of the comparison plots.

The full set of validation and comparison plots can be found here.

Here is a timing comparison:

   Evt    Hits       MD       LS      T3       T5       pLS       pT5      pT3      TC       Reset    Event     Short             Rate
   avg     44.2    324.5    115.0     46.6     91.8    501.5    111.6    149.9    100.4      2.2    1487.7     942.1+/- 246.4     406.7   explicit_cache[s=4] (target branch)
   avg     43.7    324.0    115.8     46.9     90.2    549.8    112.2    150.2    106.6      2.0    1541.2     947.7+/- 247.1     421.3   explicit_cache[s=4] (this PR)

[GNiendorf]

No timing changes, see below on GPU.

Current Timing

This PR

[GNiendorf]

Efficiency increases slightly for pT5's but no huge changes. Fake rate decreases in transition region, increases in barrel. Note: Plots below exaggerate the fake rate increase in the barrel because the y axis changes.

The AUC of the model increases significantly (from .88 -> .91) from fixing the bug but this doesn't translate to significantly better performance. Tracks in the transition region become easier to reject than tracks in the barrel. It seems like the eta value of the hit shared between the two T3's offers a lot of rejection power in that region using radius matching (inner vs. bridge, outer vs. bridge)

Overall, I don't think there is a large difference in physics performance or timing from fixing the bug. If the fake rate increase in the barrel is a concern, the easiest solution would be to choose a tighter cut in the barrel to match the performance of the current DNN in that region.

[github-actions]

The PR was built and ran successfully with CMSSW. Here are some plots.

OOTB All Tracks

The full set of validation and comparison plots can be found here.