3.Look inside MACROS

00Raw2Np

Converts the $\verb+.dat+$ files from $\verb+/data/MONTH/raw+$ into $\verb+.npz+$ files at $\verb+/data/MONTH/npy+$

runs = []; channels = []
runs = np.append(runs,info["CALIB_RUNS"])
runs = np.append(runs,info["LIGHT_RUNS"])
runs = np.append(runs,info["ALPHA_RUNS"])
runs = np.append(runs,info["MUONS_RUNS"])

channels = np.append(channels,info["CHAN_TOTAL"])

if info["RAW_DATA"][0] == "DAT":
    print("----- Taking a .dat file as input data -----")
    binary2npy(runs.astype(int),channels.astype(int),info=info,compressed=True,force=False)

⏩ $\textcolor{orange}{RUN}$ (in macros folder)

   python3 00Raw2Np.py

✔️

If everything is OK you should get new folders $\verb+/data/Feb22_2/npy/runXX_chYY+$ and see a display in the terminal similar to:

(the same structure is implemented for all the macros)

where we see the destination file and if the different files existed previously, are overwritten etc...

01PreProcess

This macro will process the {RawPedestal, RawPeak} variables for the RawADC that are computed and saved in .npz files. You will see as terminal output the saved files. By default the saving is set to force = True and so if you pre-process files that already existed they will be overwritten.

# PRE-PROCESS RAW #
for run, ch in product(runs.astype(int),channels.astype(int)):
    # Start to load_runs 
    my_runs = load_npy([run],[ch], preset="RAW", info=info, compressed=True)
    
    compute_peak_variables(my_runs,key="RawADC",label="Raw")
    compute_pedestal_variables(my_runs,key="RawADC",label="Raw")
    print_keys(my_runs)
    delete_keys(my_runs,["RawADC"]) # Delete previous peak and pedestal variables
    save_proccesed_variables(my_runs,"ALL",info=info, force=False)
    del my_runs
    gc.collect()

⏩ $\textcolor{orange}{RUN}$ (in macros folder)

   python3 01PreProcess.py MergeDebug

02Processs

This macro will process the {Pedestal, Peak} variables for the ADC are computed and saved in .npz files. You will see as terminal output the saved files. By default the saving is set to force = True and so if you pre-process files that already existed they will be overwritten. However, we delete the RawKeys from the my_runs dictionary to save computing time (RawInfo don’t change).

# PROCESS WAVEFORMS (Run in loop to avoid memory issues)
for run, ch in product(runs.astype(int),channels.astype(int)):
    
    my_runs = load_npy([run],[ch],preset=str(info["LOAD_PRESET"][2]),info=info,compressed=True)
    compute_ana_wvfs(my_runs,debug=False)

    insert_variable(my_runs,np.ones(len(channels)),"PChannel") # Change polarity!
    compute_peak_variables(my_runs,key="ADC")                  # Compute new peak variables
    compute_pedestal_variables(my_runs,key="ADC",debug=False)  # Compute new ped variables
    
    # print_keys(my_runs)
    average_wvfs(my_runs,centering="NONE") # Compute average wvfs centering (choose from: "NONE", "PEAK", "THRESHOLD")

    delete_keys(my_runs,["RawADC",'RawPeakAmp', 'RawPeakTime', 'RawPedSTD', 'RawPedMean', 'RawPedMax', 'RawPedMin', 'RawPedLim','RawPChannel']) # Delete branches to avoid overwritting
    save_proccesed_variables(my_runs,preset=str(info["SAVE_PRESET"][2]),info=info, force=False) # Try preset ANA
    del my_runs
    gc.collect()

In this macro we also compute the charge with:

$\verb+integrate_wvfs(my_runs, ["ChargeAveRange"], "AveWvf", ["DAQ", 250], [0,100])+$

It will be used for calibrating the sensors in the following macro.

⏩ $\textcolor{orange}{RUN}$ (in macros folder)

   python3 02Process.py MergeDebug

03Integration

for run, ch in product(runs.astype(int),channels.astype(int)):

    my_runs = load_npy([run],[ch],preset=str(info["LOAD_PRESET"][3]),info=info,compressed=True)
    # print_keys(my_runs)

    ## Align indivual waveforms + Average ##
    # average_wvfs(my_runs,centering="PEAK") # Compute average wvfs VERY COMPUTER INTENSIVE!
    # average_wvfs(my_runs,centering="THRESHOLD") # Compute average wvfs EVEN MORE COMPUTER INTENSIVE!

    ## Charge Integration ##
    integrate_wvfs(my_runs, info = info) # Compute charge according to selected average wvf from input file ("AveWvf", "AveWvfPeak", "AveWvfThreshold")
    save_proccesed_variables(my_runs,preset=str(info["SAVE_PRESET"][3]),info=info, force=True)
    del my_runs
    gc.collect()

04Calibration

Compute a calibration ($\verb+ChargeAveRange [pC]+$) histogram where the peaks for the PED/1PE/2PE etc are fitted to obtain the gain.

for run, ch in product(runs.astype(int),channels.astype(int)):
    my_runs = load_npy([run],[ch], preset=str(info["LOAD_PRESET"][4]), info=info,compressed=True)
    
    print_keys(my_runs)

    ## Persistence Plot ##
    # vis_persistence(my_runs)

    ## Calibration ##
    print("Run ", run, "Channel ", ch)
    popt, pcov, perr = calibrate(my_runs,int_key,OPT)
    # Calibration parameters = mu,height,sigma,gain,sn0,sn1,sn2 ##
    calibration_txt(run, ch, popt, pcov, filename="gain",info=info)
    
    ## SPE Average Waveform ##
    if all(x !=-99 for x in popt):
        SPE_min_charge = popt[3]-abs(popt[5])
        SPE_max_charge = popt[3]+abs(popt[5])
        cut_min_max(my_runs, int_key, limits = {int_key[0]: [SPE_min_charge,SPE_max_charge]})
        average_wvfs(my_runs,centering="NONE",cut_label="SPE")

        save_proccesed_variables(my_runs,info=info,branch_list=["AveWvfSPE"])

The parameters are computed from the best fit parameters and covariance matrix obtained from the curve_fit function. These parameters are saved in a txt for each channel.

With the cuts functions we also compute the SPE waveform (that can be visualized with the Vis macros)

⏩ $\textcolor{orange}{RUN}$ (in macros folder)

   python3 03Calibration.py MergeDebug

If everything is working as it should toy should obtain the following histograms (raw and fitted) and a txt file with the calibration parameters (if confirmed when asked through terminal)

✔️ $\textcolor{lightgreen}{OK}$ Everything is working well :)

05Scintillation

## Visualize average waveforms by runs/channels ##
my_runs = load_npy(runs.astype(int),channels.astype(int),branch_list=["Label","Sampling","AveWvf"],info=info,compressed=True) #Remember to LOAD your wvf
vis_compare_wvf(my_runs, ["AveWvf"], compare="RUNS", OPT=OPT)

for run, ch in product(runs.astype(int),channels.astype(int)):
    my_runs = load_npy([run],[ch], branch_list=["ADC","TimeStamp","Sampling","ChargeAveRange", "NEventsChargeAveRange","AveWvf"], info=info,compressed=True)  #preset="ANA"
    print_keys(my_runs)

    ## Integrated charge (scintillation runs) ##
    print("Run ", run, "Channel ", ch)

    popt_ch = []; pcov_ch = []; perr_ch = []; popt_nevt = []; pcov_nevt = []; perr_nevt = []
    popt, pcov, perr = charge_fit(my_runs, int_key, OPT); popt_ch.append(popt); pcov_ch.append(pcov); perr_ch.append(perr)
    
    scintillation_txt(run, ch, popt_ch, pcov_ch, filename="pC", info=info) ## Charge parameters = mu,height,sigma,nevents ## 

06Deconvolution

Before runing the deconvolution macro make sure you have a clean laser/led signal that can be used as a template. The macro will load the alpha runs and rescale the light signal to the SPE amplitude according to AveWvfSPE calculated at calibration stage.

• Firstly, the average wvfs are deconvolved. From these the gauss filter cut-off (from the wiener filter fit) is extracted and saved.
• Secondly, the previous calculated gauss filter is applied to deconvolve the ADC wvfs.
• Finally, the deconvolved wvfs are saved for further process using the standard workflow.

for idx, run in enumerate(raw_runs):
    for jdx, ch in enumerate(ana_ch):
        my_runs = load_npy([run],[ch],preset=str(info["LOAD_PRESET"][6]),info=info,compressed=True)  # Select runs to be deconvolved (tipichaly alpha)     

        if "SiPM" in str(my_runs[run][ch]["Label"]):
            light_runs =  load_npy([dec_runs[SiPM_OV]],[ch],preset="EVA",info=info,compressed=True) # Select runs to serve as dec template (tipichaly light)    
            single_runs = load_npy([ref_runs[SiPM_OV]],[ch],preset="EVA",info=info,compressed=True) # Select runs to serve as dec template scaling (tipichaly SPE)   
        elif "SC" in str(my_runs[run][ch]["Label"]):
            light_runs =  load_npy([dec_runs[idx]],[ch],preset="EVA",info=info,compressed=True) # Select runs to serve as dec template (tipichaly light)    
            single_runs = load_npy([ref_runs[idx]],[ch],preset="EVA",info=info,compressed=True) # Select runs to serve as dec template scaling (tipichaly SPE)
        else:
            print("UNKNOWN DETECTOR LABEL!")
        
        keys = ["AveWvf","SER","AveWvf"] # keys contains the 3 labels required for deconvolution keys[0] = raw, keys[1] = det_response and keys[2] = deconvolution 

        generate_SER(my_runs, light_runs, single_runs)

        OPT = {
            "NOISE_AMP": 1,
            "FIX_EXP":True,
            "LOGY":True,
            "NORM":False,
            "FOCUS":False,
            "SHOW": True,
            "SHOW_F_SIGNAL":True,
            "SHOW_F_GSIGNAL":True,
            "SHOW_F_DET_RESPONSE":True,
            "SHOW_F_GAUSS":True,
            "SHOW_F_WIENER":True,
            "SHOW_F_DEC":True,
            "WIENER_BUFFER": 800,
            "THRLD": 1e-4,
            }

        deconvolve(my_runs,keys=keys,OPT=OPT)

        OPT = {
            "SHOW": False,
            "FIXED_CUTOFF": True
            }

        keys[0] = "ADC"
        keys[2] = "ADC"
        deconvolve(my_runs,keys=keys,OPT=OPT)

        save_proccesed_variables(my_runs,preset=str(info["SAVE_PRESET"][6]),info=info,force=True)
        del my_runs,light_runs,single_runs

generate_input_file(input_file,info,label="Gauss")

0XVisEvent

We can visualize the individual events from the moment we pre-process the waveforms and obtain RawInfo.

info     = read_input_file(input_file)
runs     = [int(r) for r in input_runs.split(",")]
channels = [int(c) for c in input_channels.split(",")]

OPT  = {
    "MICRO_SEC":   True,
    "NORM":        False,                # Runs can be displayed normalised (True/False)
    "LOGY":        False,                # Runs can be displayed in logy (True/False)
    "SHOW_AVE":    "AveWvf",             # If computed, vis will show average (AveWvf,AveWvfSPE,etc.)
    "SHOW_PARAM":  True,                 # Print terminal information (True/False)
    "CHARGE_KEY":  "ChargeAveRange",     # Select charge info to be displayed. Default: "ChargeAveRange" (if computed)
    "PEAK_FINDER": False,                # Finds possible peaks in the window (True/False)
    "LEGEND":      False                 # Shows plot legend (True/False)
    }
###################################

##### LOAD RUNS #####
my_runs = load_npy(runs,channels,preset="ANA",info=info,compressed=True) # preset could be RAW or ANA
#####################

##### EVENT VISUALIZER #####
vis_npy(my_runs, ["ADC"],-1,OPT=OPT) # Remember to change key accordingly (ADC or RawADC)
############################

⏩ $\textcolor{orange}{RUN}$ (in macros folder)

   python3 0XVisEvent.py MergeDebug 1 0,1

The individual events of different channels can be visualized together and with AveWvf superposed.

This macro could be used to visualize histograms (with/without applying cuts). In the following picture you can see different options (not all at the same time):

💡 $\color{white}{Tips:}$

1. Make the zoom you need with the 🔍
2. This will add one point you need to delete with the right bottom of your mouse
3. Select the area you want to keep by adding points with the left bottom
4. When you are ready click on the mouse wheel to confirm your selection