Method to check/track previous calibration

[jradcliffe5]

No description provided.

[jmoldon]

A good way to do this is defining a dictionary. Each key will be the name of a main step, and the value will be a 2d list with all the [table, spwmap] pairs that the step is expecting. Then at the beginning of each step we check that the calibration tables required are available. If something is missing it will log a warning or an error.

We need a simple function that given a step name and that dictionary, checks if the tables exist.

[jmoldon]

Each procedure (consisting of one or more CASA tasks) will have a predefined input and output.

The names of the calibration tables will be: caldir+inputs['inbase']_<name below>

task	shortname	inputs	outputs	content
do_delay	delay0	--	delay.K	Delay calibration
do_initial_bandpass	bpcal0	delay.K	bpcal0.G0	Phase calibration of bandpass calibrator
			bpcal0.G1	A&P calibration of bandpass calibrator
			bpcal0.B0	Initial bandpass table

An example of the dictionary that should contain what tables are produced by each task. I want to think about it a bit more before including this approach in the pipeline:

caltables = {}
caltables['delay0'] = {}
caltables['delay0']['tables'] = ['delay.K']   # This is a list
caltables['delay0']['spwmaps'] = [[0]*8]   # This is a list of lists

This can be the output and input of each task. That dictionary should be stored somewhere. It is not enough that check what tables are available because that does not contain information about the spwmap that they need when being applied.

[jmoldon]

Updated behaviour in #47
Calibration tasks are run using the caltables dictionary, which contains all the information on how the task was performed (table name, path, fields, mode, gaintype, solint etc.). Note that spwmap is defined before running the calibration task, so the right spwmap will be used when the table is applied. For example:

caltable_name = 'bpcal_d.K0'
caltables[caltable_name] = {}
caltables[caltable_name]['name'] = caltable_name
caltables[caltable_name]['table'] = caltables['calib_dir']+caltables['inbase']+'_'+caltable_name
caltables[caltable_name]['field'] = bpcal
caltables[caltable_name]['gaintype'] = 'K'
caltables[caltable_name]['calmode'] = 'p'
caltables[caltable_name]['solint'] = '120s'
caltables[caltable_name]['interp'] = 'linear'
caltables[caltable_name]['spwmap'] = [0]*caltables['num_spw']
caltables[caltable_name]['combine'] = 'spw'
caltables[caltable_name]['spw'] = ''

The caltable_name will identify each dictionary entry. So when we want to apply one or more tables, we just need to pass the table names using parameter previous_cal and the task will automatically know how to fill the fields: gaintable, interp, spwmap, and gainfield.

The caltables dictionary is saved as an external file caltables.pkl so the calibration can be restarted from any step and the calibration tables will still be found and correctly applied. As a security I also save the dictionaries in independent external files after each calibration step (this is just to avoid losing the dictionary if one tasks fails to end).