[Newbie] Running it for the first time in EVN data

[bmarcote]

Hi Jack,
I am running the pipeline for the first time on EVN data so I am likely doing some things wrong but I cannot tell what.

I have an EVN data set, divided in three FITS IDI files, and with the antab and uvflg files already in the same directory (CWD) from where I run the edited input files too.

I am running all steps (= 1) except the primary beam corrections.
Then if I read correctly, I should run (for my system):

/opt/casa-6.4.3-27/bin/python3 ~/Programing/VLBI_pipeline/run_vlbi_pipe.py ./vlbi_pipe_inputs.txt
bash vp_runfile.bash

However, already with the first command I get the following error:

>>> /opt/casa-6.4.3-27/bin/python3 ~/Programing/VLBI_pipeline/run_vlbi_pipe.py ./vlbi_pipe_inputs.txt
Telemetry initialized. Telemetry will send anonymized usage statistics to NRAO.
You can disable telemetry by adding the following line to the config.py file in your rcdir (e.g. ~/.casa/config.py):
telemetry_enabled = False
--> CrashReporter initialized.
2022-03-01 15:37:33	INFO	vp_run_vlbi_pipe::/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py::casa	Initialising VLBI pipeline run
2022-03-01 15:37:33	INFO	vp_run_vlbi_pipe::/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py::casa	No previous steps have been run - creating logger
2022-03-01 15:37:33	INFO	vp_run_vlbi_pipe::rmdirs::casa	Directory/table /jop93_2/frbr6_evn/ek050b/logs found - deleting
2022-03-01 15:37:33	INFO	vp_run_vlbi_pipe::/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py::casa	Writing script for step: prepare_data
2022-03-01 15:37:33	INFO	vp_run_vlbi_pipe::/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py::casa	Writing script for step: import_fitsidi
2022-03-01 15:37:33	INFO	vp_run_vlbi_pipe::/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py::casa	Writing script for step: make_mms
2022-03-01 15:37:33	INFO	vp_run_vlbi_pipe::/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py::casa	Writing script for step: apriori_cal
2022-03-01 15:37:33	INFO	vp_run_vlbi_pipe::/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py::casa	Writing script for step: init_flag
2022-03-01 15:37:33	SEVERE	/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py::open::ek050b.ms/ANTENNA	Exception Reported: Table ek050b.ms/ANTENNA does not exist
Traceback (most recent call last):
  File "/home/marcote/Programing/VLBI_pipeline/run_vlbi_pipe.py", line 72, in <module>
    write_commands(step=i,inputs=inputs,params=params,parallel=parallel,aoflag='both',casa6=casa6)
  File "/home/marcote/Programing/VLBI_pipeline/VLBI_pipe_functions.py", line 383, in write_commands
    msinfo = get_ms_info(msfile)
  File "/home/marcote/Programing/VLBI_pipeline/VLBI_pipe_functions.py", line 516, in get_ms_info
    tb.open('%s/ANTENNA'%msfile)
  File "/opt/casa-6.4.3-27/lib/py/lib/python3.8/site-packages/casatools/table.py", line 312, in open
    return self._swigobj.open(_str_ec(tablename), _dict_ec(lockoptions), nomodify)
  File "/opt/casa-6.4.3-27/lib/py/lib/python3.8/site-packages/casatools/__casac__/table.py", line 446, in open
    return _table.table_open(self, *args, **kwargs)
RuntimeError: Table ek050b.ms/ANTENNA does not exist

As I understood, it should not run the steps (the second line would do), and hence that's why the MS does not exist (yet).
However, even when I run manually job_import_fitsidi.bash, I noticed that it finds the three FITS IDI files but actually removes at some point the latest one and then complains that it does not exist:

2022-03-01 15:56:46	INFO	vp_import_fitsidi::/home/marcote/Programing/VLBI_pipeline/run_import_fitsidi.py::casa	Searching for location of fitsidifiles
2022-03-01 15:56:46	INFO	vp_import_fitsidi::/home/marcote/Programing/VLBI_pipeline/run_import_fitsidi.py::casa	Fitsidifiles to be located in .
2022-03-01 15:56:46	INFO	vp_import_fitsidi::find_fitsidi::casa	Will attempt to find fitsidifiles in .
2022-03-01 15:56:46	INFO	vp_import_fitsidi::find_fitsidi::casa	FOUND - ek050b_1_1.IDI3
2022-03-01 15:56:46	INFO	vp_import_fitsidi::find_fitsidi::casa	FOUND - ek050b_1_1.IDI2
2022-03-01 15:56:46	INFO	vp_import_fitsidi::find_fitsidi::casa	FOUND - ek050b_1_1.IDI1
2022-03-01 15:56:46	INFO	vp_import_fitsidi::/home/marcote/Programing/VLBI_pipeline/run_import_fitsidi.py::casa	Fitsidifiles in different directory. Moving to cwd.
2022-03-01 15:56:46	INFO	vp_import_fitsidi::rmfiles::casa	File /jop93_2/frbr6_evn/ek050b/ek050b_1_1.IDI3 found - deleting
rsync: [sender] link_stat "/jop93_2/frbr6_evn/ek050b/ek050b_1_1.IDI3" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]
cp: cannot stat './ek050b_1_1.IDI3': No such file or directory
2022-03-01 15:56:47	SEVERE	vp_import_fitsidi::/home/marcote/Programing/VLBI_pipeline/run_import_fitsidi.py::casa	Could not move files with rsync or cp ... exiting

Am I missing something obvious here?

[jradcliffe5]

Hi Benito,

For the first issue, the pipeline needs to get the metadata before writing the init_flag batch script. Aoflagger needs to know the field index in the measurement set so needs the metadata to convert between source name and the source index. At the moment, it requires the measurement set, so I often just run the prepare_data and importfits steps first. The function that pulls this information from the IDI files directly and will negate this issue is still under development.

For the second issue, r.e. prepare_data. I originally coded it so that you have a directory where you have untampered raw data and then it gets transferred to the cwd to get adjusted. This works well for the compute architecture I use where you have faster SSD/strip processing and a slower RAID storage. For your case, there should be an error brought up to prevent the deleting the IDI files (as you've found out) and to tell the user to move the raw IDI files to a separate folder.

[jradcliffe5]

Just to add to this, my documentation is rubbish and is on my to-do list. Teaching is sapping all of my time at the moment.