Merge pull request #120 from VERITAS-Observatory/0.2.0-dev

Refactoring and cleanup of code; improved performance for Eventdisplay
VERITAS-Observatory · Feb 16, 2022 · 71adbc4 · 71adbc4
2 parents ecd7322 + 6c4db1f
commit 71adbc4
Show file tree

Hide file tree

Showing 50 changed files with 2,382 additions and 1,977 deletions.
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
@@ -0,0 +1,83 @@
+---
+name: CI
+
+on: [pull_request]
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.8]
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install conda dependencies
+        env:
+          PYTHON_VERSION: ${{ matrix.python-version }}
+        run: |
+          source $CONDA/etc/profile.d/conda.sh
+          conda config --set always_yes yes --set changeps1 no
+          conda update -q conda
+          conda install -c conda-forge mamba
+          mamba env create -n v2dl3ED -f environment-eventdisplay.yml
+          conda activate v2dl3ED
+          conda info
+
+      - name: flake8
+        shell: bash -leo pipefail {0}
+        run: |
+          source $CONDA/etc/profile.d/conda.sh
+          conda activate v2dl3ED
+          conda install flake8
+          #flake8 . --count --exit-zero --max-line-length=90 --ignore=E203,W503 --select=W504 --show-source --statistics
+          flake8 . --count --select=E9,F63,F7,F82 --ignore=E203,W503 --show-source --statistics
+          flake8 . --count --exit-zero --max-complexity=25 --ignore=E203,W503 --max-line-length=127 --statistics
+
+      - name: pytest
+        shell: bash -leo pipefail {0}
+        env:
+          PYTHONPATH: .
+        run: |
+          source $CONDA/etc/profile.d/conda.sh
+          conda activate v2dl3ED
+          pytest .
+
+      - name: download test data
+        run: |
+          ./download_data_for_tests.sh
+
+      - name: V2Dl3-Eventdisplay
+        shell: bash -leo pipefail {0}
+        env:
+          PYTHONPATH: .
+        run: |
+          source $CONDA/etc/profile.d/conda.sh
+          conda activate v2dl3ED
+          # point-like tests
+          python pyV2DL3/script/v2dl3_for_Eventdisplay.py \
+               -f ./64080.anasum.root \
+                  ./effArea-v486-auxv01-CARE_June2020-Cut-NTel2-PointSource-Moderate-TMVA-BDT-GEO-V6_2012_2013a-ATM62-T1234-testCI.root \
+                  test-pointlike-CI.fits.gz
+          python notebooks/eventdisplay-testing/compareFitsFiles.py \
+                -f ED-0.2.0-pointlike-CI.fits.gz test-pointlike-CI.fits.gz \
+                -d test-pointlike.log
+          cat test-pointlike.log
+          # full-enclosure tests
+          python pyV2DL3/script/v2dl3_for_Eventdisplay.py --full-enclosure \
+               -f ./64080.anasum.root \
+                  ./effArea-v486-auxv01-CARE_June2020-Cut-NTel2-PointSource-Moderate-TMVA-BDT-GEO-V6_2012_2013a-ATM62-T1234-testCI.root \
+                  test-full-enclosure-CI.fits.gz
+          python notebooks/eventdisplay-testing/compareFitsFiles.py \
+                -f ED-0.2.0-fullenclosure-CI.fits.gz test-full-enclosure-CI.fits.gz \
+                -d test-full-enclosure.log
+          cat test-full-enclosure.log
+      
diff --git a/LICENSE b/LICENSE
@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2019, 2020, 2021 V2DL3 Developers
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/README.md b/README.md
@@ -1,124 +1,137 @@
-# V2DL3
-VERITAS (VEGAS and Eventdisplay) to DL3 Converter
+# V2DL3 - VERITAS (VEGAS and Eventdisplay) to DL3 Converter.
 
-Contact:
-	Ralph Bird (ralph.bird.1@gmail.com)
-	Tarek Hassan (tarek.hassan@desy.de)
-	Tony Lin (xyxlin@gmail.com)
-	Tobias Kleiner (tobias.kleiner@desy.de)
-
-## Aim
+V2DL3 is a tool to convert VERITAS data products to DL3 FITS format, allowing to use e.g. the [Gammapy science tools](https://gammapy.org/) for analysis. 
 
-This repository is for the code that will be used to convert VERITAS data into DL3 format. It can be used to convert point-like IRFs (as included in the joint-Crab paper), as well as full-enclosure IRFs.
+The converter can be used to convert point-like and full-enclosure IRFs. 
+The FITS output follows the data formats for gamma-ray astronomy as defined in open [gamma-astro-data-formats](https://github.com/open-gamma-ray-astro/gamma-astro-data-formats) (GADF) repository.
 
-The project follows the most recent DL3 format definition from the [open gamma-ray astro data formats repository](https://github.com/open-gamma-ray-astro/gamma-astro-data-formats).
+The projects tries to share as many tools as possible between VEGAS and Eventdisplay, especially those used for writing the FITS files.
 
----
-# pyV2DL3 
-
-The python package for converting stage5/anasum files to the DL3 FITS format. Other than useful functions that can be called, a commandline tool `v2dl3` comes with the package.
+The two main tools required to convert VERITAS data products to DL3 FITS format and use them with gammapy are:
+- converter to DL3 (`v2dl3` for VEGAS, `v2dl3_for_Eventdisplay.py` for Eventdisplay)
+- tool to generate observation index tables
 
-### Requirements
-The requirements are listed in the ```environment.yml``` file.
+For contributors: please note the section for developers below.
 
-#### VEGAS
+---
+# V2DL3 for VEGAS
 
 * vegas version >= 2.5.7
+* requirements are listed in the ```environment.yml``` file.
 
-#### EventDisplay
-
-* The converter does not depend on EventDisplay. However, make sure that the EventDisplay anasum stage runs with version >= v485 to include the DL3EventTree in the ansum file.
-
-### Install pyV2DL3
+## Installation
 
-To install the needed python dependencies, use of conda is recommended. The necessary python environment can be created from ```environment.yml```.
-
-Just run:
+Use the [conda package manager](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html) to install the dependenies:
 ```
 conda env create -f environment.yml
 ```
-and the environment ```v2dl3``` will be created. After activating the environment (`conda activate v2dl3`), install pyV2DL3 as follows:
+The environment ```v2dl3``` will be created and can be activated with:
 
 ```
-pip install .
+conda activate v2dl3
 ```
-### Usage of commandline tool v2dl3
 
+Install now pyV2DL3:
 ```
-Usage: v2dl3 [OPTIONS] <output>
-
-  Command line tool for converting stage5/anasum file to DL3
-
-  There are two modes:
-      1) Single file mode
-          When --file_pair is invoked, the path to the stage5/anasum file and the
-          corresponding effective area should be provided. The <output> argument
-          is then the resulting fits file name.
-      2) File list mode
-          When using the option --runlist, the path to a stage6/anasum runlist should be used.
-          The <output> is then the directory to which the fits files will be saved to.
-
-  Note: One one mode can be used at a time.
-
-Options:
-  -f, --file_pair PATH...  A stage5/anasum file (<file 1>) and the corresponding
-                           effective area (<file 2>).
-  -l, --runlist PATH       Stage6/anasum runlist
-  -g, --gen_index_file     Generate hdu and observation index list files. Only
-                           have effect in file list mode.
-  -m, --save_multiplicity  Save telescope multiplicity into event list
-  -e, --ed                 Eventdisplay mode
-  -d, --debug
-  -v, --verbose            Print root output
-  --help                   Show this message and exit.
+pip install . --use-feature=in-tree-build
 ```
 
----
-### Examples
+## The commandline tool v2dl3 with VEGAS
 
-### VEGAS
+Run `v2dl3 --help` to see all options.
 
 Make sure you have ROOT with pyROOT enabled and VEGAS(>=v2.5.7) installed to proceed.
 Now, lets create the DL3 fits files from the stage 5 files in the ```./VEGAS/``` folder. 
 
-##### One file at a time
+### One file at a time
 
 To convert a single stage 5 file to DL3 fits you need to provide the path to the stage 5 file as well as the corresponding effective area file using the flag ```-f```. The last argument is the name of the ouput DL3 file.
 
-
 ```
 v2dl3 -f ./VEGAS/54809.med.ED.050.St5_Stereo.root ./VEGAS/EA_na21stan_medPoint_050_ED_GRISU.root ./test.fits
 ```
 
-##### Generate from a VEGAS stage6 runlist
+### Generate from a VEGAS stage6 runlist
 
 You can also provide a stage6 runlist to the command line tool. In this case the last argument is the folder where all the output DL3 files will be saved. Beware that the file names for the outputs are inferred from the root file name (xxx.root -> xxx.fits)
 
 ```
 v2dl3 -l ./runlist.txt  ./test
 ```
 
-### EventDisplay
+---
 
-ROOT is installed directly into the environment from conda-forge following the above steps using the ```environment.yml```.
-Now, lets create the DL3 fits files from the anasum files in the ```./eventDisplay/``` folder. 
+# V2DL3 for EventDisplay
 
-##### One file at a time
+- use Eventdisplay version >= 487
+- recipes for Docker containers are available from the [V2DL3-Docker-recipes](https://github.com/Eventdisplay/V2DL3-Docker-recipes) repository.
 
-To convert an anasum file to DL3 you need to provide the path to the anasum file as well as the corresponding effective area file using the flag ```-f```. The only difference with VEGAS is that you need to add the '--ed' flag. The last argument is the name of the ouput DL3 file.
+## Installation
 
+Use the [conda package manager](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html) to install the dependenies:
+```
+conda env create -f environment-eventdisplay.yml
+```
+
+Activate the environment ```v2dl3ED``` and set `PYTHONPATH`:
 
 ```
-v2dl3 --ed -f ./eventDisplay/54809.anasum.root [Effective Area File] ./eventDisplay/54809.anasum.fits
+conda activate v2dl3ED
+export PYTHONPATH=$PYTHONPATH:"${PWD}"
 ```
 
-##### Full-enclosure
+Note that no pip is required for using the v2dl3 tool with Eventdisplay.
+
+## Usage of commandline tool v2dl3
+
+Run `python pyV2DL3/script/v2dl3_for_Eventdisplay.py --help` to see all options.
 
-For full-enclosure IRFs you need to pass the additional flag --full-enclosure and be sure to provide the proper effective area files:
+Convert an anasum output file to DL3.
+The following input is required:
+- anasum file for a given run
+- effective area file for the corresponding cut applied during the preparation of the anasum file (DL3 version)
+
+Example for point-like analysis:
+```
+python pyV2DL3/script/v2dl3_for_Eventdisplay.py -f 54809.anasum.root [Effective Area File] ./outputdir/54809.anasum.fits
+```
+Example for full-enclosure analysis:
 ```
-v2dl3 -ed --full-enclosure -f 64080.anasum.root $VERITAS_EVNDISP_AUX_DIR/EffectiveAreas/effArea-v485-auxv01-CARE_June2020-Cut-NTel2-PointSource-Hard-TMVA-BDT-GEO-V6_2012_2013a-ATM62-T1234.root ./FITS/64080.anasum.fits
+python pyV2DL3/script/v2dl3_for_Eventdisplay.py --full-enclosure -f 64080.anasum.root [Effective Area File] ./outputdir/64080.anasum.fits
 ```
 
+---
+# Data storage and generating index files
+
+Two index files are required for DL3-type analysis and can be generated with the tool `generate_index_file.py`.
+
+The tables are descriped on the [GADF website](https://gamma-astro-data-formats.readthedocs.io/en/v0.2/data_storage/index.html):
+- [Observation index table](https://gamma-astro-data-formats.readthedocs.io/en/v0.2/data_storage/obs_index/index.html)
+- [HDU index table](https://gamma-astro-data-formats.readthedocs.io/en/v0.2/data_storage/hdu_index/index.html)
+
+To use `generate_index_file.py`, run:
+- `generate_index_file --help` when using VEGAS
+- `python pyV2DL3/script/v2dl3_for_Eventdisplay.py --help` when using Eventdisplay 
+
+---
+
+# Contributing and Developing Code
+
+A few remarks when contributing code:
+- goal is to keep as much common code for converting from VEGAS or Eventdisplay data products
+- put package specific code into the [pyV2DL3/vegas](pyV2DL3/vegas) and [pyV2DL3/eventdisplay](pyV2DL3/eventdisplay) directories. As different environments are used for both packages, do not put any imports to vegas/eventdisplay in modules in pyV2DL3
+
+To ensure readability, we try follow the Python [PEP8](https://www.python.org/dev/peps/pep-0008/) style guide. 
+
+Functions and classes should contain a docstring with a short description.
+
+Unit tests are encouraged and are available for few cases at this point. Unit tests are in the tests directory and can be called using [pytest](http://docs.pytest.org/). 
+
+Use the [python logging system](https://docs.python.org/3/howto/logging.html) instead of the ‘print()’ function to output text. This allows to pipe all output into a log file and for different logging levels (INFO, DEBUG, …).
+
+---
+**TEXT BELOW REQUIRES REVIEW**
+
 ##### Multi file processing
 
 To convert many runs at once with different Effective Area files there is a modified anasum script here ( ``` VERITAS-Observatory/Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum_parallel_from_runlist_v2dl3.sh ``` ), that can be used to create a ``` v2dl3_for_runlist_from_ED485-anasum.sh ``` script. This script then contains one line for each processed file in the formatting as shown above in the full-enclosure case. 

diff --git a/download_data_for_tests.sh b/download_data_for_tests.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+set -e
+
+wget https://desycloud.desy.de/index.php/s/K5PkymyEJkFa46t/download/64080.anasum.root
+wget https://desycloud.desy.de/index.php/s/Ete9eC3Z448bXwd/download/effArea-v486-auxv01-CARE_June2020-Cut-NTel2-PointSource-Moderate-TMVA-BDT-GEO-V6_2012_2013a-ATM62-T1234-testCI.root
+wget https://desycloud.desy.de/index.php/s/cHQXEcNeN2ojFfR/download/ED-0.2.0-pointlike-CI.fits.gz
+wget https://desycloud.desy.de/index.php/s/4rF8r27QmpqmGQZ/download/ED-0.2.0-fullenclosure-CI.fits.gz
diff --git a/environment-eventdisplay.yml b/environment-eventdisplay.yml
@@ -0,0 +1,17 @@
+# installation:
+# create: conda env create -f environment-eventdisplay.yml
+# activate: conda activate v2dl3ED
+#
+name: v2dl3ED
+channels:
+  - astropy
+  - conda-forge
+dependencies:
+  - astropy>=4.0
+  - click
+  - numpy>=1.13
+  - pytest
+  - python>=3.8
+  - pyyaml
+  - scipy
+  - uproot
diff --git a/environment.yml b/environment.yml
@@ -11,6 +11,7 @@ dependencies:
   - numpy>=1.13
   - pandas
   - pkgconfig
+  - pytest
   - python>=3.8
   - pyyaml
   - root

diff --git a/notebooks/README.md b/notebooks/README.md
@@ -0,0 +1,10 @@
+# Example notebooks on how to use V2DL3 and gammpay
+
+Running those notebook requires gammapy and jupyter installed. 
+
+Do so by running:
+``
+conda env create -f environment.yml
+conda activate v2dl3_examples
+```
+