Merge pull request #1604 from pypeit/sensfunc_blaze_jwst_lists

Improvements to HIRES, JWST, fluxing, and coadding
pypeit · Aug 2, 2023 · 89fe8d4 · 89fe8d4
2 parents d5d06c7 + 902cca3
commit 89fe8d4
Show file tree

Hide file tree

Showing 162 changed files with 6,213 additions and 3,696 deletions.
diff --git a/CHANGES.rst b/CHANGES.rst
@@ -11,10 +11,21 @@
 - Hotfix for skysub regions GUI that used np.bool
 - Hotfix to stop pypeit_setup from crashing on data from lbt_luci1, lbt_luci2, magellan_fire,
   magellan_fire_long, p200_tspec, or vlt_sinfoni.
+- Adds Keck/ESI to PypeIt
 - Instrumental FWHM map is calculated and output in ``Calibrations`` and ``spec1d`` files.
 - Adds Keck/ESI to PypeIt
 - Add MDM/Modspec spectrograph
 - Store user-generated wavelength solution in pypeit cache
+- Improvements to wavelength grids and masking in coadd routines.
+- Fixed a bug in echelle coadding where the wrong coadded spectra were being
+  used in final stacks.
+- Sensitivity function models can now be computed relative to the blaze
+  spectrum.
+- Refactored coadding routines to work with lists to support coadding data from
+  different setups.
+- Changes to how masking is dealt with in extraction to fix a bug in how masks
+  were being treated for echelle data
+- Various fixes and changes required to add more support for Keck/HIRES and JWST
 - Fix a bug in ``spectrograph.select_detectors``, where a list of ``slitspatnum`` could not be used.
 - Improvements in 2D coaddition
     - Fix a bug in `pypeit_setup_coadd2d` for the output file name of the .coadd2d file
@@ -23,6 +34,7 @@
     - Added ``exclude_slits`` parameter in `pypeit_coadd_2dspec` to exclude specific slits
     - Fix wrong RA & Dec for 2D coadded serendips
 
+
 1.13.0 (2 June 2023)
 --------------------
 
@@ -31,7 +43,6 @@
 - Allow user control of the local sky subtraction window
 - Deprecate use of python 3.8 with PypeIt, allow python 3.11
 - Make pypeit_show_2dspec (somewhat) backwards compatible.
-- Added the option to disable strict version checking for 1d coadds.
 - Hotfix for KCWI when using alignment (aka ContBars) frames for the astrometric correction.
 - Sensitivity function masking and output updates
 - Fixed a bug in the `variance_model` calculation for combined images.
@@ -51,7 +62,8 @@
 - The ``'calib'`` column is now always added to the pypeit file, regardless of
   whether or not you also request the ``'comb_id'`` and ``'bkg_id'`` columns.
 - Names of associated calibration frames now written to ``spec2d`` file headers.
-- Major quicklook updates.  ql_multislit.py deprecated.
+- Added the option to disable strict version checking for 1d coadds.
+- Major quicklook updates.  ql_multislit.py temporarily deprecated.
 - Improve speed in ginga visualization of traces and added
   `pypeit_chk_tilts`. Note that this script uses an update
   of the tilts datamodel, so it will not work on older reductions.

diff --git a/doc/api/pypeit.scripts.arxiv_solution.rst b/doc/api/pypeit.scripts.arxiv_solution.rst
@@ -0,0 +1,8 @@
+pypeit.scripts.arxiv\_solution module
+=====================================
+
+.. automodule:: pypeit.scripts.arxiv_solution
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.scripts.ql_multislit.rst b/doc/api/pypeit.scripts.ql_multislit.rst
diff --git a/doc/api/pypeit.scripts.rst b/doc/api/pypeit.scripts.rst
@@ -7,6 +7,7 @@ Submodules
 .. toctree::
    :maxdepth: 4
 
+   pypeit.scripts.arxiv_solution
    pypeit.scripts.cache_github_data
    pypeit.scripts.chk_alignments
    pypeit.scripts.chk_edges
@@ -34,7 +35,6 @@ Submodules
    pypeit.scripts.parse_slits
    pypeit.scripts.qa_html
    pypeit.scripts.ql
-   pypeit.scripts.ql_multislit
    pypeit.scripts.run_pypeit
    pypeit.scripts.scriptbase
    pypeit.scripts.sensfunc

diff --git a/doc/api/pypeit.spectrographs.keck_esi.rst b/doc/api/pypeit.spectrographs.keck_esi.rst
@@ -0,0 +1,8 @@
+pypeit.spectrographs.keck\_esi module
+=====================================
+
+.. automodule:: pypeit.spectrographs.keck_esi
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.spectrographs.mdm_modspec.rst b/doc/api/pypeit.spectrographs.mdm_modspec.rst
@@ -0,0 +1,8 @@
+pypeit.spectrographs.mdm\_modspec module
+========================================
+
+.. automodule:: pypeit.spectrographs.mdm_modspec
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.spectrographs.rst b/doc/api/pypeit.spectrographs.rst
@@ -15,6 +15,7 @@ Submodules
    pypeit.spectrographs.jwst_nircam
    pypeit.spectrographs.jwst_nirspec
    pypeit.spectrographs.keck_deimos
+   pypeit.spectrographs.keck_esi
    pypeit.spectrographs.keck_hires
    pypeit.spectrographs.keck_kcwi
    pypeit.spectrographs.keck_lris
@@ -26,6 +27,7 @@ Submodules
    pypeit.spectrographs.ldt_deveny
    pypeit.spectrographs.magellan_fire
    pypeit.spectrographs.magellan_mage
+   pypeit.spectrographs.mdm_modspec
    pypeit.spectrographs.mdm_osmos
    pypeit.spectrographs.mmt_binospec
    pypeit.spectrographs.mmt_bluechannel

diff --git a/doc/calibrations/image_proc.rst b/doc/calibrations/image_proc.rst
@@ -198,7 +198,7 @@ in our documentation as ``(nspec,nspat)``.  The operations required to
 flip/transpose the image arrays to match the PypeIt convention are dictated by
 instrument-specific :class:`~pypeit.images.detector_container.DetectorContainer`
 parameters and performed by
-:func:`~pypeit.spectrograph.spectrographs.Spectrograph.orient_image`.  Image
+:func:`~pypeit.spectrographs.spectrograph.Spectrograph.orient_image`.  Image
 orientation will be performed if the ``orient`` parameter is true.
 
 .. warning::

diff --git a/doc/calibrations/tilt.rst b/doc/calibrations/tilt.rst
@@ -58,7 +58,7 @@ Current TiltImage Data Model
 ============================
 
 Internally, the image is held in
-:class:`pypeit.tiltimage.TiltImage`
+:class:`pypeit.images.buildimage.tiltimage.TiltImage`
 which subclasses from :class:`pypeit.images.pypeitimage.PypeItImage` and
 :class:`pypeit.datamodel.DataContainer`.
 

diff --git a/doc/conf.py b/doc/conf.py
@@ -120,9 +120,16 @@
 
 # When nit-picking, ignore these warnings:
 nitpick_ignore = [ ('py:class', 'optional'),
+                   ('py:class', 'iterable'),
+                   ('py:class', 'ndarray'),
+                   ('py:class', 'dict-like'),
                    ('py:class', 'array-like'),
+                   ('py:class', 'table-like'),
+                   ('py:class', 'float-like'),
                    ('py:class', 'scalar-like'),
-                   ('py:class', 'default') ]
+                   ('py:class', 'default'),
+                   ('py:class', 'callable'),
+                    ]
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.

diff --git a/doc/fluxing.rst b/doc/fluxing.rst
@@ -589,6 +589,5 @@ FluxSpec Class
 ==============
 
 The guts of the flux algorithms are guided by the
-:class:`~pypeit.fluxcalibrate.FluxCalibrate`
-class.
+:func:`~pypeit.fluxcalibrate.apply_flux_calib` class.
 
diff --git a/doc/help/pypeit_cache_github_data.rst b/doc/help/pypeit_cache_github_data.rst
@@ -10,13 +10,14 @@
       spectrograph    A valid spectrograph identifier: bok_bc, gemini_flamingos1,
                       gemini_flamingos2, gemini_gmos_north_e2v,
                       gemini_gmos_north_ham, gemini_gmos_north_ham_ns,
-                      gemini_gmos_south_ham, gemini_gnirs, gtc_maat, gtc_osiris,
-                      gtc_osiris_plus, jwst_nircam, jwst_nirspec, keck_deimos,
-                      keck_hires, keck_kcwi, keck_lris_blue, keck_lris_blue_orig,
-                      keck_lris_red, keck_lris_red_mark4, keck_lris_red_orig,
-                      keck_mosfire, keck_nires, keck_nirspec_low, lbt_luci1,
-                      lbt_luci2, lbt_mods1b, lbt_mods1r, lbt_mods2b, lbt_mods2r,
-                      ldt_deveny, magellan_fire, magellan_fire_long, magellan_mage,
+                      gemini_gmos_south_ham, gemini_gnirs_echelle, gemini_gnirs_ifu,
+                      gtc_maat, gtc_osiris, gtc_osiris_plus, jwst_nircam,
+                      jwst_nirspec, keck_deimos, keck_esi, keck_hires, keck_kcwi,
+                      keck_lris_blue, keck_lris_blue_orig, keck_lris_red,
+                      keck_lris_red_mark4, keck_lris_red_orig, keck_mosfire,
+                      keck_nires, keck_nirspec_low, lbt_luci1, lbt_luci2,
+                      lbt_mods1b, lbt_mods1r, lbt_mods2b, lbt_mods2r, ldt_deveny,
+                      magellan_fire, magellan_fire_long, magellan_mage, mdm_modspec,
                       mdm_osmos_mdm4k, mmt_binospec, mmt_bluechannel, mmt_mmirs,
                       not_alfosc, not_alfosc_vert, ntt_efosc2, p200_dbsp_blue,
                       p200_dbsp_red, p200_tspec, shane_kast_blue, shane_kast_red,

diff --git a/doc/help/pypeit_chk_for_calibs.rst b/doc/help/pypeit_chk_for_calibs.rst
@@ -17,14 +17,15 @@
                             gemini_flamingos1, gemini_flamingos2,
                             gemini_gmos_north_e2v, gemini_gmos_north_ham,
                             gemini_gmos_north_ham_ns, gemini_gmos_south_ham,
-                            gemini_gnirs, gtc_maat, gtc_osiris, gtc_osiris_plus,
-                            jwst_nircam, jwst_nirspec, keck_deimos, keck_hires,
-                            keck_kcwi, keck_lris_blue, keck_lris_blue_orig,
-                            keck_lris_red, keck_lris_red_mark4, keck_lris_red_orig,
-                            keck_mosfire, keck_nires, keck_nirspec_low, lbt_luci1,
-                            lbt_luci2, lbt_mods1b, lbt_mods1r, lbt_mods2b,
-                            lbt_mods2r, ldt_deveny, magellan_fire,
-                            magellan_fire_long, magellan_mage, mdm_osmos_mdm4k,
+                            gemini_gnirs_echelle, gemini_gnirs_ifu, gtc_maat,
+                            gtc_osiris, gtc_osiris_plus, jwst_nircam, jwst_nirspec,
+                            keck_deimos, keck_esi, keck_hires, keck_kcwi,
+                            keck_lris_blue, keck_lris_blue_orig, keck_lris_red,
+                            keck_lris_red_mark4, keck_lris_red_orig, keck_mosfire,
+                            keck_nires, keck_nirspec_low, lbt_luci1, lbt_luci2,
+                            lbt_mods1b, lbt_mods1r, lbt_mods2b, lbt_mods2r,
+                            ldt_deveny, magellan_fire, magellan_fire_long,
+                            magellan_mage, mdm_modspec, mdm_osmos_mdm4k,
                             mmt_binospec, mmt_bluechannel, mmt_mmirs, not_alfosc,
                             not_alfosc_vert, ntt_efosc2, p200_dbsp_blue,
                             p200_dbsp_red, p200_tspec, shane_kast_blue,

diff --git a/doc/help/pypeit_coadd_1dspec.rst b/doc/help/pypeit_coadd_1dspec.rst
@@ -8,13 +8,18 @@
     Coadd 1D spectra produced by PypeIt
     
     positional arguments:
-      coadd1d_file          File to guide coadding process. This file must have the
+      coadd1d_file          File to guide coadding process.
+                             
+                            ------------------------
+                                   MultiSlit
+                            ------------------------
+                             
+                            For coadding Multislit spectra the file must have the
                             following format (see docs for further details including
                             the use of paths):
                              
                             [coadd1d]
                                coaddfile='output_filename.fits' # Optional
-                               sensfuncfile = 'sensfunc.fits' # Required only for Echelle
                              
                                coadd1d read
                                     filename | obj_id
@@ -46,6 +51,52 @@
                             inspect the spec1d_*.txt files or run pypeit_show_1dspec
                             spec1dfile --list
                              
+                            ------------------------
+                                     Echelle
+                            ------------------------
+                             
+                            For coadding Echelle spectra the file must have the
+                            following format (see docs for further details):
+                             
+                            [coadd1d]
+                               coaddfile='output_filename.fits' # Optional
+                             
+                               coadd1d read
+                                    filename | obj_id | sensfile  | setup_id 
+                                 spec1dfile1 | objid1 | sensfile1 | setup_id1
+                                 spec1dfile2 | objid2 | sensfile2 | setup_id2
+                                 spec1dfile3 | objid3 | sensfile3 | setup_id3
+                                    ...    
+                               coadd1d end
+                             
+                             OR the coadd1d read/end block can look like
+                             
+                              coadd1d read
+                                    filename | obj_id | sensfile  | setup_id
+                                 spec1dfile1 | objid1 | sensfile  | setup_id
+                                 spec1dfile2 |        |           |         
+                                 spec1dfile3 |        |           |         
+                                    ...    
+                               coadd1d end
+                             
+                            That is the coadd1d block is a four column list of
+                            spec1dfiles, objids, sensitivity function files, and
+                            setup_ids, but you can specify only a single objid,
+                            sensfile, and setup_id for all spec1dfiles on the first
+                            line
+                             
+                            Here:
+                             
+                            spec1dfile: full path to a PypeIt spec1dfile
+                             
+                            objid: the object identifier (see details above)
+                             
+                            sensfile: full path to a PypeIt sensitivity function
+                            file for the echelle setup in question
+                             
+                            setup_id: string identifier for the echelle setup in
+                            question, i.e. 'VIS', 'NIR', or 'UVB'
+                             
                             If the coaddfile is not given the output file will be
                             placed in the same directory as the first spec1d file.
                              

diff --git a/doc/help/pypeit_flux_calib.rst b/doc/help/pypeit_flux_calib.rst
@@ -1,7 +1,7 @@
 .. code-block:: console
 
     $ pypeit_flux_calib -h
-    usage: pypeit_flux_calib [-h] [--debug] [--par_outfile] [-v VERBOSITY] flux_file
+    usage: pypeit_flux_calib [-h] [--par_outfile] [-v VERBOSITY] flux_file
     
     Flux calibrate 1D spectra produced by PypeIt
     
@@ -48,7 +48,6 @@
     
     options:
       -h, --help            show this help message and exit
-      --debug               show debug plots?
       --par_outfile         Output to save the parameters
       -v VERBOSITY, --verbosity VERBOSITY
                             Verbosity level between 0 [none] and 2 [all]. Default:

diff --git a/doc/help/pypeit_flux_setup.rst b/doc/help/pypeit_flux_setup.rst
@@ -1,15 +1,21 @@
 .. code-block:: console
 
     $ pypeit_flux_setup -h
-    usage: pypeit_flux_setup [-h] [--objmodel {qso,star,poly}] sci_path
+    usage: pypeit_flux_setup [-h] [--name NAME] [--objmodel {qso,star,poly}]
+                             paths [paths ...]
     
-    Setup to perform flux calibration
+    Setup configuration files to perform flux calibration, 1D coadding, and telluric
+    correction.
     
     positional arguments:
-      sci_path              Path for Science folder
+      paths                 One or more paths for Science folders or sensitivity
+                            functions. Sensitivity functions must start with 'sens_'
+                            to be detected.
     
     options:
       -h, --help            show this help message and exit
+      --name NAME           The base name to use for the output files. Defaults to
+                            the instrument name is used.
       --objmodel {qso,star,poly}
                             science object model used in the telluric fitting. The
                             options are:

diff --git a/doc/help/pypeit_obslog.rst b/doc/help/pypeit_obslog.rst
@@ -14,14 +14,15 @@
                             gemini_flamingos1, gemini_flamingos2,
                             gemini_gmos_north_e2v, gemini_gmos_north_ham,
                             gemini_gmos_north_ham_ns, gemini_gmos_south_ham,
-                            gemini_gnirs, gtc_maat, gtc_osiris, gtc_osiris_plus,
-                            jwst_nircam, jwst_nirspec, keck_deimos, keck_hires,
-                            keck_kcwi, keck_lris_blue, keck_lris_blue_orig,
-                            keck_lris_red, keck_lris_red_mark4, keck_lris_red_orig,
-                            keck_mosfire, keck_nires, keck_nirspec_low, lbt_luci1,
-                            lbt_luci2, lbt_mods1b, lbt_mods1r, lbt_mods2b,
-                            lbt_mods2r, ldt_deveny, magellan_fire,
-                            magellan_fire_long, magellan_mage, mdm_osmos_mdm4k,
+                            gemini_gnirs_echelle, gemini_gnirs_ifu, gtc_maat,
+                            gtc_osiris, gtc_osiris_plus, jwst_nircam, jwst_nirspec,
+                            keck_deimos, keck_esi, keck_hires, keck_kcwi,
+                            keck_lris_blue, keck_lris_blue_orig, keck_lris_red,
+                            keck_lris_red_mark4, keck_lris_red_orig, keck_mosfire,
+                            keck_nires, keck_nirspec_low, lbt_luci1, lbt_luci2,
+                            lbt_mods1b, lbt_mods1r, lbt_mods2b, lbt_mods2r,
+                            ldt_deveny, magellan_fire, magellan_fire_long,
+                            magellan_mage, mdm_modspec, mdm_osmos_mdm4k,
                             mmt_binospec, mmt_bluechannel, mmt_mmirs, not_alfosc,
                             not_alfosc_vert, ntt_efosc2, p200_dbsp_blue,
                             p200_dbsp_red, p200_tspec, shane_kast_blue,

diff --git a/doc/help/pypeit_ql.rst b/doc/help/pypeit_ql.rst
@@ -21,14 +21,15 @@
                             gemini_flamingos1, gemini_flamingos2,
                             gemini_gmos_north_e2v, gemini_gmos_north_ham,
                             gemini_gmos_north_ham_ns, gemini_gmos_south_ham,
-                            gemini_gnirs, gtc_maat, gtc_osiris, gtc_osiris_plus,
-                            jwst_nircam, jwst_nirspec, keck_deimos, keck_hires,
-                            keck_kcwi, keck_lris_blue, keck_lris_blue_orig,
-                            keck_lris_red, keck_lris_red_mark4, keck_lris_red_orig,
-                            keck_mosfire, keck_nires, keck_nirspec_low, lbt_luci1,
-                            lbt_luci2, lbt_mods1b, lbt_mods1r, lbt_mods2b,
-                            lbt_mods2r, ldt_deveny, magellan_fire,
-                            magellan_fire_long, magellan_mage, mdm_osmos_mdm4k,
+                            gemini_gnirs_echelle, gemini_gnirs_ifu, gtc_maat,
+                            gtc_osiris, gtc_osiris_plus, jwst_nircam, jwst_nirspec,
+                            keck_deimos, keck_esi, keck_hires, keck_kcwi,
+                            keck_lris_blue, keck_lris_blue_orig, keck_lris_red,
+                            keck_lris_red_mark4, keck_lris_red_orig, keck_mosfire,
+                            keck_nires, keck_nirspec_low, lbt_luci1, lbt_luci2,
+                            lbt_mods1b, lbt_mods1r, lbt_mods2b, lbt_mods2r,
+                            ldt_deveny, magellan_fire, magellan_fire_long,
+                            magellan_mage, mdm_modspec, mdm_osmos_mdm4k,
                             mmt_binospec, mmt_bluechannel, mmt_mmirs, not_alfosc,
                             not_alfosc_vert, ntt_efosc2, p200_dbsp_blue,
                             p200_dbsp_red, p200_tspec, shane_kast_blue,