Merge branch 'develop' into coadd1d_badspec1d

CHANGES.rst

@@ -1,5 +1,5 @@
-1.13.1dev (6 June 2023)
-------------------------
+1.13.1dev
+---------
 
 - Add support for Gemini/GNIRS (IFU)
 - Added a script to convert a wavelength solution into something that can be placed in the reid archive.
@@ -33,6 +33,13 @@
     - Now ``only_slits`` parameter in `pypeit_coadd_2dspec` includes the detector number (similar to ``slitspatnum``)
     - Added ``exclude_slits`` parameter in `pypeit_coadd_2dspec` to exclude specific slits
     - Fix wrong RA & Dec for 2D coadded serendips
+- Changed calibration frame naming as an attempt to avoid very long names for
+  files with many calibration groups.  Sequential numbers are reduced to a
+  range; e.g., ``'0-1-2-3-4'`` becomes ``'0+4'`` and
+  ``'3-5-6-10-11-12-15-18-19'`` becomes ``'3-5+6-10+12-15-18+19'``
+- HIRES wavelength solution improvements galor
+- Added `redo_slits` option
+- Refactored ``load_line_lists()`` yet again!
 
 
 1.13.0 (2 June 2023)

doc/calibrations/calibrations.rst

@@ -175,15 +175,22 @@ All reduced calibration frames are named according to their primary calibration
 type (e.g., ``Arc``).  They are also assigned a unique identifier that is a
 combination of:
 
-    - the instrument configuration (setup) identifier (e.g., ``A``),
+    #. the instrument configuration (setup) identifier (e.g., ``A``),
 
-    - the list of associated calibration groups (e.g., ``1-2`` or ``all``), and
+    #. a compressed list of associated calibration groups (e.g., ``1+2`` or ``all``), and
 
-    - the detector or mosaic identifier (e.g., ``DET01`` or ``MSC01``).
+    #. the detector or mosaic identifier (e.g., ``DET01`` or ``MSC01``).
 
-.. note::
+For the second component, sequential numbers are reduced to a range; e.g.,
+``'0-1-2-3-4'`` becomes ``'0+4'`` and ``'3-5-6-10-11-12-15-18-19'`` becomes
+``'3-5+6-10+12-15-18+19'``.
+
+.. warning::
 
     If you have a lot of calibration groups in your pypeit file, you may end up
-    with very long file names!
+    with very long file names!  This may cause a fault when the file name is
+    included in the header of the output fits files.  If using the calibration
+    group ``all`` doesn't solve the problem or isn't possible given your
+    application, please `Submit an issue`_.
 
 

doc/calibrations/wave_calib.rst

@@ -259,6 +259,57 @@ observations, long-slit observations where wavelengths
 vary (*e.g.*, grating tilts).  We are likely to implement
 this for echelle observations (*e.g.*, HIRES).
 
+.. _wvcalib-echelle:
+
+Echelle Spectrographs
+=====================
+
+Echelle spectrographs are a special case for wavelength
+solutions, primarily because the orders follow the
+grating equation.
+
+In general, the approach is:
+
+    1. Identify the arc lines in each order
+    2. Fit the arc lines in each order to a polynomial, individually
+    3. Fit a 2D solution to the lines using the order number as a basis
+    4. Reject orders where the RMS of the fit (measured in binned pixels) exceeds ``rms_threshold``
+    5. Attempt to recover the missing orders using the 2D fit and 
+      a higher RMS threshold
+    6. Refit the 2D solution
+
+One should always inspect the outputs, especially the 2D solution
+(global and orders).  One may then need to modify the ``rms_threshold``
+parameter and/or hand-fit a few of the orders to improve the solution.
+
+.. _wvcalib-rms-threshold:
+
+rms_threshold
+-------------
+
+All of the echelle spectrographs have a default ``rms_threshold`` 
+matched to a default ``FWHM`` parameter (also measured in binned pixels).
+The ``rms_threshold`` adopted in the analysis is one 
+scaled by the measured FWHM from the arc lines 
+(again, binned pixels) of the adopted calibration files
+
+That is, each order must satisfy the following:
+
+.. code-block:: ini
+
+    RMS < rms_threshold * (measured_FWHM/default_FWHM)
+
+Note: in a future release, we will re-define ``rms_threshold`` to be
+in units of the measured FWHM.
+
+Mosaics
+-------
+
+For echelle spectrographs with multiple detectors *per* camera
+that are mosaiced (e.g. Keck/HIRES), we fit the 2D solutions on a *per* detector
+basis.  Ths is because we have found the mosaic solutions to be
+too difficult to make sufficiently accurate.
+
 .. _wvcalib-byhand:
 
 By-Hand Approach

doc/cookbook.rst

@@ -13,6 +13,10 @@ should adhere to the following approach.
 
 Note that:
 
+  - Essentially all PypeIt reduction steps are done via executing command-line
+    scripts using a terminal window.  We provide specific commands below and
+    throughout our documentation.
+
   - This cookbook provides minimal detail, but serves as a basic introduction to
     the primary steps used to reduce your data with PypeIt.  We guide you to
     other parts of our documentation that explain specific functionality in more
@@ -27,8 +31,9 @@ Note that:
   - Specific advice on :doc:`spectrographs/spectrographs` is provided in their own doc page
     (although not every supported spectrograph has stand-alone documentation).
 
-  - Invariably something will be out of date.  When you see an egregious
-    example, please holler on GitHub or Slack.
+  - Invariably something will be out of date in our doc pages.  When you see an
+    egregious example, please holler on `GitHub
+    <https://github.com/pypeit/PypeIt>`__ or `Slack <pypeit-users.slack.com>`__.
 
 Finally, note that before you keep going, you should have already done the following:
 

doc/include/shane_kast_blue_A.pypeit.rst

@@ -33,7 +33,6 @@
     b11.fits.gz | pixelflat,illumflat,trace |            144.955 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 |  57162.07897476852 |            1.0 |    15.0 |      d55 |     0
     b12.fits.gz | pixelflat,illumflat,trace |  145.0908333333333 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 | 57162.079351388886 |            1.0 |    15.0 |      d55 |     0
     b13.fits.gz | pixelflat,illumflat,trace | 145.22791666666666 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 | 57162.079728240744 |            1.0 |    15.0 |      d55 |     0
-     b2.fits.gz | pixelflat,illumflat,trace | 143.36208333333335 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 |  57162.07473645834 |            1.0 |    30.0 |      d55 |     0
      b3.fits.gz | pixelflat,illumflat,trace | 143.86791666666667 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 |  57162.07596400463 |            1.0 |    15.0 |      d55 |     0
      b4.fits.gz | pixelflat,illumflat,trace | 144.00458333333333 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 | 57162.076341782406 |            1.0 |    15.0 |      d55 |     0
      b5.fits.gz | pixelflat,illumflat,trace | 144.14041666666665 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 |  57162.07671956019 |            1.0 |    15.0 |      d55 |     0

doc/scripts/make_example_files.py

@@ -75,10 +75,11 @@ def make_example_gnirs_pypeit_files(version, date):
     oroot = Path(resource_filename('pypeit', '')).resolve().parent / 'doc' / 'include'
 
     # Create the default pypeit file
-    droot = Path(os.getenv('PYPEIT_DEV')).resolve() / 'RAW_DATA' / 'gemini_gnirs_echelle' / '32_SB_SXD'
+    droot = Path(os.getenv('PYPEIT_DEV')).resolve() / 'RAW_DATA' / 'gemini_gnirs_echelle' \
+                / '32_SB_SXD'
 
-    pargs = setup.Setup.parse_args(['-r', str(droot), '-s', 'gemini_gnirs_echelle', '-b', '-c', 'A',
-                                    '-d', str(oroot),
+    pargs = setup.Setup.parse_args(['-r', str(droot), '-s', 'gemini_gnirs_echelle', '-b',
+                                    '-c', 'A', '-d', str(oroot),
                                     '--version_override', version, 
                                     '--date_override', date])
     setup.Setup.main(pargs)

doc/spectrographs/keck_hires.rst

@@ -0,0 +1,18 @@
+==========
+Keck HIRES
+==========
+
+Overview
+========
+
+This file summarizes several instrument specific settings that are related to the Keck/HIRES spectrograph.
+
+
+Wavelengths
+===========
+
+See :ref:`wvcalib-echelle` for details on the wavelength calibration.
+
+We also note that several Orders from 40-45 are 
+frequently flagged as bad in the wavelength solution.  
+This is due, in part, to very bright ThAr line contamination. 

doc/spectrographs/xshooter.rst

@@ -10,3 +10,51 @@ This file summarizes several instrument specific
 settings that are related to the VLT/XShooter spectrograph.
 
 
+Wavelengths
+===========
+
+As it is common for ESO to obtain calibrations with different
+slit widths and binning, this can lead to various challenges
+for PypeIt.
+
+As regards wavelengths, the varying binning and slit widths lead
+to differing FWHM of the arc lines.  And because the RMS threshold
+for a good solution is scaled to FWHM, the default is to measure
+the FWHM from the lines themselves.
+
+If too many orders are being rejected, you may wish to adjust things
+in one or more ways.
+
+FWHM
+----
+
+For the UVB or the VIS, you may turn off measuring the FWHM (in units
+of binned pixdels) from the arc lines
+by adding this to your :doc:`pypeit_file`:
+
+
+.. code-block:: ini
+
+    [calibrations]
+        [[wavelengths]]
+            fwhm_fromlines = False
+
+This will set the FWHM to the default value for UVB/VIS which 
+may yield a better set of discovered arc lines.
+
+RMS
+---
+
+Another option is to increase the RMS threshold for a good solution.
+This may be done in the :doc:`pypeit_file` as well:
+
+.. code-block:: ini
+
+    [calibrations]
+        [[wavelengths]]
+            rms_threshold = 1.5
+
+
+Note that this is scaled by the ratio of the measured FWHM value
+to the default value.  See :ref:`_wvcalib-echelle` for
+further details.