HIRES wavelengths and a bit more

CHANGES.rst

@@ -33,6 +33,9 @@
     - Now ``only_slits`` parameter in `pypeit_coadd_2dspec` includes the detector number (similar to ``slitspatnum``)
     - Added ``exclude_slits`` parameter in `pypeit_coadd_2dspec` to exclude specific slits
     - Fix wrong RA & Dec for 2D coadded serendips
+- HIRES wavelength solution improvements galor
+- Added `redo_slits` option
+- Refactored ``load_line_lists()`` yet again!
 
 
 1.13.0 (2 June 2023)

doc/calibrations/wave_calib.rst

@@ -259,6 +259,57 @@ observations, long-slit observations where wavelengths
 vary (*e.g.*, grating tilts).  We are likely to implement
 this for echelle observations (*e.g.*, HIRES).
 
+.. _wvcalib-echelle:
+
+Echelle Spectrographs
+=====================
+
+Echelle spectrographs are a special case for wavelength
+solutions, primarily because the orders follow the
+grating equation.
+
+In general, the approach is:
+
+    1. Identify the arc lines in each order
+    2. Fit the arc lines in each order to a polynomial, individually
+    3. Fit a 2D solution to the lines using the order number as a basis
+    4. Reject orders where the RMS of the fit (measured in binned pixels) exceeds ``rms_threshold``
+    5. Attempt to recover the missing orders using the 2D fit and 
+      a higher RMS threshold
+    6. Refit the 2D solution
+
+One should always inspect the outputs, especially the 2D solution
+(global and orders).  One may then need to modify the ``rms_threshold``
+parameter and/or hand-fit a few of the orders to improve the solution.
+
+.. _wvcalib-rms-threshold:
+
+rms_threshold
+-------------
+
+All of the echelle spectrographs have a default ``rms_threshold`` 
+matched to a default ``FWHM`` parameter (also measured in binned pixels).
+The ``rms_threshold`` adopted in the analysis is one 
+scaled by the measured FWHM from the arc lines 
+(again, binned pixels) of the adopted calibration files
+
+That is, each order must satisfy the following:
+
+.. code-block:: ini
+
+    RMS < rms_threshold * (measured_FWHM/default_FWHM)
+
+Note: in a future release, we will re-define ``rms_threshold`` to be
+in units of the measured FWHM.
+
+Mosaics
+-------
+
+For echelle spectrographs with multiple detectors *per* camera
+that are mosaiced (e.g. Keck/HIRES), we fit the 2D solutions on a *per* detector
+basis.  Ths is because we have found the mosaic solutions to be
+too difficult to make sufficiently accurate.
+
 .. _wvcalib-byhand:
 
 By-Hand Approach

doc/spectrographs/keck_hires.rst

@@ -0,0 +1,18 @@
+==========
+Keck HIRES
+==========
+
+Overview
+========
+
+This file summarizes several instrument specific settings that are related to the Keck/HIRES spectrograph.
+
+
+Wavelengths
+===========
+
+See :ref:`wvcalib-echelle` for details on the wavelength calibration.
+
+We also note that several Orders from 40-45 are 
+frequently flagged as bad in the wavelength solution.  
+This is due, in part, to very bright ThAr line contamination. 

doc/spectrographs/xshooter.rst

@@ -10,3 +10,51 @@ This file summarizes several instrument specific
 settings that are related to the VLT/XShooter spectrograph.
 
 
+Wavelengths
+===========
+
+As it is common for ESO to obtain calibrations with different
+slit widths and binning, this can lead to various challenges
+for PypeIt.
+
+As regards wavelengths, the varying binning and slit widths lead
+to differing FWHM of the arc lines.  And because the RMS threshold
+for a good solution is scaled to FWHM, the default is to measure
+the FWHM from the lines themselves.
+
+If too many orders are being rejected, you may wish to adjust things
+in one or more ways.
+
+FWHM
+----
+
+For the UVB or the VIS, you may turn off measuring the FWHM (in units
+of binned pixdels) from the arc lines
+by adding this to your :doc:`pypeit_file`:
+
+
+.. code-block:: ini
+
+    [calibrations]
+        [[wavelengths]]
+            fwhm_fromlines = False
+
+This will set the FWHM to the default value for UVB/VIS which 
+may yield a better set of discovered arc lines.
+
+RMS
+---
+
+Another option is to increase the RMS threshold for a good solution.
+This may be done in the :doc:`pypeit_file` as well:
+
+.. code-block:: ini
+
+    [calibrations]
+        [[wavelengths]]
+            rms_threshold = 1.5
+
+
+Note that this is scaled by the ratio of the measured FWHM value
+to the default value.  See :ref:`_wvcalib-echelle` for
+further details.

pypeit/calibrations.py

@@ -837,7 +837,7 @@ def get_wv_calib(self):
             self.wv_calib.chk_synced(self.slits)
             self.slits.mask_wvcalib(self.wv_calib)
             # Return
-            if self.par['wavelengths']['redo_slit'] is None:
+            if self.par['wavelengths']['redo_slits'] is None:
                 return self.wv_calib
 
         # Determine lamp list to use for wavecalib
@@ -863,7 +863,9 @@ def get_wv_calib(self):
         self.wv_calib = waveCalib.run(skip_QA=(not self.write_qa),
                                       prev_wvcalib=self.wv_calib)
         # If orders were found, save slits to disk
-        if self.spectrograph.pypeline == 'Echelle' and not self.spectrograph.ech_fixed_format:
+        #   or if redo_slits
+        if (self.par['wavelengths']['redo_slits'] is not None) or (
+            self.spectrograph.pypeline == 'Echelle' and not self.spectrograph.ech_fixed_format):
             self.slits.to_file()
         # Save calibration frame
         self.wv_calib.to_file()

pypeit/core/arc.py

@@ -18,8 +18,6 @@
 
 from pypeit import msgs
 from pypeit import utils
-from pypeit.core.wavecal import wvutils
-from pypeit.core.wavecal import wv_fitting
 from pypeit.core import fitting
 from IPython import embed
 

pypeit/core/gui/identify.py

@@ -275,11 +275,7 @@ def initialise(cls, arccen, lamps, slits, slit=0, par=None, wv_calib_all=None,
         detns = tdetns[icut]
 
         # Load line lists
-        if 'ThAr' in lamps:
-            line_lists_all = waveio.load_line_lists(lamps)
-            line_lists = line_lists_all[np.where(line_lists_all['ion'] != 'UNKNWN')]
-        else:
-            line_lists = waveio.load_line_lists(lamps)
+        line_lists, _, _ = waveio.load_line_lists(lamps, include_unknown=False)
 
         # Trim the wavelength scale if requested
         if wavelim is not None: