Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HIRES wavelengths and a bit more #1628

Merged
merged 43 commits into from
Aug 13, 2023
Merged

HIRES wavelengths and a bit more #1628

merged 43 commits into from
Aug 13, 2023

Conversation

profxj
Copy link
Collaborator

@profxj profxj commented Jul 10, 2023

This significantly advances the HIRES wavelength solutions
and deals with rms_threshold for other echelle spectrographs.

I am directing this PR to the open sensfunc branch but will
redirect to develop once that is merged. It is ready now,
however, for a review.

@codecov-commenter
Copy link

codecov-commenter commented Jul 10, 2023

Codecov Report

Merging #1628 (f3fcde3) into develop (5980c0d) will increase coverage by 0.01%.
The diff coverage is 14.35%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@             Coverage Diff             @@
##           develop    #1628      +/-   ##
===========================================
+ Coverage    41.01%   41.02%   +0.01%     
===========================================
  Files          189      189              
  Lines        43076    43185     +109     
===========================================
+ Hits         17666    17718      +52     
- Misses       25410    25467      +57     
Files Changed Coverage Δ
pypeit/core/arc.py 43.56% <ø> (-0.28%) ⬇️
pypeit/core/gui/identify.py 8.62% <0.00%> (+0.03%) ⬆️
pypeit/core/wavecal/echelle.py 20.63% <0.00%> (+0.32%) ⬆️
pypeit/spectrographs/keck_esi.py 23.95% <0.00%> (-0.30%) ⬇️
pypeit/spectrographs/keck_nires.py 25.70% <0.00%> (-0.13%) ⬇️
pypeit/spectrographs/magellan_mage.py 31.14% <0.00%> (-0.52%) ⬇️
pypeit/spectrographs/p200_tspec.py 30.08% <0.00%> (-0.25%) ⬇️
pypeit/spectrographs/vlt_xshooter.py 24.66% <0.00%> (-0.20%) ⬇️
pypeit/core/wavecal/autoid.py 26.36% <8.06%> (-0.35%) ⬇️
pypeit/wavecalib.py 55.21% <9.09%> (-7.79%) ⬇️
... and 5 more

... and 1 file with indirect coverage changes

pypeit/par/pypeitpar.py Outdated Show resolved Hide resolved
self.wv_calib = self.build_wv_calib(
self.arccen, self.par['method'],
skip_QA=skip_QA,
prev_wvcalib=prev_wvcalib)

# Fit 2D?
if self.par['echelle']:
# Assess the fits
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this control flow be moved to self.echelle_2dfit? It seems out of place in the run method. Also, please think carefully about whether these operations can be broken off into core methods that are functions or self-contained simple classes. I realize that all the modular methods are interacting for echelle wavelength calibration, but please also understand that you are carrying out core functionality in a very complicated class, whereas the model is that base classes like wavecalib are just supposed to wrap core methods.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this was too much for run()

But it doesn't fit into echelle_2dfit() which does at it says.
The code here is to redo individual echelle orders. So, I've
moved that code out of run() and into an apt named method
of BuildWaveCalib.

It mainly is control flow + calls to core methods, although still
not elegant..

Someday we need to do a major refactor of wavecalib.py
(together with autoid.py). Will be a painful day, but definitely
needed..

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay for now maybe just add a TODO that this control flow needs to be consolidated in the future.

Copy link
Collaborator

@jhennawi jhennawi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @profxj. Minor changes requested.

@profxj
Copy link
Collaborator Author

profxj commented Aug 4, 2023

Failing a few tests now (mainly related to the removal of b2.fits)

Test Summary

--- PYTEST PYPEIT UNIT TESTS PASSED 232 passed, 73 warnings in 361.67s (0:06:01) ---
--- PYTEST UNIT TESTS FAILED 5 failed, 118 passed, 192 warnings in 811.70s (0:13:31) ---
--- PYTEST VET TESTS FAILED 1 failed, 52 passed, 43 warnings in 830.56s (0:13:50) ---
--- PYPEIT DEVELOPMENT SUITE FAILED 1/210 TESTS ---
Failed tests:
shane_kast_red/600_7500_d55_ret pypeit_sensfunc
Skipped tests:
shane_kast_red/600_7500_d55_ret pypeit_flux

Copy link
Collaborator

@rcooke-ast rcooke-ast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only minor comments from me - all looks good!!

doc/spectrographs/keck_hires.rst Outdated Show resolved Hide resolved
pypeit/core/wavecal/patterns.py Show resolved Hide resolved
pypeit/core/wavecal/waveio.py Outdated Show resolved Hide resolved
pypeit/par/pypeitpar.py Outdated Show resolved Hide resolved
pypeit/core/wavecal/waveio.py Show resolved Hide resolved
pypeit/wavecalib.py Outdated Show resolved Hide resolved
pypeit/wavecalib.py Outdated Show resolved Hide resolved
pypeit/core/wavecal/autoid.py Show resolved Hide resolved
pypeit/core/wavecal/autoid.py Outdated Show resolved Hide resolved
pypeit/core/wavecal/autoid.py Show resolved Hide resolved
@profxj
Copy link
Collaborator Author

profxj commented Aug 7, 2023

Almost there..

Test Summary

--- PYTEST PYPEIT UNIT TESTS PASSED 232 passed, 73 warnings in 374.94s (0:06:14) ---
--- PYTEST UNIT TESTS PASSED 123 passed, 194 warnings in 937.02s (0:15:37) ---
--- PYTEST VET TESTS PASSED 53 passed, 53 warnings in 905.03s (0:15:05) ---
--- PYPEIT DEVELOPMENT SUITE FAILED 2/216 TESTS ---
Failed tests:
gemini_gmos/GN_HAM_R400_885 pypeit
keck_hires/HS1700+6416_H45aH_RED_B2_ECH_0.00_XD_-0.00_1x2 pypeit

That is, each order must satisfy the following:

.. code-block:: ini

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion is that we instead define:

rms_pixels = rms_threshold*FWHM

Where FWHM is either the default FWHM or the measured FWHM, i.e. it will be the latter when the automatic FWHM determination is turned on.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is what we are trying to handle:

  1. For a given spectrograph, we have a default rms_threshold (pixels) corresponding to a default fwhm (pixels) which we tune from the DevSuite. Both of these depend on slit width and binning
  2. Now, consider a new exposure with different slit width and/or binning. The sensible thing is to scale rms_threshold by the measured fwhm. This works relatively well but is not perfect. Nothing ever is for wavelengths.. This is what is currently implemented in the code and described in the docs.

The fwhm_fromlines parameter only refers to whether to use the measured_fwhm when finding arclines.
It doesn't play a role rms_threshold.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an issue to updated the docs and spectrograph files to the convention that we discussed, i.e using rms_threshold_frac_fwhm. That need not be done for this PR, but clearly should be done asap.

Copy link
Collaborator

@jhennawi jhennawi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with everything else but still stuck on rms_threshold. Please see my suggestion there.

@profxj
Copy link
Collaborator Author

profxj commented Aug 9, 2023

Test Summary

--- PYTEST PYPEIT UNIT TESTS PASSED 232 passed, 73 warnings in 378.98s (0:06:18) ---
--- PYTEST UNIT TESTS PASSED 123 passed, 194 warnings in 938.59s (0:15:38) ---
--- PYTEST VET TESTS PASSED 53 passed, 54 warnings in 916.95s (0:15:16) ---
--- PYPEIT DEVELOPMENT SUITE PASSED 215/215 TESTS ---
Testing Started at 2023-08-07T13:41:09.322477
Testing Completed at 2023-08-07T22:22:31.269665
Total Time: 8:41:21.947188

Copy link
Collaborator

@jhennawi jhennawi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending issue being added.

@profxj
Copy link
Collaborator Author

profxj commented Aug 11, 2023

Relaunching tests, hopefully for the last time

@profxj profxj mentioned this pull request Aug 11, 2023
@profxj
Copy link
Collaborator Author

profxj commented Aug 13, 2023

Test Summary

--- PYTEST PYPEIT UNIT TESTS PASSED 232 passed, 73 warnings in 521.81s (0:08:41) ---
--- PYTEST UNIT TESTS PASSED 123 passed, 193 warnings in 925.67s (0:15:25) ---
--- PYTEST VET TESTS PASSED 53 passed, 53 warnings in 986.12s (0:16:26) ---
--- PYPEIT DEVELOPMENT SUITE PASSED 215/215 TESTS ---
Testing Started at 2023-08-12T13:56:35.018313
Testing Completed at 2023-08-13T04:58:40.724097
Total Time: 15:02:05.705784

@profxj profxj merged commit 4d8cd81 into develop Aug 13, 2023
25 checks passed
@profxj profxj deleted the hires_waves_and_refactor branch August 13, 2023 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants