forked from pypeit/PypeIt
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGES.rst
1617 lines (1506 loc) · 72.9 KB
/
CHANGES.rst
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
****************************************************************************
USE OF THIS FILE IS NOW DEPRECATED. CHANGES SHOULD BE ADDED TO THE RELEVANT
RELEASE FILE IN doc/releases
****************************************************************************
1.14.0 (18 Sep 2023)
--------------------
- Add support for Gemini/GNIRS (IFU)
- Add support for Keck/KCRM
- Added a script to convert a wavelength solution into something that can be placed in the reid archive.
- Hotfix for GTC/OSIRIS lamp list
- Hotfix for Arc1D stats annotations on the QA
- Hotfix for metadata (correctly set config_independent frames when multiple configurations are being setup)
- Hotfix for metadata (support lists in ``config_independent_frames()``)
- Hotfix for rebin (speed-up and conserves flux)
- Hotfix for skysub regions GUI that used np.bool
- Hotfix to stop pypeit_setup from crashing on data from lbt_luci1, lbt_luci2, magellan_fire,
magellan_fire_long, p200_tspec, or vlt_sinfoni.
- Hotfix to set BPM for each type of calibration file.
- Adds Keck/ESI to PypeIt
- Instrumental FWHM map is calculated and output in ``Calibrations`` and ``spec1d`` files.
- Adds Keck/ESI to PypeIt
- Add MDM/Modspec spectrograph
- Store user-generated wavelength solution in pypeit cache
- Improvements to wavelength grids and masking in coadd routines.
- Fixed a bug in echelle coadding where the wrong coadded spectra were being
used in final stacks.
- Sensitivity function models can now be computed relative to the blaze
spectrum.
- Refactored coadding routines to work with lists to support coadding data from
different setups.
- Changes to how masking is dealt with in extraction to fix a bug in how masks
were being treated for echelle data
- Various fixes and changes required to add more support for Keck/HIRES and JWST
- Fix a bug in ``spectrograph.select_detectors``, where a list of ``slitspatnum`` could not be used.
- Improvements in 2D coaddition
- Fix a bug in `pypeit_setup_coadd2d` for the output file name of the .coadd2d file
- Added possibility to specify more than one Science folder in `pypeit_setup_coadd2d`
- Now ``only_slits`` parameter in `pypeit_coadd_2dspec` includes the detector number (similar to ``slitspatnum``)
- Added ``exclude_slits`` parameter in `pypeit_coadd_2dspec` to exclude specific slits
- Fix wrong RA & Dec for 2D coadded serendips
- Changed calibration frame naming as an attempt to avoid very long names for
files with many calibration groups. Sequential numbers are reduced to a
range; e.g., ``'0-1-2-3-4'`` becomes ``'0+4'`` and
``'3-5-6-10-11-12-15-18-19'`` becomes ``'3-5+6-10+12-15-18+19'``
- HIRES wavelength solution improvements galor
- Added `redo_slits` option
- Refactored ``load_line_lists()`` yet again!
- Improvements for keck/LRIS
- Generated wavelength templates for all the LRIS grism & grating
- Added FeAr line list
- Improved calibration association and frame typing
- Improved and added documentation
- Changes to ``metadata.py`` including commenting out, in the pypeit file,
files that have frametype None (this prevent ``run_pypeit`` to crash)
- Added a function ``check_spectrograph()`` (currently only defined for LRIS),
that checks (during ``pypeit_setup``) if the selected spectrograph is the
corrected one for the data used.
1.13.0 (2 June 2023)
--------------------
- Implemented a resample algorithm when generating datacubes
- Hotfix to docs to ensure pypeit_loaders api doc is generated
- Allow user control of the local sky subtraction window
- Deprecate use of python 3.8 with PypeIt, allow python 3.11
- Make pypeit_show_2dspec (somewhat) backwards compatible.
- Hotfix for KCWI when using alignment (aka ContBars) frames for the astrometric correction.
- Sensitivity function masking and output updates
- Fixed a bug in the `variance_model` calculation for combined images.
- Added the possibility to use dither offsets saved in the header of the science frames for
coadding 2D spectra (``dithoff`` must be part of the spectrograph metadata).
- Calibration group numbers can now be anything, as long as there are no more
than 63 unique integers.
- Removed use of the term "master", renamed to calibration frames/files.
Default output directory for calibration frames is now ``Calibrations``.
Calibration frames renamed; e.g., ``MasterArc`` is now ``Arc``.
- Calibration frame naming now done via ``calibframe.CalibFrame`` class.
- Start to deprecate use of ``os.path`` in favor of ``pathlib``
- Deprecated ``pypeit_parse_calib_id`` script, but improved the ``.calib`` file
provided by ``pypeit_setup``. The ``.calib`` file is now always written, and
provides a direct association between input raw files and output calibration
files. Discussed in new docs.
- The ``'calib'`` column is now always added to the pypeit file, regardless of
whether or not you also request the ``'comb_id'`` and ``'bkg_id'`` columns.
- Names of associated calibration frames now written to ``spec2d`` file headers.
- Added the option to disable strict version checking for 1d coadds.
- Major quicklook updates. ql_multislit.py temporarily deprecated.
- Improve speed in ginga visualization of traces and added
`pypeit_chk_tilts`. Note that this script uses an update
of the tilts datamodel, so it will not work on older reductions.
- Updates to reduction parameters for LDT/DeVeny
1.12.2 (29 Mar 2023)
--------------------
- Gemini/GMOS mask design slurping and usage
- New GMOS wavelength solution
- Added NIRES tutorial doc
- reid_arxiv templates for all MMTO Blue Channel gratings and for MMTO Binospec G600 and G1000
- Various bug fixes and enhancements to mmt_bluechannel and mmt_binospec support
- Include the S/N of extracted spectra in the SpecObj datamodel
- Added new specutils interface
- Fixed bugs when only performing calibrations and (1) calib groups are all set
to 'all' or (2) anything other than '0'.
- Added `MASKDEF_OBJMAG` and `MASKDEF_OBJMAG_BAND` in spec1d datamodel.
- Improved NIRES dither pattern parsing and automatic assignment of `comb_id` and `bkg_id`.
1.12.1 (21 Feb 2023)
--------------------
- (Hotfix) Specify sphinx versions to correctly work with
sphinx_rtd_theme
- (Hotfix) Fixed bug that caused crash of sensfunc routines using
telluric grids in offline processing
- (Hotfix) Fixed error when showing flats in Ginga when the fine correction is not performed
- Implemented the upgraded GTC/OSIRIS+
- (Hotfix) keymap error when displaying GUIs
- Added support for more NOT/ALFOSC grisms as well as NOT recommended standards
- Implemented the SOAR/Goodman blue 400 grating (setup M1)
- Added support for SOAR/Goodman red 600 grating (setup RED)
- Implemented the SOAR/Goodman (blue) M1 only
- New docs on OneSpec
- Modify install notes to allow python 3.10; python3.8 no longer explicitly supported
- Allow for bad orders during extraction (without crashing)
1.12.0 (31 Jan 2023)
--------------------
- (Hotfix) Fixed bug that allowed science frames to be assigned to multiple
instrument configurations
- (Hotfix) Fixed typo related to GitHub download for offline processing
- Started modifications and support for JWST.
- Limit LRISr header crashes
- Added spectral flexure and reference frame corrections for IFU data
- Allow separate sky frame to be used for sky subtraction with IFU data
- Limit the images written to the MasterEdges file to only the trace
image, mask, and detector.
- Refactor quicklook scripts
- OpenMP link fix
- Enable boxcar_radius for manual extraction
- Prevent flexure crash
- Fixed error with deprecated numpy types
- Improved optimization of bspline c code
- Parse Keck/NIRES dither patterns, similar to MOSFIRE
- Introduce BitMaskArray class to ease use of bitmasks
- Fixed memory hogging by matplotlib when using version >= 3.6.1
1.11.0 (21 Oct 2022)
--------------------
- Add ability for users to specify custom arc line lists for
wavelength calibration, saved in the user's PypeIt cache
- Added Keck/NIRES frame-typing development doc.
- Now more than one setup can be assigned to the same calibration frame,
allowing to associate the same calibration frames to different science/standard
frames, if desired.
- Correctly associate calibrations with science data for MOSFIRE longslit and long2pos masks.
- Automatically assign `comb_id` and `bkg_id` to MOSFIRE science data,
using the information on the dither pattern.
- Allow verbosity specification for various post-processing command-line scripts.
- Allow for the specification of a specific UVIS extinction file for sensitivity
function computation and flux calibration.
- Adding Keck/HIRES functionality.
- Restructred coadd2d in order to work with images that have different
sizes.
- Restructured extraction and find_objects classes to work
better with 2d coadds.
- Refactor and general update of documentation
1.10.0 (11 July 2022)
---------------------
- Modify tweak_standard for Mosfire/J2
- Apply find_min_max when clipping the image for object finding
- Mask bad detector regions for global sky flexure calculation
- Detector structure correction included in flatfield calibration
- Apply find_min_max when clipping the image for object finding
- Mask bad detector regions for global sky flexure calculation
- Fixed a bug associated with 2d interpolation of waveimg in extraction.
- Refactor PypeIt input files
- Added wavelength diagnostics to the spec2d output
1.9.1 (13 June 2022)
--------------------
- Hotfix for bug related to downloading from the `reid_arxiv` when using
the `reidentify` wavelength calibration method.
1.9.0 (31 May 2022)
-------------------
- When using glob to get files in pypeit_setup, added automatic sorting
so that the default `comb_id` ordering matches the sorted file name.
- Improve Keck/KCWI automatic frame typing.
- Implemented Keck/KCWI flux calibration
- Wavelength templates (OH lines and arc lamps) created for Keck/MOSFIRE
- Mosaic is now available for Keck/DEIMOS too.
- Various package data (e.g., reid_arxiv, sensfunc) are no longer
distributed via PyPI to reduce package size; introduce mechanisms for
downloading/caching needed data either at runtime or on demand.
- Save output wavelength calibration from `pypeit_identify` to the cache
for direct reuse in data reduction.
- The `pypeit_identify` GUI can now toggle between linear and log
scaling of the arc spectrum flux.
- Improved wavelength solution for Gemini-Nort E2V detector
- Keck/DEIMOS now uses gain/RN values measured periodically by WMKO
- Add bok_bc 300 grating template
- Added more flexible quicklook that can handle dithering.
- Expose exposure time scaling for dark frames as an image processing
parameter, and set the default behavior to ignore any difference in
exposure time. Also fixes a bug in the variance calculation.
- Refactored object finding
- Bug fixes in local sky subtraction and extraction
- Fixed pypeit setup issues due to bad LRIS headers.
- Added support for VLT FORS2 600z grism.
- Added enhancements and fixes for Keck lris red Mark4.
- Fixed a bug in 2d coadding when objects were not being identified.
Refactored 2d extraction.
- Added code to better parse Gemini/GNIRS dither sequences
- Add spectrograph child for VLT X-SHOOTER UVB arm
- Minor enhancements to `pypeit_identify` GUI
- Refactoring of `pypeit_show_wvcalib` GUI
1.8.1 (23 Feb 2022)
-------------------
- various hotfixes
- Include preliminary support for fluxing with archived SensFunc files
for DEIMOS.
1.8.0 (12 Feb 2022)
-------------------
- Fixed a bug about how `maskdef_offset` is assigned to each detector
- Changed default behavior for how PypeIt computes `maskdef_offset` for
DEIMOS. It now uses by default the stars in the alignment boxes.
- Introduces pypeit_parse_calib_id script
- Refactor manual extraction
- Fixed 2Dcoadd spec bugs for central wavelength dithers.
- GMOS doc updates
- Add 2D wavelength calibration image to MasterFlat output; include
wavelength calibration in pypeit_chk_flat ginga display.
- Introduce mosaicing
- `det` arguments can now be tuples with a list of detectors to
combine into a mosaic. Mosaics can now be defined in the pypeit
file using `detnum`; e.g., `detnum=(1,2)` creates a mosaic of
detectors 1 and 2.
- The tuples must be one among an allowed set defined by each
spectrograph class; see `gemini_gmos.py`.
- `DETECTOR` extensions in output files can now be either a
`DetectorContainer` object or a `Mosaic` object. Both are now
written using `astropy.table.Table` instances. `Mosaic` objects
just have more columns.
- The `otype` of `DataContainer` data-model components can now be a
tuple of `DataContainer` subclasses indicating that the component
has an optional type.
- Added the `one_row_table` class attribute to `DataContainer`,
which will try to force all the elements of a datamodel into a
binary table extension with a single row.
- Started propagation of name changes from, e.g., `DET01` to
`MSC01`, where the latter indicates the reduction uses the first
mosaic option for the spectrograph. Keys for master calibration
frames are now, e.g., `A_1_DET01` instead of `A_1_01`.
- Currently only implemented for `gemini_gmos`.
- During processing, bias and dark images are left as separate
detector images, whereas all other images are mosaiced for further
processing. This means that `RawImage` is now *always* 3D, where
`PypeItImage` can be either 2D or 3D.
- Added a `det_img` to `PypeItImage` datamodel to keep track of the
parent detector for each pixel in a mosaic.
- Added a `amp_img` to `PypeItImage` datamodel to keep track of the
parent amplifier for each pixel in a mosaic; this is the result of
mosaicing the `datasec_img` objects for each detector.
- Improve performance of L.A.Cosmic algorithm:
- Switch to using ndimage.binary_dilation for growing masked regions
- Switch to astropy convolution for Laplace convolution
- Added faster block replication algorithm
- Fix iteration logic
- Intermediate update to BPM. Preference given to pulling this from the
relevant `PypeItImage` calibration image instead of always building it
from scratch. That latter complicated things for mosaics.
- First steps toward more robust treatment of saturation.
- Dark counts used for calculating the shot noise now includes measured
dark images if provided
- `PypeIt` file parameters can now parse sets of tuples; e.g.,
`detnum=(1,2),(3,4)` should get parsed as `par['detnum'] = [(1,2),
(3,4)]`.
- `PypeIt.select_detectors` has been moved to `Spectrograph`.
- Update for `LDT/DeVeny` including support for binned data,
`use_header` for reading arc lamps used from frames, and `reid_arxiv`
templates for three additional gratings.
- Slurps in and uses slitmask design for Keck/LRIS (limited usage)
- Hotfix for `gemini_gmos` mosaic tracing parameters
- Include sky model in 2nd pass of global sky subtraction (not for IR
redux).
- Skymask is now computed also for the maskdef_extract objects.
- Added dedicated fwhm and boxcar_radius for maskdef_extract objects.
- Added pypeit_version to the pypeit file header.
- Set DEIMOS `find_fwhm` default to 0.8" in binned pixels.
- Added row-dependent pattern-noise calculation
- Improvements in `pypeit_coadd_2dspec`:
- `maskdef_id` assigned to each slit
- Assigning object's name, ra and dec to detected objects is now
available
- Force extract of undetected objects is now available
- `maskdef_offset` can be use as offsets in the coadd
- Coadding only a specific sets of slits is now possible with the
parset `--only_slits`
- If the user inputs a list of offsets, the weights can still be
computed if a bright object is found, otherwise uniform weigths
will be used
- Fixed manual extraction bug
- Various improvements in the flow of the code
- spec1d*.txt is now produced also for coadd2d
- Scripts to explore the noise residuals in PypeIt
- Added Coadd2D HOWTO docs
- Fixes a bug in echelle object finding
- Attempt to make the threshold computation for object finding more robust.
- Fixed a bug in extraction for echelle spectrographs for IR reductions.
- Tuned up preivious refactor of object finding and extraction classes.
- Fixed a bug that was introduced in skymask definition.
- Fixed a bug where negative objects were not being found for IR reductions of standard stars.
- Add template wavelength solution for soar_goodman_red 400_SYZY
1.7.0 (19 Nov 2021)
-------------------
- Introduces pypeit_parse_calib_id script
- Throw a warning if the chosen spectrograph has a header which does not
match expectation
- Pypeit can now read (currently for Keck DEIMOS only) the list of arc
lamps from the header and use it for wavelength calibration.
- Allow one to restrict the wavelength range of the arxiv template
- Fixed a bug in HolyGrail that did not allow for sigdetect and rms_wavelength to be
slit dependent lists.
- Set DEIMOS FWHM default to 10 pixels
- Fixed a bug in HolyGrail that did not allow for sigdetect and
rms_wavelength to be slit dependent lists.
- Improvements for MOSFIRE:
- uses slitmask info in the slit edge tracing
- associates RA, Dec and Object name to each extracted object
- extracts undetected objects using the predicted position from
slitmask info
- uses dither offeset recorded in the header as default
slitmask_offset, but the user can provide the maskdef_id of a slit
with a bright object that can trace the offset.
- improvements in the frame typing
- Implements new Mark4 detector for Keck/LRISr (aka keck_lris_red_mark4)
- QL script for Keck/DEIMOS
- Implemented flux calibration and grating correction for datacubes.
1.6.0 (1 Oct 2021)
------------------
- Modifications to reduce header crashes
- Added `image_proc.rst` doc, which includes a table with the primary parameters
that affect the control flow of the image processing.
- Added exptime and units to the PypeItImage data model.
- Made bias subtraction available to the dark image processing (i.e., if people
request bias subtraction for darks, the bias needs to be passed). Similarly,
added dark to the buildimage calls in get_arc and get_tiltimage.
- Streamlining of the operations in pypeit.core.flat.flatfield.
- Digitization noise no longer added to readnoise calculation by default.
- Include "processing error" in error budget. Accounts for, e.g., readnoise in
dark image, etc.
- Include error calculation in overscan subtraction. The error estimate is the
standard error in the median, which will be an overestimate for the savgol
method.
- Allow for pinhole and sky frames in buildimage_fromlist.
- In pypeit.images.rawimage.RawImage:
- Conversion from ADU to counts is now the first step for all processing.
- Added an `empirical_rn` parameter that allows the users to use the
overscan region to estimate the detector readnoise for each image
processed, and this estimation of the readnoise is now in its own method.
- Subtraction of the dark is now done after the conversion of the image to
counts.
- Dark subtraction is now always performed using the tabulated values for
each detector. A warning is thrown if the dark frames are provided and
the measured dark-current from a dark image is more than 50% different
from the tabulated value.
- Whether or not you add the shot noise and a noise floor to the variance
image are now optional and controlled by parameters in ProcessImagesPar.
- Changes to default ProcessImagesPar parameters: use_specillum = False for
all frame types; shot_noise = False and noise_floor = 0 for biases; and
use_overscan=True, use_biasimage=True, noise_floor=0., and mask_cr=True
for darks. Adjustments propagated to individual spectrographs.
- BPM is not recalculated after applying the flat-field correction because
it is not longer changed by that function.
- The code keeps track of the image scaling via the flat-field correction,
and propagates this to the noise model.
- Compute and save a "base-level variance" that includes readnoise, dark
current, and processing error as part of the PypeItImage datamodel.
- Added `base_var` and `img_scale` to the datamodel of PypeItImage, as well
as the noise_floor and shot_noise booleans. All of these are used by
pypeit.core.procimg.variance_model to construct the error model.
- Added BADSCALE bit to ImageBitMask to track when flat-field corrections
are <=0.
- Added `update_mask` and `select_flag` methods to PypeItImage as convenience
methods used to update and extract information from the fullmask bitmask
attribute.
- CombineImage now re-calculates the variance model using the stacked estimate
of the counts instead of propagating the estimates from the individual
exposures.
- CombineImage performs a masked median when combine_method = 'median', and the
error is the standard error in the median.
- Simplifies stacking of bits in CombineImage.
- Calculation of the variance in processed images separated into two functions,
pypeit.core.procimg.base_variance and pypeit.core.procimg.variance_model.
These replace variance_frame.
- Added a "detectors" doc, and an automatically generated table with relevant
detector parameters (including the dark current) used for instrument.
- Improved fidelity of bspline timing tests using timeit.
- Added inverse variance images to MasterBias and MasterDark frames so that they
are available for re-use.
1.5.0 (11 Aug 2021)
-------------------
- Doc updates, including reorganization of the installation doc, fluxing and
telluric docs, and automatic construction of the package dependencies.
- Add new pixelflat_min_wave parameter below which the mspixelflat is set to 1.
- Add `pypeit_install_telluric` and `pypeit_install_ql_masters` scripts. The
latter creates a symlink to the directory with the QL masters that will be
used if the QL_MASTERS environmental variable does not exist.
- Improved `edgetrace.maskdesign_matching` to always return syncronized traces.
- Pypeit can now deal with dithered observations (only for DEIMOS for now), by
finding the offset of the observed slitmask from the expected position in the design file.
- There are three options the user can use to find the slitmask offset: bright objects,
selected slit, or alignment boxes.
- Pypeit run object finding for the alignment boxes but it does not extract them.
- `reduce.run` is now split in two methods: `run_objfind` and `run_extraction`.
- There are now 2 loops over the detectors in `pypeit.reduce_exposure`: the first
one runs calibrations and object finding for all the detectors and the second one
runs the extraction. In between the two loops, the slitmask offset is computed.
- A script (`get_telescope_offset`) to determine the telescope pointing offsets is
added to `pypeit/spectrographs/keck_deimos.py`
- Improve SOAR Goodman fluxing
1.4.2 (06 Jul 2021)
-------------------
- Added a common base class for all scripts
- Script methods now included in Sphinx documentation
- Updated `pypeit.scripts.scriptbase.SmartFormatter` to enable wrapping
long lines and specify lines with a fixed format using `F|`.
- Made `pypeit.core.telluric.Telluric` subclass from
`pypeit.datamodel.DataContainer`, and added some basic unit tests.
This led to some changes in the existing datamodel.
- Made `pypeit.sensfunc.SensFunc` subclass from
`pypeit.datamodel.DataContainer`, and added some basic unit tests.
This led to some changes in the existing datamodel.
- Allowed `pypeit.datamodel.DataContainer` parsing methods to used
pseudonyms for HDU extension names and base classes to read the
datamodels of subclasses. Both added new keywords that default to
previous behavior.
- Moved some functions to avoid circular imports
- `pypeit.coadd1d.OneSpec` -> `pypeit.onespec.OneSpec`
- `pypeit.core.coadd.get_wave_grid` ->
`pypeit.core.wavecal.wvutils.get_wave_grid`
- `pypeit.core.coadd.sensfunc_weights` ->
`pypeit.sensfunc.sensfunc_weights`
- Add LDT/DeVeny spectrograph
- Add 6440.25A CdI line (LDT/DeVeny)
- Modify SOAR to read their (truly) raw files
- GMOS doc updates
1.4.1 (11 Jun 2021)
-------------------
- Adds SOAR/Goodman red camera
- Update to Gemini-S telescope info
- Make PypeIt ISO 8160 (more) compliant
- Address an Identify bug
- Add blocking filter to DEIMOS config
- NOT/Alfosc updates
- A pair of fixes for shane_kast_red
- Add NTT EFOSC2 spectrograph
- Add standard stars CD-34241 and CD-329927 to esofil
- Add wavelength solution for keck_lris_red 600/10000
- `pypeit_show_2dspec` shows traces of forced extraction and manual
extraction with different colors
- Updated docs about extraction and DEIMOS
- Implement multi-detector flexure estimates
- Fix error in variance for numpy fitting routines
- Introduce HOWTO for DEIMOS
- Method for slupring in a standard observed and reduced by WMKO
1.4.0 (23 Apr 2021)
-------------------
- Include a fix for when no edges are detected in `EdgeTraceSet` by
adding the `bound_detector` parameter. Most instruments have a
default of `bound_detector = False` meaning that the code will skip
processing any detector where no slit edges are found. Some
instuments set the default to be `bound_detector = True` because the
slit edges always or often fall off the edge of the detector (i.e.,
the detector is fully illuminated). These instruments are currently
`mmt_mmirs`, `mmt_bluechannel`, `not_alfosc`, and `shane_kast`; note
that some `gemini_gmos` data in the DevSuite require
`bound_detector=True`, as well.
- Improved wavelength template for DEIMOS gratings: 600ZD, 830G.
- Added new ArI, KrI, NeI, XeI arc lines.
- PypeIt can now compute arc line FWHM from the lines themselves. This
is controlled by a new parset, ``fwhm_fromlines``, which is set to
False by default, except for DEIMOS.
- Added a development document about the DEIMOS wavelength calibration.
- Limit reduction to detectors 3 and 7 when DEIMOS LVM mask is used
(other detectors are empty)
- Add `pypeit_obslog` script that simple compiles and prints metadata
from a set of fits files needed by pypeit to run.
- Change `PypeItSetup.from_file_root` to *require* the output path to
write the vanilla pypeit file. If no path is provided, the object is
instatiated without creating any output.
- Fixed bug in sensitivity function code adressing issue #747. Revamped
sensitivity function completely to compute zeropoints and throughput.
Enhanced sensfunc.py QA.
- Added MOSFIRE QL script.
- Added support for VLT/SINFONI K 25mas (0.8x0.8 arcsec FOV) platescale
- Updated docs for differencing imaging sky subtraction.
- Added "sky" frametype for difference imaging sky subtraction
addressing issue # 1068
- Improved and sped up sensitivity function telluric codes.
- Fixed bugs in ArchiveReid automatic wavelength identification.
- Removed numba dependency.
- Improved pypeit_view_fits script.
- Fixed ginga bugs in display.py and added automatic cuts to show_2dspec
- Added latin hypercube sampler to pypeit.utils which is required for
differential evolution optimizations.
- Improved GMOS R400 wavelength solution
- Turned off GMOS-S binning restriction
- Add GTC OSIRIS spectrograph
- Updates for docs on adding new spectrographs. And a bok test
- Added a new ``pypeit_collate_1d`` tool to automatically group 1D
Spectra from multiple files by group and coadd them.
- PypeIt will now add HISTORY keyword entries to FITS files.
- `use_maskdesign` is turned off for DEIMOS LVM masks
- a new parameter `use_user_fwhm` is added in `ExtractionPar` to allow
the user to set their preferred fwhm
- Improved `slittrace.assign_maskinfo`
- PypeIt can now force extractions of DEIMOS non detected objects at the
location expected from slitmask design.
- SpecObj and SlitTrace datamodel versions updated
1.3.3 (24 Feb 2021)
-------------------
- (Hotfix) Command-line argument bug in `pypeit_coadd_1dspec` script.
- (Hotfix) Bug fix in `pypeit_obslog` script.
- (Hotfix) X-Shooter bits
1.3.2 (08 Feb 2021)
-------------------
- (Hotfix) Bug in content type of README file that prevented upload to
PyPI
1.3.1 (01 Feb 2021)
-------------------
- pypeit_chk_wavecalib script
- Option to limit channels shown for pypeit_show_2dspec
- sigdetect on in full_template
- Added new ArI, ArII lines
- Improved 1Dfit QA
- Final wavelength template for DEIMOS 900ZD
- Fix a bug in `pypeit/core/arc.py` and `pypeit/core/wavecal/autoid.py` due
to the padding to the arc frames
- Added a new XeI line
- Turn off sigma clipping for DEIMOS arc frames.
- Refactor setup.py to use setup.cfg to define package configuration
- Refactor version handling to use setuptools_scm to grab version info from git tags
- Add support for testing within isolated environments via tox
- Refactor CI to use tox to run tests
- Add cron-scheduled tests to CI
- Add tests to CI to cover macos, windows, and conda installations
- Refactor wrapper scripts in bin/ to be entry_points defined in setup.cfg
- Deprecate check_requirements now that dependencies are handled by the installation
1.3.0 (13 Dec 2020)
-------------------
- DATE-OBS, UTC, AMPMODE, and MOSMODE added to metadata for DEIMOS, and
the first three are now included in the auto-generated pypeit files.
- DEIMOS AMPMODE is now included in the list of metadata used to
determine the DEIMOS configuration (setup).
- Frames ignored by
`pypeit.metadata.PypeItMetaData.unique_configurations` used to
establish the unique configurations are now set by
`pypeit.spectrographs.spectrograph.Spectrograph.config_independent_frames`.
These default to 'bias' and 'dark' frames.
- `pypeit.spectrographs.spectrograph.Spectrograph.config_independent_frames`
can also return a *single* keyword selecting the metadata column used
to match these frames to a given configuration. For DEIMOS, this is
used to match bias and dark frames to a configuration observed on the
same date. Currently these frames can only be set to a single
configuration.
- Added `pypeit.metadata.PypeItMetaData.clean_configurations` that
ignores frames that cannot be reduced by pypeit, as set by
`pypeit.spectrographs.spectrograph.Spectrograph.valid_configuration_values`.
For DEIMOS, this is used to ignore frames that are taken in
direct-imaging mode or using anything except the B amplifier to read
the data. The ignored frames are removed from the metadata table
(`fitstbl`).
- `update_docs` script now builds the html as well as the api rst files.
It also prints a pass/fail comment.
- Added tests to `pypeit/tests/test_setups.py` to test that PypeIt
correctly and automatically identifies frames from multiple DEIMOS
configurations and that `pypeit.pypeitsetup.PypeItSetup` correctly
produces separate pypeit files for each configuration.
- Added a development document reporting that PypeIt now satisfies the
`PD-3` requirement Keck outlined for the DEIMOS PypeIt pipeline.
- Building the docs now dynamically generates an example pypeit and
sorted file for inclusion in the PypeIt documentation.
- The setup block is now a simple listing of the keywords and values
used to identify the instrument configuration.
- Refactor identify GUI and improve its docs
- Modest refactoring of templates.py
- Construction of wavelength arxiv files for DEIMOS 1200B and blue 1200G
- Pypeit now adds DEIMOS slits that are expected from the slitmask design
but not found in the tracing process.
- PypeIt now flags as “BOXSLT” DEIMOS slits that are expected to be
alignment boxes from slitmask design.
- Added a table with DEIMOS slitmask design and objects info to the
SlitTraceSet datamodel
- Add support for MMTO Blue Channel Spectrograph
- Add GitHub Actions CI workflow
- Incorporates a procedure to enable GMOS Nod and Shuffle observations
- New GMOS wavelength solutions
- Remove Travis CI config
- General housecleaning of spectrographs
- Documentation improvements
- Dynamically builds table of available spectrographs; see
`pypeit.spectrographs.available_spectrographs`
- `pypeit.defs` is now deprecated
- Removed usage from `pypeit.pypmsgs` and moved it to `run_pypeit.py`
- Many Spectrograph instance attributes are now class attributes; in
particular, previous instance attribute `spectrograph` is now `name`.
- Added class attributes that set if the spectrograph is supported and any
comments for the summary table.
- `default_pypeit_par` is now a class method, which allows the name of the
spectrograph to be defined in a single place
- Valid spectrographs are no longer checked by
`pypeit.par.pypeitpar.ReduxPar`. This caused a circular import in the
new strucuture. The parameter `par['rdx']['spectrograph']` is virtually
always checked by `load_spectrograph`, so I don't think this is a
problem.
- Kastr 300 grating solutions
- Hotfix to include the solutions!
- Improved DEIMOS slitmask design matching
- Assign RA/DEC to DEIMOS extractions
- DEIMOS object RA, Dec, and name returned when running `pypeit_show_1d --list` and saved in
the .txt file with the list of 1d spectra.
- DEIMOS object name and `maskdef_id` visible in ginga when running `pypeit_show_2d`
- Fix sigma clipping bug!
1.2.0 (15 Oct 2020)
-------------------
- Frame-typing tweaks for DEIMOS
- Exposure-time ranges removed
- All frame types now key off OBSTYPE
- Added more detail on citation policy to main page on readthedocs
- Added docs for BitMasks
- Altered scripts interface to allow for dynamically making the help doc
files
- full spatial/spectral flexure and heliocentric corrections implemented
for IFU reductions
- optimal weights in datacube generation
- Docs for skysub, extraction, flat fielding
- New skysub options for masking and suppressing local
- Added `pypeit/core/convert_DEIMOSsavfiles.py` to convert .sav files
into fits files
- Added "amap" and "bmap" fits files in
`pypeit/data/static_calibs/keck_deimos/` for DEIMOS optical model
- Added `pypeit/core/slitdesign_matching.py` and `maskdesign_matching`
to `EdgeTraceSet`
- Added ParSet for switching ON the slit-mask design matching. Default
is ON for `keck_deimos`
- Pypeit registers `maskdef_id` in SlitTraceSet if instrument is
`keck_deimos`
- Fix assignment bug in fitting bspline
1.1.1 (10 Sep 2020)
-------------------
- (Hotfix) Fluxing doc edits
- (Hotfix) Fix sdist pip installation
1.1.0 (8 Sep 2020)
------------------
- Fixed a bug for IR reductions for cases where only negative object
traces are identified. These were accidentally being written to the
spec1d file.
- Fixed a bug fixes a bug in full_template wavelength reidentification
for situations where extreme wavelength coverage slits results in
reidentification with a purely zero-padded array.
- Fixed a bug fixes a bug in full_template wavelength reidentification
for situations where extreme wavelength coverage slits results in
reidentification with a purely zero-padded array.
- Fixed another such bug arising from these zero-padded arrays.
- (Hotfix) Deal with chk_calibs test
- Script to generate combined datacubes for IFU data.
- Changed numpy (> 1.18.0) and scipy (> 1.4.0) version requirements
- Allow show2d_spec, chk_edges, chk_flats to load older Spec2DObj
datamodel versions
- Implemented a plugin kindly provided by the ginga developers to
display images with a secondary wavelength image WCS.
- Removes dependency on @profxj's ginga fork, and avoids a bug when
using WCS image registration in that fork.
- `pypeit/ginga.py` moved to `pypeit/display/display.py` and ginga
plugin added to `pypeit/diplay` directory.
- ginga plugin registered as an entry point in `setup.py`
- Added a script to check that the plugins are all available.
- Installation docs updated. Both `ginga` and `linetools` are now
installed via pip.
- Deprecated `pypeit/debugger.py` and `pypeit/data/settings`
- Removed h5py as a dependency
- `linetools` is now listed in `pypeit/requirements.txt` until I can
check if it still causes readthedocs to fail...
- Modify Spec2DObj 2D model for float32 images
- `pypeit.tracepca.TracePCA` and `pypeit.edgetrace.EdgeTraceSet` now
subclass from `pypeit.datamodel.DataContainer`
- Refactor WaveCalib into a DataContainer
- Refactor fitting + PypeItFit DataContainer
- Coadd2D bug fixes
- Coadd2D without spec1d files
- Coadd2D offsets
- Some Coadd2D docs
- Manual extraction
- Improve LBT/LUCI
- Add MMT/MMIRS
- QL script for Keck/MOSFIRE (beta version)
- Correct det bug in keck_lris
- Modifications to allow for flailing LRISr detector
- Modifications for parse LRIS LAMPS prior to 2010 upgrade
- Added support for P200/DBSP and P200/TripleSpec
1.0.6 (22 Jul 2020)
-------------------
- (Hotfix) Deal with wavecalib crash
- Fix class and version check for DataContainer objects.
- Script to check for calibration files
- No longer require bias frames as default for DEIMOS
- Implement grism19 for NOT/ALFOSC
- Introduced another parameter used to identify box slits, as opposed to
erroneous "slits" found by the edge tracing algorithms. Any slit that
has `minimum_slit_length < length < minimum_slit_length_sci` is
considered a `BOXSLIT`, any slit with `length < minimum_slit_length`
is considered a `SHORTSLIT`; the latter are always ignored.
- Introduced order matching code into EdgeTraceSet.
- This helps fix an issue for GNIRS_10L caused by the orders
shifting.
- Introduces two paramters in `EdgeTraceSetPar` to assist the
matching: `order_match` and `order_offset`
- Echelle spectrographs should now always have `ech_order` defined
in the SlitTraceSet object.
- Removes the need for `Spectrograph.slit2order` and
`Spectrograph.order_vec`. Changes propagated, primarily in
`wavecalib.py`, `autoid.py`, and `reduce.py`.
- Adds in Keck/LRISr with the original detector
- Adds in Keck/LRISb with the FITS format
1.0.5 (23 Jun 2020)
-------------------
- Add median combining code
- Make biasframes median combine by default
- Implemented IFU reduction hooks
- KCWI reduction complete up to spec2D frames
- Implemented new flatfield DataContainer to separate pixelflat and
illumflat
1.0.4 (27 May 2020)
-------------------
- Add a script (pypeit_flux_setup) for creating fluxing, coadd1d and
tellfit pypeit files
- Add telluric fitting script, pypeit_tellfit
1.0.3 (04 May 2020)
-------------------
- Add illumflat frametype
- Enable dark image subtraction
- Refactor of Calibrations (remove cache, add get_dark)
- Enable calibration-only run
- Clean up flat, bias handling
- Make re-use masters the default mode of run_pypeit
- Require Python 3.7
- Fixed a bug in NIRES order finding.
- Add NOT/ALFOSC
- Fluxing docs
- Fix flexure and heliocentric bugs
- Identify GUI updates
1.0.2 (30 Apr 2020)
-------------------
- Various doc hotfixes
- wavelength algorithm hotfix, such that they must now generate an entry
for every slit, bad or good.
1.0.1 (13 Apr 2020)
-------------------
- Various hot fixes
1.0.0 (07 Apr 2020)
-------------------
- Replaces usage of the `tslits_dict` dictionary with
`pypeit.slittrace.SlitTraceSet` everywhere. This `SlitTraceSet`
object is now the main master file used for passing around the slit
edges once the edges are determined by `EdgeTraceSet`.
- Removes usage of `pypeit.pixels.tslits2mask` and replaces it with
`pypeit.slittrace.SlitTraceSet.slit_img`.
- Significant changes to flat-fielding control flow.
- Added `rej_sticky`, `slit_trim`, `slit_pad`, `illum_iter`,
`illum_rej`, `twod_fit_npoly` parameters to FlatFieldPar.
- Illumination flat no longer removed if the user doesn't want to
apply it to the data. The flat was always created, but all that
work was lost if the illumination correction wasn't requested.
- Replaced tweak edges method with a more direct algorithm.
- `pypeit.core.flat.fit_flat` moved to
`pypeit.flatfield.FlatField.fit`.
- Reoriented trace images in the `EdgeTraceSet` QA plots. Added the
sobel image to the ginga display.
- Added `bspline_profile_qa` for generic QA of a bspline fit.
- Eliminate MasterFrame class
- Masks handled by a DataContainer
- Move DetectorPar into a DataContainer (named DetectorContainer) which
enables frame-level construction
- Advances to DataContainer (array type checking; nested DataContainers;
to_master_file)
- Dynamic docs for calibration images
- Every calibration output to disk is help within a DataContainer,
separate from previous classes. Exception is WaveCalib (this needsd a
fit DataContainer first)
- Substantial refactoring of Calibrations
- Add MDM OSMOS spectrograph
- Moved pypeit.core.pydl.bspline into its own module, `pypeit.bspline`
- Introduced C backend functions to speed up bspline fitting
- now require `extension_helpers` package to build pypeit and
necessary files/code in `setup.py` to build the C code
- C functions will be used by default, but code will revert to pure
python, if there's some problem importing the C module
- Added tests and pre-cooked data to ensure identical behavior
between the pure python and C functions.
- Moved some basis function builders to pypeit.core.basis
- Release 1.0 doc
- Lots of new docs
- pypeit_chk_2dslits script
- DataContainer's for specobj, bspline
- Introduction of Spec2DObj, AllSpec2DObj, and OneSpec (for Coadd1D)
- Added bitmask to SlitTraceSet
- Introduced SlitTraceSet.spat_id and its usage throughout the code
- Spatial flexure corrections
- Significant refactor of flatfield.BuildFlatField.fit()
- Spatial flexure measuring code
- PypeItPar control
- Modifications to SlitTraceSet methods
- Illumflat generated dynamically with different PypeIt control
- waveimage generated dynamicall and WaveImage deprecated
- Moved RawImage into ProcessRawImage and renamed the latter to the
former
- Continued refactoring of Calibrations
- Initial code for syncing SpecObjs across exposures
- Option to ignore profile masking during extraction
- Additional code in DataContainer related to MasterFrames
- Eliminated WaveImage
- Updates to QL scripts
- Lots of new tests
0.13.2 (17 Mar 2020)
--------------------
- Added PypeIt identify GUI script for manual wavelength calibration
- Add bitmask tests and print bitmask names that are invalid when
exception raised.
- Parameter set keywords now sorted when exported to an rst table.
- Enable user to scale flux of coadded 1D spectrum to a filter magnitude
- Hold RA/DEC as float (decimal degrees) in PypeIt and knock-on effects
- Add more cards to spec1d header output
- Fixes a few sensfunc bugs
- Added template for LRIS 600/7500
- Deal with non-extracted Standard
- docs docs and more docs
- A QA fix too
0.13.1 (07 Mar 2020)
--------------------
- Missed a required merge with master before tagging 0.13.0.
0.13.0 (07 Mar 2020)
--------------------
- Refactored sensitivity function, fluxing, and coadding scripts and
algorithms.
- Added support for additional near-IR spectrographs.
- Restrict extrapolation in tilt fitting
- Implemented interactive sky region selection
0.12.3 (13 Feb 2020)
--------------------
- Implemented DataContainer
- Added fits I/O methods
- Implemented SlitTraceSet
- Setup of `pypeit.par.pypeitpar` parameter sets should now fault if the
key is not valid for the given parameter set. NOTE: The check may
fail if there are identical keys for different parameter sets.
- Modification to add_sobj() for numpy 18
0.12.2 (14 Jan 2020)
--------------------
- Introduces quick look scripts for MOS and NIRES
- Bumps dependencies including Python 3.7
- Modest refactoring of reduce/extraction/skysub codes
- Refactor of ScienceImage Par into pieces
- Finally dealt with 'random' windowing of Shane_kast_red
- Dynamic namp setting for LRISr when instantiating Spectrograph
0.12.1 (07 Jan 2020)
--------------------
- Hotfixes: np.histogram error in core/coadd1d.py, np.linspace using
float number of steps in core/wave.py, and sets numpy version to 1.16
0.12.0 (23 Dec 2019)
--------------------
- Implemented MOSFIRE and further implemented NIRSPEC for Y-band
spectroscopy.
- Fixed bug in coadd2d.
- Add VLT/FORS filters to our database
- Improved DEIMOS frame typing
- Brings Gemini/GMOS into the suite (R400)
- Also an important change for autoid.full_template()
- Fixed trace extrapolation, to fix bugs in object finding. Tweaks to
object finding algorithm.
- Major improvements to echelle object finding.
- Improved outlier rejection and coefficient fitting in pca_trace
- Major improvements to coadd routines in coadd1d
- Introduced telluric module and telluric correction routines
- Implemented tilt image type which is now a required frame type
- Streamlined and abstracted echelle properties and echelle routine in
spectrograph classes.
- Revamped 2-d coadding routines and introduced 2-d coadding of
MultiSlit data
- Improved ginga plotting routines.
- Fixed bug associated with astropy.stats.sigma_clipped_stats when
astropy.stats.mad_std is used.
- Refactor BPM generation
- Merge raw_image loading with datasec_img and oscansec_img generation
- Sync datasec_img to image in ProcessRawImage
- Started (barely) on a path to having calibration images in counts and
not ADU
- Refactors GMOS for get_rawimage method
- Enables GMOS overscan subtraction
- Adds R400 wavelength solution for old E2V chip
- Revises simple_calib() method for quick and dirty wavelength
calibration
- Adds a related show_wvcalib script
- Changes to ech_combspec to better treat filenames
- Fixed bug when bias was set to 'force' which was not bias subtracting
- Implemented changes to vlt_xshooter_nir to now require darks taken
between flats
- Made flat fielding code a bit more robust against hot pixels at edge
of orders
- Added pypeit_chk_flat script to view flat images
- Refactored image objects into RawImage, ProcessRawImage, PypeItImage,
BuildImage
- Moved load() and save() methods from MasterFrame to the individual
calibration objects
- Converted ArcImage and FlatImages into counts
- Added code to allow for IVAR and RN2 image generation for calibs
- Added several from_master_file() instantiation methods
- Use coadd2d.weighted_combine() to stack calibration images
- Major refactor of slit edge tracing