-
Notifications
You must be signed in to change notification settings - Fork 0
/
RELEASE_NOTES
5796 lines (5216 loc) · 301 KB
/
RELEASE_NOTES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
v5_11_0
Tag for final eBOSS pipeline
JEB (2018-09-24)
Removing exposure 258988 of plate 9438 mjd 58125 because of trail in data
- pro/spec2d/spplan2d.pro
Added iterative sky-subtraction. Basically we just perform the
sky subtraction two times. This reduces the average sky residuals.
Added nskyiter option to spreduce2d.pro
- pro/spec2d/spreduce2d.pro
- pro/spec2d/spreduce.pro
- pro/spec2d/extract_object.pro
- pro/spec2d/skysubtract_iter.pro
Fixed bug on the djs_median calls where the reflect option was
not correctly implemented. This was causing some extra residuals
near the edges of the spectra.
- spflux_v5.pro
- spcoadd_v5.pro
- skysubtract.pro
- extract_bundle_image.pro
- combine1fiber.pro
-------------------------------------------------------------------------------
v5_10_11
JEB (2018-07-24)
Implemented DESI stellar templates for flux calibration
- pro/spec2d/spflux_read_desi.pro (new)
- pro/spec2d/spflux_v5.pro
Tests on this change are documented in
https://trac.sdss.org/attachment/wiki/eBOSS/Pipeline/desi_stars_in_eboss.pdf
JEB( 2018-07-09)
Increased by one digit EXPID header keywords to solve
issue with plate 6138-56598 which contains more than 100 frames
- bolton_biasgen.pro
- spbiasgen.pro
- spcoadd_v5.pro
- spflatgen.pro
- spflatten2.pro
- bin/reformat_spectra.py
-------------------------------------------------------------------------------
v5_10_10
JEB (2018-04-15)
Fixed bug (nPoly was still set to 2 while it should be 0 for the new
extraction)
- pro/spec2d/spcalib.pro
Fixed MAG field in eboss23 ELG plates with FIBER2MAG from DECals data.
This modification happened directly in $SPECLOG_DIR.
-------------------------------------------------------------------------------
v5_10_9
JEB (2018-03-02)
Implementation of background fit in extraction provided by Julien Guy.
The background contribution is removed from the full 2D image using
the flux between bundles.
- pro/spec2d/extract_bundle_image.pro
- pro/spec2d/extract_bundle_row.pro
- pro/spec2d/extract_object.pro
Modified algorithm to compute plate S/N for ELGs. We are doing similar than
the SOS, using z-band fluxes free of emission lines.
We did not implement the artificial scaling of S/N values for the offline
reductions.
- pro/spec2d/platesn.pro
This version will require use of version 1_29 of SPECFLAT
- etc/idlspec2d.module
-------------------------------------------------------------------------------
v5_10_8
VM (2018-01-23)
Modifications for RM plates to have the original depth of B=10 and R=22
Hardcoded plateids !!
Changes in the following routines
-pro/apo2d/quickextract.pro : Multiplied SN2 values by a scale factor=0.8
-pro/apo2d/apo_plotsn.pro : For RM plates, use plotsn_rm.pro
Adding:
-pro/spec2d/plotsn_rm.pro : Hardcoded scale factor = 0.8
-------------------------------------------------------------------------------
v5_10_7
VM (2017-10-30)
-pro/spec2d/sdssproc.pro : Change in sdssproc_badformat routine to take care
of the shift in R2 camera transient location after electronics change on MJD 58055
JEB
- pro/spec2d/sdssproc.pro: updated gain values for all cameras
-------------------------------------------------------------------------------
v5_10_6
VM (2017-10-03)
-examples/opLimits.par : new SN2 thresholds 8 and 18 for blue and red cameras respectively.
-pro/apo2d/quickextract_elg.pro : new ELG scalefactor 2.11 instead of 2.58 to match the new depth of 18 in red cameras
-pro/spec2d/plotsn_elg.pro : new ELG scalefactor 2.11 instead of 2.58 to match the new eboss depth of 18 in red cameras
JEB (2017-05-05)
- etc/lamphgcdne.dat : using MANGA list of lines that excludes faint ones.
JEB (2017-03-03)
- pro/spec2d/spcalib.pro: check if traces are too close only over valid pixels [#317]
- pro/spec2d/trace320crude.pro: use mpfitfun instead of djs_polyfit as suggested by David Law [#2068]
VM(2016-12-21) Updating XMID & WAVEMID ranges for spectrograph 2 in examples/opLimits.par
-------------------------------------------------------------------------------
v5_10_3
VM (2016-11-19) SoS modifications
-changed the scale factor in quickextract_elg.pro and plotsn_elg.pro
JEB
- bin/fluxcorr_prior.py: fixed a bug (removing '.gz' from the framefile name)
- bin/reformat_spectra_daily: changed mkdir for makedirs to create spectra/full and spectra/lite
- bin/reformat_spectra_batch: changed mkdir for makedirs to create spectra/full and spectra/lite
-------------------------------------------------------------------------------
v5_10_2
VM (2016-09-09) SoS modifications to handle the ELG plates
-Based on keyword 'programname' in plugmap files, ELG plates will be treated differently for SN2-
measurements. Use sky free wavelengths in z-band to measure meansn2.
-Included a scale factor in quickextract_elg.pro and plotsn_elg.pro to tune the ELG depth.
-Included a constant (=0.083 corresponding to the scalefactor) in the equation on SN plots; plotsn_elg.pro
-Updated opSNlimits.par to include z-band with a slope 0.6
-Updated new gain measurements post 2016 Summer shutdown
-Updated plotsn to exclude objtype='NA'
Adding:
- pro/apo2d/quickextract_elg.pro
- pro/spec2d/plotsn_elg.pro
Updating:
- pro/apo2d/aporeduce.pro
- pro/apo2d/quicktrace.pro
- pro/apo2d/apo_plotsn.pro
- pro/spec2d/sdssproc.pro
- opfiles/opSNlimits.par
-------------------------------------------------------------------------------
v5_10_1
JEB (2016-06-17) Post DR14 tag for daily reductions
- Few bugs fixed after DR14 run
-------------------------------------------------------------------------------
v5_10_0
JEB (2017-01-19) Fix for DR14 tag
- pro/spec1d/spreduce1d.pro: change of specobjid type from LONG64 to ULONG64
- pro/spec1d/readspec.pro: adding /unsigned keyword to get correct values for SPECOBJID in zans structure
- bin/reformat_spectra.py: replace fitsio by astropy.io.fits (fitsio crashes with uint64 values) in tables
JEB (2016-05-24) Tag for DR14.
New python fluxcorr code to avoid biases for low S/N spectra (by Stephen Bailey)
IDL code to perform ADR corrections on individual exposures (by Daniel Margala, S. Bailey and J. Bautista)
Adding:
- bin/fluxcorr_prior.py
- pro/spec2d/xythrucorr.pro
- pro/spec2d/fiberfraction.pro
Updating:
- pro/spec2d/spcoadd_v5.pro
- pro/spec2d/spcombine_v5.pro
- pro/spec2d/spflux_v5.pro
- pro/spec2d/spfluxcorr_v5.pro
- pro/spec2d/spframe_read.pro
- etc/idlspec2d.module : adding platedesign and yannytools modules, required by the new changes.
VM (2016-02-29)
-pro/apo2d/quicktrace.pro : Hack to avoid resetting the magnitudes to 0 for OBJTYE = NA for ELG test plates (8954 to 8959)
-------------------------------------------------------------------------------
v5_9_1
Tag for post-DR13 daily runs
VM (2015-11-13)
-pro/apo2d/quicktrace.pro : Hack to avoid resetting the magnitudes to 0 for OBJTYE = NA for QSO test plates (8788 to 8793)
-bin/guidermon_checkin.sh : updated the script to point to correct SPECLOG directory
v5_9_0 (2015-11-05)
Fix of the previous v5_9_0 for DR13
JEB (2015-10-12)
- pro/plate/plate_to_string.pro:
Correcting plate number definition for plates >= 10k
- pro/spec1d/platemerge.pro:
Correcting trailing blank spaces when generating spAll for some programs
-------------------------------------------------------------------------------
v5_9_0
Tag for DR13
JEB (2015-09-24)
- bin/reformat_spectra_daily:
Script to generate spec files after the daily reductions
- bin/uurundaily_script:
Added reformat_spectra_daily command
- bin/reformat_spectra.py:
If update option on, look for existing spec files per plate-mjd-fiber
(not per plate only)
- bin/reformat_spectra_batch:
Include --update option for catch up of missing plates using the utah cluster
- pro/spec2d/spplan2d.pro:
Add dr13 keyword to exclude eboss plates and some RM plates.
-------------------------------------------------------------------------------
v5_8_2
Test tag for DR13
JEB (2015-09-18)
platelist.pro: adding eBOSS LRGs and ELGs to the N_gal column and eBOSS quasars
to N_QSO column in platelist.
Added eBOSS LRGs to %LRG1 column in platequality.
Added eBOSS quasars to %QSO column in platequality.
Added a %ELG column in platequality.
JEB (2015-09-05) 6-digit plate number armaggedon fixed, but not tested yet with a
problematic plate
Added pro/plate/plate_to_string.pro to translate plate numbers into
strings.
-------------------------------------------------------------------------------
VM : v5_8_1 (2015-08-11)
Adding bin/speclog_svn_update.sh : script called by the cronjob to auto update the $SPECLOG_DIR every morning
Changes in the following files :
bin/sos_apostart : disabled the ssh agent
bin/sos_runnerd.py : disabled doBookKeeping
examples/opLimits.par : updated WAVEMID values post 2015 summershutdown
-------------------------------------------------------------------------------
v5_8_0 (2015-06-12)
== New extraction algorithm by Julien Guy ==
For details, see notes on the new extraction on
https://trac.sdss.org/wiki/eBOSS/Pipeline/v5_8_0
sdssproc.pro :
- transmit the rdnoise vector
extract_object.pro :
- reject highrej=4 lowrej=4
extract_bundle_image :
- new argument :use_image_ivar=use_image_ivar
- saves chi2pdf
- rejection of pixels based on chi2pdf per spectrum
extract_bundle_row :
- new args rdnoise, chi2pdf=chi2pdf, use_image_ivar=use_image_ivar
- lots of changes (see notes)
skysubtract.pro :
- create model of ivar (complex)
- fit
- put back variane
- compute error on sky model
combine1fiber.pro :
- create model of ivar (simple)
- fit
- put back variane
- compute error on sky model
spflux_v5.pro :
- outlier rejection when combining several exposures of std star
spfluxcorr_v5.pro :
- don't fit spfluxcorr if very few valid data
spcalib.pro :
- highrej = 100 lowrej = 100 (100 sigma!!)
- use_image_ivar=1 ; JG more robust to trace offsets for arcs
=== New S/N calculations for plates ==
Mode details on :
https://trac.sdss.org/wiki/eBOSS/Pipeline/PlateSN
fitsn_jb:
- new formula that applies for a wider magnitude range
general_sn:
- implementation of formula above S/N = a*flux/ sqrt( flux + b)
plotsn_jb:
- modified to deal with new S/N fits
opSNlimits.par:
- new magnitude limits: [16., 24.] for all bands
-------------------------------------------------------------------------------
v5_7_13 (2015-05-08) JEB
spreduce1d: fixed bug in the delta redshift to velocity conversion
-------------------------------------------------------------------------------
v5_7_12
VM 2015-04-21:
opLimits.par : Changed b1 camera XSIGMA/WSIGMA tolerances to >1.25 (yellow) and >1.30 (red)
SJB 2015-04-13:
apofix: correct logic when SOS logfile doesn't exist and print more
helpful status/error messages.
VM 2015-03-23:
quicktrace : hack to ignore OBJTYPE="NA" only for plateid > 7517 (eboss)
-------------------------------------------------------------------------------
v5_7_9 (2015-03-10) JEB
uuplotspec: improvements for visual inspection, new database.
People can now login with trac.sdss.org username and password and
save inspections.
We did this to visual inspect LRGs and obtain n(z).
plotsn: coeffs keyword to recover fitsn results for each spectro and each band
plotsn: use dereddened magnitudes in all fits
platesn: same as plotsn
opSnlimits: new slopes derived from the median distribution of slopes, when these were
let free in a test sample. The slopes are still fixed.
VM 2015-02-23
opSnlimits : g and i band sn fitting slope changed to -0.32 and -0.36
QUICKTRACE: Hack to ignore the OBJTYPE='NA' targets to address ticket #2184.
-------------------------------------------------------------------------------
v5_7_6 (2014-11-05) JEB
VM 2014-12-06:
quickextract : adding 'splitsky' to r1 camera for MJD ge 56858 and r2 ge 55300
readplugmap : edit for new extinctiion coefficients (Shalfy 2011)
reject_science : removed the last override 'qbad = 0'
JEB 2014-11-05:
spreduce1d: change redshift range for eboss and sequels plates:
from [-0.01,1.] to [-0.01,2.]
spreduce2d: adding splitsky to r1 camera for MJD ge 56858 due to strange
behaviour
platemerge: added eBOSS mask bits
plotspec: added eBOSS mask bits
uuplotspec: added eBOSS mask bits
uubatchpbs: added stderr in each plate folder in addition to stdout
SJB+JEB 2014-10-25:
sdssproc: updated r1 gain for CCD that was replaced summer 2014.
-------------------------------------------------------------------------------
v5_7_5 (2014-10-10) SJB
Missing one commit to trunk for yet another sdss3 -> sdss4 change.
Committing that and tagging again.
-------------------------------------------------------------------------------
v5_7_4 (2014-10-10) SJB
Tag for SOS
SJB 2014-09-22:
sos_runnerd: make robust to blank PLATETYP keyword for bias and darks.
guidermon_checkin: observer@sos3 -> eboss@sdss4-eboss changes; not yet working
loadSN2Value: BOSS -> eBOSS
SJB 2014-08-28:
sos_runnerd: make more robust if munged data is missing NAME keyword.
SJB 2014-08-25:
platelist: fix bug when creating platelist with multiple run2d
platemerge: (re)add ability to write a single spAll file with multiple RUN2D
versions in it.
reformat_spectra.py : add --mjd option
-------------------------------------------------------------------------------
v5_7_3 (2014-08-26) SJB
Tag for new season of daily eBOSS data processing.
SJB 2014-08-25:
loadSN2Value : update for new platedb connection methods.
apofix : change /not_sos option to /force, and check for eboss@sdss4-eboss
rather than checking for running on sos or sos3.apo.nmsu.edu
DJS 2014-08-24
apod: only reduce eBOSS files, not MaNGA. (This is the very much simplified
script for running SOS, without connecting to platedb or svn.)
-------------------------------------------------------------------------------
v5_7_2 (2014-07-23 ?) SJB
Tag for final DR12 run of post-BOSS data
JCR 2014-07-23:
extract_object: Looks up file spSplitSky.par in $SPECLOG_DIR/opfiles for
plate and camera to determine if splitsky should occur. If plate and
camera are in that file, then splitsky does not occur.
SJB 2014-05-23:
spreduce1d: fix parsing bug when spPlate files have a different prefix
or plateids that have more than 4 digits. regex to the rescue.
SJB 2014-04-28:
rundaily_riemann_script: make sure to svn update speclog and platelist trunk,
not some tagged version
-------------------------------------------------------------------------------
v5_7_1 (2014-04-24) SJB
Tag for continuing with daily processing post BOSS LSS DR12.
Code is identical except addition of spx_to_sdss.pro
DJS 2014-04-22:
spec1d/spx_to_sdss.pro : converter of spectroperfectionism output
to spPlate format for testing.
-------------------------------------------------------------------------------
v5_7_0 (2014-04-10) SJB
Tag for BOSS main survey DR12
SJB 2014-04-09:
SPPLAN2D: typo! Exclude 'test' exposures. Only 'excellent'. Smithers.
SPHDRFIX: Include exposure number in the splog line when changing something.
SDSS_PLATE_SORT (ticket 1896):
- Don't propagate upstream bug where failed position matches could still
have thing_id>0. Replace with thing_id=-1. Photo data already 0.
- Replace ra,dec of failed matches with plug_ra, plug_dec.
- Final result:
thing_id=-1 means failed match and photo data=0 for those.
ra,dec = position match if successful, otherwise plug_ra,plug_dec
SJB 2014-04-08:
SPPLAN2D: check PLATETYP keyword to reject non-BOSS non-EBOSS data.
bin/getPlugmap : update call to catPlPlugMapM to keep freaking SqlAlchemy
warning messages out of our plugmap files.
-------------------------------------------------------------------------------
v5_6_12 (2014-04-04) SJB
Second pre-DR12 tag.
BATCHMERGE: re-enabled pngs with a /makepng option
-------------------------------------------------------------------------------
v5_6_11 (2014-04-24) SJB
Tag for pre-DR12 testing. Will move to v5_7_x series when we think we are
ready to go for real.
SJB 2014-04-04:
PLATEMERGE: removing re-reading plugmap hack for propagating eboss_target0,
in preparation for pre-DR12 testing.
SPCOADD_V5: allow missing spectrographs. Missing data have flux=ivar=0
but still have valid plugmap propagated.
ZCOMPUTE: if all ivar=0, don't bother trying to compute a redshift
SPREDUCE1D: if all ivar=0, flag ZWARNING with NODATA
PLUG2TSOBJ: bug fix in determining indir
SJB 2014-04-01:
SPPLAN2D: tweaks to give more informative warning messages.
SJB 2014-03-27:
opfiles/washers.par, spec2d/readplugmap.pro :
Added final entries for washer usage at the end of BOSS.
SJB 2014-03-25:
SDSSPROC: refactor to separate options for pixflat from badpix mask.
Change default to using both. Use /nopixflat and /nopixmask to turn
them off. Updated multiple other codes to handle changed defaults.
apo2d/*.pro : start using pixflat and pixmask for apo2d routines.
--> Not yet tested with end-to-end pipeline run
SJB 2014-03-25:
pro/testsuite/check_cmass_changes.scr.pro
Added hooks for comparing two productions for how many targets changed
from good to bad, etc.
SJB 2014-02-06:
PLATELIST: added hooks for updating interactive platelist website.
MDO 2014-02-05
COMBSMALLCOLLIMATE: Updated offsets from data on 56691 , second test, in both comsmallcollimate and in sosactor
for the record (offsets are now stored in Hartmann.__init__(),specifically the "fudge" dictionary that is defined at line 73.
https://trac.sdss3.org/browser/repo/ops/actors/sosActor/trunk/python
/sosActor/utils/boss_collimate.py (MDO)
SJB 2014-02-02:
sos_runnerd.py : add check for MaNGA plates and skip them for SOS.
Make sure to run against a new version of specflat that includes badpixel mask and pixbias for r2 holiday 2013 problems (56649) MDO
SJB 2014-01-22:
READPLUGMAP, PLATEMERGE : Added hooks to propagate eboss_target0 flags from
plateHoles files. Includes code in platemerge to override plugmap in
spPlate -- this should be removed for the DR12 tag.
COMBSMALLCOLLIMATE: Updated offsets from data on 56528 for the record (offsets
are now stored in Hartmann.__init__(),specifically the "fudge"
dictionary that is defined at line 100.
https://trac.sdss3.org/browser/repo/ops/actors/sopActor/trunk/python
/sopActor/utils/boss_collimate.py (MDO)
PLUGHISTORY: Report for cart #18, and don't crash when trying to read
data for MANGA data (ie, plate 6655 on cart 1).
Only do SDSS-I/II and BOSS plates by looking at "platetype".
Report fibers plugged less than 95% of the time.
READPLUGMAP: Fix typo in #906 PSF/fiber2flux corrections (CPL)
SPECTRO_SUMMARY: Merged with version in boss/papers/OneDee. Added entries
from DR9 publications which weren't calculated here. (SJB)
SPFLUX_V5: Wrote new SPFLUX_BALMERFIX routine to address ticket #1958
in simplest possible manner. Called on line 1054 of main routine.
Comment that out to restore previous behavior. (ASB, 2014-Jan-17)
-------------------------------------------------------------------------------
v5_6_5 (2013-06-24) ASB
Candidate tag for DR11 galaxy LSS analysis
Built against:
elodie/v1_3
specflat/v1_20
idlutils/v5_5_10
BOLTON_BIASSUB, SDSSPROC: New code to deal with r2 power-supply bias
instability over MJD range [56352, 56371] (Ticket #1759).
COMBSMALLCOLLIMATE: Change offsets because focus ring changes on 56434 (MDO)
PLATELIST: exclude LITTLE_COVERAGE, UNPLUGGED, and BAD_TARGET fibers from
statistics for platelist. Ticket #1727.
SPREDUCE1D: Add ability to override default redshift-fitting range for
galaxy templates by using a file $SPECLOG_DIR/opfiles/spPlateZrange.par
(ticket #1084).
templates/spEigenGal-56436.fits: Copy of galaxy templates pEigenGal-55740.fits
with an extrapolation further to the red to logwave=3.0730, such that
these can be used to z=2.
-------------------------------------------------------------------------------
v5_6_4 (2013-01-22) SJB
New tag for SOS for high read noise warning.
FINDSPEC: Fix to work with new behaviour of PLATELIST looking in RUN2D
subdir names for the platelist.fits file
RDNOISE_HISTORY: Skip hartmanns
SDSSPROC: Add warning for high read noise in addition to low read noise.
SOS_LOGS_CONCAT, SOS_LOGS_PLOTSKY: New procs for looking at summary
SOS log statistics
-------------------------------------------------------------------------------
v5_6_3 (2013-01-22) MDO
Changed collimate offsets and r1 focus to minimize xsig/wsig
MDO 2013-01-21:
collimate.pro: add offset for r1 so focus includes minimizing
xsig+wsig in regions used for SOS
combsmallcollimate.pro: change offsets for xsig/wsig problems
from temperature and focus ring changes
BAW 2013-01-16:
bin/knownMissing.py: script to search for missing files in $SPECTRO_REDUX.
SJB 2012-01-11:
rundaily_riemann_script: always use trunk for platelist.
Turn off platemerge to reduce load on master node; this is
now done in the morning batchmerge job.
SJB 2012-01-10:
PLATELIST: Added /alldirs option to call to get_mjd_dir when using it
to get a list of possible run2d and run1d dirs. This is required because
of recent change to get_mjd_dir to by default only return numeric subdirs.
-------------------------------------------------------------------------------
v5_6_2 (2012-12-28) Tag for SoS with new offset values for XMID and WAVEMID
MDO 2012-12-18:
combsmallcollimate: change offsets from tests on 12-12 and 12-13 due to temperature and focus ring changes.
ASB 2012-12-17:
SPECTRO_SUMMARY: exclude BAD_TARGET spectra from effective Nspectra count.
SJB 2012-12-17:
PLATEMERGE: gzip spAll*.dat file for faster downloads. Leave fits file as is.
SJB 2012-12-05:
SPREDUCE1D : allow RUN1D strings which don't match vX_Y_Z.
PLATELIST: Added columns PROGRAMNAME and TILEID to platelist.txt and .html
--> resolves ticket 1716.
PLATEMERGE: Don't include public bad plates in spAll (e.g. 3698).
-------------------------------------------------------------------------------
v5_6_1 (2012-11-29) Tag for SoS to ignore cart<10 data
SJB 2012-11-29:
bin/reformat_spectra.py: fix COEFF0 keyword when trimming ivar=0 leading
pixels. Back propagate by hand into v5_5_12.
bin/copy_spPlan.py : refactor to only copy plans for good plate-mjd
unless --bad option
BATCHPBS: if topdir option is used to override BOSS_SPECTRO_REDUX,
put that into the batch script as well.
SJB 2012-11-28:
bin/sos_filesequencer: don't process arc/flat/science for cart<10
to protect BOSS from MaNGA tests. Process dark/bias from any cart
to allow afternoon checkout tests while APOGEE carts are loaded.
-------------------------------------------------------------------------------
v5_6_0 (2012-11-16) Tag to reprocess all data; starts a new season toward DR11
ASB 2012-11-15: Added MATCH_SPALL (routine) and
CHECK_CMASS_CHANGES (script) in pro/testsuite
SJB 2012-11-02:
SDSSPROC : Added warning for readnoise<1.0 which could mean
CCD voltages are wrong
SJB 2012-10-28: reformat_spectra.py - Added x column to exp HDUs of spec files.
-------------------------------------------------------------------------------
v5_5_12 (2012-10-23) -- DR10 release candidate
SJB 2012-10-23:
SPREDUCE2D: add photolog_version to log output
SPCOADD_V5: add EXTNAME keywords to spCFrame and spPlate files
JB 2012-10-22:
PLATEMERGE, SPREDUCE1D: Add CAS-styled specobjid to spreduce1d output
ASB 2012-10-20:
FLUX_DISTORTION, SPREDUCE1D: call SDSS_ASTROMBAD rather than looking
for CALIB_STATUS bit to flag bad fields. (Ticket #1583).
MDO 2012-10-09:
SDSSPROC: add case to check if saturated pixel occurs within 3 pixels of top and bottom of image. (Ticket $1424)
opfiles/washers.par: Remove duplicate entry 3537-55116
-------------------------------------------------------------------------------
v5_5_11 (2012-10-04) -- dry run for DR10
BATCHPBS: Added /skip1d option
BBSPEC_EXTRACT: bbspec ymodel moved to HDU 6 in extracted spectra.
BOSSLOG: New proc.
COMBINE1FIBER: only set NODATA in AND_MASK if there really is no data
on any input spectrum; resolves ticket 1301.
COMBINEBPM: updated hand edited bad columns (ticket 1424, MDO)
FLUX_DISTORTION: mask CALIB_STATUS->ASTROMBAD fluxes
(ticket #1583, ASB)
PLATELIST: Change defult input/output location to be in RUN2D subdir
to make it easier to work with multiple versions. Can be overriden
with TOPDIR and OUTDIR options, and multiple RUN2D versions can
still be combined into a single platelist using RUN2D='*' or
RUN2D=[x, y, z], in which case the platelist is written to
$BOSS_SPECTRO_REDUX (not in the RUN2D subdir).
Resolves ticket 1659.
PLATEMERGE: removed Z_PERSON, CLASS_PERSON, Z_CONF_PERSON,
COMMENTS_PERSON from spAll so that quasar catalogs can proceed on
timescale indepdenent of pipeline. DRxQ will be authoritative
source of FPG scans rather than also propagating to spAll.
PLATEMERGE: Added BOSSPRIMARY tag to replicate SPECPRIMARY but to be
immune from downstream modification; changed SPECOBJ_ID to
BOSS_SPECOBJ_ID (Ticket #1634).
READPLUGMAP: append CALIB_STATUS when /calibobj is passed
(ticket #1583, ASB)
SDSS_PLATE_SORT: Adding ERROR to error messages for easier grepping
SDSSPROC: added masking for hotpixel trails (MDO)
SN_MEDIAN: Added "sn_all" keyword for median S/N across all bands
(ticket #1634, ASB)
SPECTRO_SUMMARY: removed N_QSO_SCANNED, N_LYA_INCOMP, and N_LYA_IMPURE
statistics.
SPREDUCE1D: push CALIB_STATUS->ASTROMBAD to ZWARNING->BAD_TARGET
(ticket #1583, ASB)
SPREDUCE1D: Added SN_MEDIAN_ALL (median S/N across all bands) to
output structure (ticket #1634, ASB).
SPREDUCE1D: Added ZNUM_NOQSO (Brownstein)
bin/copy_spPlan.py - New (SJB)
-------------------------------------------------------------------------------
v5_5_10 (2012-08-21)
BOLTON_BIASSUB: Bug fix for CRs in row-by-row kluge-processed overscan
region of wonky b2 quadrant (ASB, ticket #1651).
QUICKTRACE: Bug fix for SOS reductions, which now need to pass cartID
as a keyword to TRACE320CRUDE to be consistent with recent changes
(ticket #1656).
-------------------------------------------------------------------------------
v5_5_9 (2012-08-15)
Tag for electronics bias drift on b2 UR quadrant.
Built against elodie/v1_3, specflat/v1_19, idlutils/v5_4_28+.
APO_CHECKLIMITS: Bug fix to test against values of exactly zero,
although this wouldn't have been triggered.
BAD_BOSSFORMAT: Print mean and 95% of each sub-image.
BOLTON_BIASSUB: Adding kluge to do row-by-row bias subtraction for
upper right quadrant of b2 post-Summer 2012, due to crazy bias
phenomenon that has appeared there (ASB, 2012-Aug-15).
FITFLATWIDTH,SPCALIB,SPREDUCE,TRACE320CRUDE: Minor changes to use a
different number of traces for the SDSS-I spectra (ticket #541).
PLATELIST: Remove %Main from the HTML file, since those targets do
not exist in BOSS.
OPFIBER_GENERATE: Bug fix to work for SDSS-I plates with plate # < 1000
and read traces from spFlat files for the old data model.
READPLUGMAP : reads opfiles/washer.par to get override status for
ZOFFSET. (SJB 2012-08-07)
SDSS_PLATE_SORT: The photoMatch,photoPlate,photoMatch files for the SDSS-I
plates are alread sorted to a particular plugging, and have MJD
in their names. Rather than assume these are sorted, take the first
matching file for a plate and re-sort, as we do for the BOSS files.
SPECTRO_SUMMARY: Summarize BOSS spectro totals from spAll file.
(ASB: motivated by DR9.)
opfiles/opFibers.par: Add trace positions for SDSS-I spectra using
the following cart+plate+MJDs, but then edited to start at MJD=50000
in this file:
1 414 51901
2 400 51820
3 404 51877
4 418 51884
5 406 51869
6 431 51877
7 412 51931
8 410 51816
9 401 51788
-------------------------------------------------------------------------------
v5_5_8 (2012-04-05, ASB)
Tag for spectrophoto bug fix (Ticket #1509), as well as
various SOS-related changes, file cleanups, bug fixes, etc.
Built against elodie/v1_3, specflat/v1_18, idlutils/v5_4_28.
Documentation: Fixed the formatting for IDL documentation in a number of files:
pro/spec1d/bandpassfilter.pro
pro/spec1d/filter_check_chi2.pro
pro/spec1d/filter_solve.pro
pro/spec1d/filter_check.pro
pro/spec1d/filter_select.pro
pro/spec1d/hogg_extinction.pro
pro/spec1d/bolton_mlpca.pro
pro/spec2d/extract_object.pro
pro/spec2d/bolton_sn_fit.pro
pro/spec2d/gausspix.pro
pro/spec2d/bbspec_test.pro
pro/spec2d/trace_sparse_crude.pro
pro/spec2d/smooth_halo.pro
pro/spec2d/smooth_halo2d.pro
pro/inspect/inspectgen.pro
pro/templates/pca_qso_dev.pro
pro/templates/pca_gal_dev.pro
pro/templates/mlpca_star.pro
pro/templates/align_rest.pro
pro/templates/mlpca_cvstar.pro
Fixing some files that don't need to be executable
examples/opECalib-51430.par
examples/opECalib-blank.par
examples/opECalib-51577.par
examples/opECalib-55026.par
DECOLLIDE,RANDOM_DECOLLIDE: Fixed problem with duplicate function names.
EXTRACT_OBJECT: Commented out dispset in calls to SKYSUBSTRACT.
(ticket #1475, JCR).
GAMA_TO_SDSS: New proc for converting 2dF GAMA spectra files to spPlate
format for redshift-fitting with Spectro-1D.
PCA_QSO: Add /allatonce keyword.
SPFLUXCORR_V5: changed multiplicative polynomial fit parameters and
changed fit to depend on index of refraction of air (ASB, ticket #1509)
Makefile (top-level): make doc is now not part of default install
Makefile (lower-level): All unnecessary Makefiles (that only did make install)
are removed.
.cvsignore: Removed unnecessary .cvsignore files
bin/idlspec2d_version: Change from bash to sh.
doc/mk_doc: Removed, document generation now handled directly by doc/Makefile
etc/emlines.par: The H_gamma line fit width was accidentally tied to the
forbidden lines rather than the Balmer lines. Fixed.
examples/opECalib-*.par: Removed executable tags on some files.
examples/opLimits.par: increased limits on r1 wsig by 0.05.
b1 xsig goes high for high fibers at middle pixel numbers but after
collimator swap still within old limits so didn't change
r1 wsig goes high for middle fibers at high pixel numbers
include/export.h: Remove old export.h and link to $IDL_DIR/external/export.h
This is the same as what is done in the idlutils product.
ups/idlspec2d.table: Remove explicit version dependency idlutils v5_4_26.
Remove outdated notation.
-------------------------------------------------------------------------------
v5_5_7 (2012-01-18)
--> Installed at APO
APO_PLOTSN: Line 92 changed to address ticket #1506: SOS not plotting
data from very low S/N exposures.
bin/updatePluggingStatus: Remove to avoid conflict with version in
the platedb product.
ups/idlspec2d.table: Force explicit version dependency idlutils v5_4_26.
-------------------------------------------------------------------------------
v5_5_4 - v5_5_6 (2012-1-10)
[several busted tag iterations sorting out dependencies]
--> Never installed at APO; skip to v5_5_7
BATCHPBS: Turned off calls to plate_spec_image: post-DR9 one should use
plotspec or the spectrawebapp for casual browsing of spectra.
BBSPEC_EXTRACT,FITSCATTER: Implement a naive scattered light model for
the bbspec extraction.
QUICKEXTRACT: Bug fix to sky-subtract in 25 bundles, not just 16.
Use 2-D sky-subtraction with NPOLY=3 in call to SKYSUBTRACT.
SKYSUBTRACT: Cleaned up use of DISPSET argument; needs test (ticket #1475).
VDISPFIT: Change to flatten probability in optional redshift marginalization,
may or may not be retained (ASB)
VDISP_PLATEFIT: New routine to facilitate re-batching of vdisp fitting code
(ASB)
-------------------------------------------------------------------------------
v5_5_3 (2011-11-14)
--> Installed at APO
Tag for SOS with small changes after b2 triplet replacement.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
BOLTON_BIASSUB: add code to handle subarray readout bias subtraction
gracefully (ASB, re: ticket #1466)
COMBINEBPM : Added bad columns for 55800-55850 on r1. After putting the set
temperature for this CCD higher by 5 deg C, r1 looks good with no bad cols.
COMBSMALLCOLLIMATE: Changed offsets for b2 after triplet swap
SDSS_PLATE_SORT : Add MOD 360 to match_ra to prevent crashes on match_ra>360
in photoMatchPlate files (affects plates 4405 and 4277).
SDSS_SPEC_IMAGE, PLATE_SPEC_IMAGE : Added /silent hooks
bin/rundaily_cron, rundaily_riemann_script, rundaily_riemann:
Added silent hooks for sdss_spec_image and plate_spec_image
-------------------------------------------------------------------------------
v5_5_2 (2010-10-19)
Tag for SOS bug fixes.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
COMBSMALLCOLLIMATE: Correctly apply offsets.
examples/opLimits.par: Tweaks to all XMID,WAVEMID values for year 3 (KSD).
-------------------------------------------------------------------------------
v5_5_1 (2011-09-17)
Tag for SOS bug fixes.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
COMBSMALLCOLLIMATE: Update offsets using all hartmanns from 55800-817
examples/opLimits.par: Switch incorrectly-swapped the values for WAVEMID
for r1 & b1. Set error for EXPTIME < 300 sec.
Set error for SN2 less than 0.2.
-------------------------------------------------------------------------------
v5_5_0 (2011-09-09)
Tag for full reductions using replacement r1 CCD.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
COLLIMATE: Add test keyword to help find offsets with SMALLCOLLIMATE.
COMBSMALLCOLLIMATE: Update offsets and add /TEST keyword.
PLATECOMPARE: Fix to work with change from SNMEDIAN from vector to array.
PLOTSPEC, UUPLOTSPEC: Show 3 digits after decimal in rchi2 (ASB)
SDSSPROC: Added new r1 CCD gains to sdssproc for data after MJD 55800:
amps 0-4 gains [1.9253, 1.5122, 1.4738, 1.5053]
bin/reformat_spectra.py: New proc to reformat spectra into one-file-per-object
format (basically ticket #837)
bin/sos_apocommand : updatePluggingStatus no longer accepts -d -v options,
so remove them from the call.
bin/updatePluggingStatus - this has been replaced by
PLATEDB_DIR/bin/updatePluggingStatus. Rename this copy to .._old just
in case we need to get to it again. After next APO install, if things are
running smoothly and we don't need this, we can delete it for good (history
will still be in svn)
-------------------------------------------------------------------------------
v5_4_45 (2011-08-27, ASB)
Bug fixes for DR9 data release.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
BUNDLETHRU: New proc to measure relative throughput of fiber bundles from
flat-fields
PLATESN,PLOTSN: Bug fix(es) for new calling sequence to FITSN, where we no
longer are passing the plugmap but instead the redden values.
-------------------------------------------------------------------------------
v5_4_44 (2011-08-26, ASB)
Tag to catch that last change
If good, this tag will be the DR9 tag.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
EXTRACT_OBJECT: Compute the SKYCHI2 output in the science frame headers
only for good pixels (ticket #712)
-------------------------------------------------------------------------------
v5_4_43 (2011-08-25, ASB)
Tag to clean up a bunch of bugs and straggler tickets.
If good, this tag will be the DR9 tag.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
BOLTON_BIASSUB, SDSSPROC: Fix to make use of SDSSPROC_BADFORMAT
image corrections (ASB, re: ticket # 1408)
CONFIGURATION__DEFINE: Remove some deprecated functions.
EXTRACT_BUNDLE_ROW, EXTRACT_BUNDLE_IMAGE: documentation cleanup
to reflect the bundle-wise extraction mods (ASB)
EXTRACT_OBJECT,SPCALIB,QUICKWAVE: Remove call to getDetectorFormat and
determine the image size from the image.
EXTRACT_OBJECT,FITSN,SPREDUCE: Pass the median reddening values for
the plate from the plugmap file (as "reddeningMed" in the header)
as a set of "REDDEN*" keywords in the output science frames.
Use these values for evaluating the reddening-corrected S/N values
in FITSN. (ticket #879)
FITSN: If there are fewer than 10 points in the mag fit range,
then extend that range to all brighter magnitudes. This will
not effect any proper survey plates. But it allows the S/N fit
to occur for some special plates, such as 5017-5019 which have
much fainter targets. For those plates, then the spectro-photo
stars are selected for the fit. (ticket #1351)
PLATEMERGE: Add DIMS0,DIMS1 keywords to spAllLine file (ticket #1031)
SDSSPROC: Add option to return CCDMASK pixel mask that sets the 'NODATA'
bit for regions of the BOSS CCD that are unused. This should allow
the option of including more mask bits at this stage to propogate
through the pipeline, such as CR masks.
SPBIASGEN: Add /NOPROC option for generating average bias frames
without overscan-trimming, etc.
SPCALIB: Remove some clutter of commented-out code.
SPECLINEFIT,LINEBACKFIT: Add new CONTRANGE keyword for setting the region
for the continuum-fitting for each line. A mean (unweighted) continuum
is evaluated in the blueward and redward bands, and then the average of
those two bands. This is used at a range of [300,600] km/s for stars
and galaxies. QSOs retain the default of selecting the continuum level
at the mid-point of the line, which is actually equivalent since the
background terms are all straight lines for QSOs. (ticket #1243)
SPREDUCE1D: Fix spurious "broadline" subclassing (ASB, ticket #1316)
SPREDUCE,EXTRACT_OBJECT: Make use of the CCDMASK from SDSSPROC,
and include those mask values through the pixel mask. A 'NODATA'
value is currently inherited from the initial SDSSPROC.
Erase several of the bits in the 'NODATA' regions:
['NEARBADPIXEL','LOWFLAT','SCATTEREDLIGHT','NOSKY'].
This partially addresses ticket #1301.
bin/loadSN2Value : fix platedb database connection method.
bin/sos_apostart: Correct instructions about which password to use for ssh
daemon.
-------------------------------------------------------------------------------
v5_4_42 (2011-08-18, ASB)
Tag to fix 8000ang break feature with discontinuous sky model.
If good, this tag will be the DR9 tag.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
EXTRACT_OBJECT, SPREDUCE, SPREDUCE2D: Enabling split sky model between
spatial halves (ASB, chasing ticket #1388)
Changed RAWDATA_DIR to BOSS_SPECTRO_DATA in 40 files (Ticket #1067).
Script that did this is in misc/rawdata2bossdata (ASB)
-------------------------------------------------------------------------------
v5_4_41 (2011-08-08, ASB)
Tag for new bias recipe, new pixflats, misc 1d changes, misc 2d bugfixes.
Built against elodie/v1_3, specflat/v1_16, idlutils/v5_4_24.
BOLTON_BIASGEN: Minimum-chi^2 master bias model-building routine (ASB)
BOLTON_BIASSUB: Bias-subtraction using bolton_biasgen outputs (ASB)
FITVACSET: Added handling of 2-phase discontinuity info (ASB)
SDSSPROC: Add measured readnoise for each amp as fits header keywords
RDNOISE0/1/2/3. We might want that info some day for noise modeling.
PLATEMERGE: Set SPECPRIMARY for GALAXY targets now based upon
ZWARNING_NOQSO (if present) rather than ZWARNING (ASB, ticket #1389)
PLATEMERGE: Test for inheritance of Z_NOQSO & related before computing
(see SPREDUCE1D changes) (ASB, ticket #1385).
SDSSPROC: Change to use BOLTON_BIASSUB for BOSS MJD >= 55170 (ASB)
(Will require coordinated tag with specflat.)
SPADD_GUIDERINFO: Limit reporting of seeing to in-focal-plane fibers only
(ticket #1391).
SPREDUCE1D: Compute Z_NOQSO and related values here, rather than in
PLATEMERGE (ASB, ticket #1385).
Also compute redshift-marginalized vdisp likelihood curves
with the Shu et al. 2011 parameter settings.
VDISPFIT: Subtract the # of invvar=0 pixels from vdispdof (ASB).
-------------------------------------------------------------------------------
v5_4_40 (2011-07-05, ASB)