Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix spectra coaddition and cosmic masking #2377

Merged
merged 59 commits into from
Oct 7, 2024
Merged

Fix spectra coaddition and cosmic masking #2377

merged 59 commits into from
Oct 7, 2024

Commits on Sep 17, 2024

  1. initial fix

    segasai committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    6561cc4 View commit details
    Browse the repository at this point in the history
  2. cleanup

    segasai committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    d911f70 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. clarify the code

    segasai committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    a3abe0c View commit details
    Browse the repository at this point in the history
  2. get rid of old variable

    segasai committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    491840a View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Configuration menu
    Copy the full SHA
    6207f1c View commit details
    Browse the repository at this point in the history
  2. add an advanced test that checks the that relationship between

    extracted spectrum S and true spectrum S = R * M  where R is
    resolution matrix is preserved exactly after the coadd
    segasai committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    ca818df View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. unused variable

    segasai committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    39089e6 View commit details
    Browse the repository at this point in the history
  2. a couple more tests

    and fix the bug with masked pixels uncovered by the test
    segasai committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    b2f71bb View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2024

  1. add another test

    segasai committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    86dcd70 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cf3ff51 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5e6c013 View commit details
    Browse the repository at this point in the history
  4. further get rid of duplicated code

    There was some divergence there already i.e.
    gradivar=1./gradvar vs gradivar = (gradvar>0)/(gradvar+(gradvar==0))
    segasai committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    5ba2743 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. add missing log=get_logger()

    Stephen Bailey authored and Stephen Bailey committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    176c7ca View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. add docs for mask_cosmics

    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    4aae55b View commit details
    Browse the repository at this point in the history
  2. fix doc

    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    ae2e569 View commit details
    Browse the repository at this point in the history
  3. fix incorrect behaviour of coadd_cameras when mask is not present

    The test case was also incorrect
    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    67962a7 View commit details
    Browse the repository at this point in the history
  4. fix another bug in the coadd_cameras

    When resolution matrices were added, the non-masked ivars were used.
    (this is kinda irrelevant with the oncoming fix, but important first step)
    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    fc8abf2 View commit details
    Browse the repository at this point in the history
  5. fix coadd_cameras in a similar fashion to coadd()

    Now it passes a similar dedicated test
    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    5fd5f47 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    9409a3a View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    29e6d7b View commit details
    Browse the repository at this point in the history
  8. fix the coadd_cameras recent changes

    Also add a generic test of coadd_cameras validating the coadded matrix
    The currently added test cannot be fully enabled, (it was enabled with the threshold instead)
    because of what happens with edge pixels and how to interpret resolution matrix there
    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    1cd2542 View commit details
    Browse the repository at this point in the history
  9. fix test error

    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    1c914f0 View commit details
    Browse the repository at this point in the history
  10. update the coadd_cameras test so it test the exact correctness

    of the flux + resolution matrix. It now only exclude the pixels
    that are next to the arm edges
    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    cfb8966 View commit details
    Browse the repository at this point in the history
  11. simplify the logic in coadd() to avoid multiple

    if mask is None
    instead set to the mask to zero if there is no one. this way we
    need just one code-path
    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    26895ff View commit details
    Browse the repository at this point in the history
  12. produce correct resolution matrix even for pixels that are flagged

    in all exposures
    Also move the resolution_weight into sepearate function
    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    a308dab View commit details
    Browse the repository at this point in the history
  13. put common functionality in separate function

    Make sure that if the combined ivar is zero from masked pixels we still
    get preserve the ivar and resolution matrix
    segasai committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    0d37f14 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    526e298 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. replace equality by allclose

    segasai committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    a6a5d28 View commit details
    Browse the repository at this point in the history
  2. fix the cosmic masking issue

    segasai committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    d947fb0 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b0febb8 View commit details
    Browse the repository at this point in the history
  4. if we are masking a cosmic using gradient we must mask both pixels

    leading to large gradient
    Add a test of cosmics
    segasai committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    07683d9 View commit details
    Browse the repository at this point in the history
  5. add missing import

    Stephen Bailey authored and Stephen Bailey committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    dd74fc1 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. correctly deal with masked pixels when coadd()'ing

    i.e. the only way to get mask=0 in a pixel must be to have
    at least one mask=0 pixel in the stack
    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    b7396fa View commit details
    Browse the repository at this point in the history
  2. add tests of known coadd bugs

    Stephen Bailey authored and Stephen Bailey committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    49ab8b5 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'resol_fix' of https://github.com/desihub/desispec into …

    …resol_fix
    Stephen Bailey authored and Stephen Bailey committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    5848546 View commit details
    Browse the repository at this point in the history
  4. fix previous commit

    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    f8163c8 View commit details
    Browse the repository at this point in the history
  5. Merge branch 'resol_fix' of https://github.com/desihub/desispec into …

    …resol_fix
    Stephen Bailey authored and Stephen Bailey committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    f149fe0 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d9062cc View commit details
    Browse the repository at this point in the history
  7. Merge branch 'resol_fix' of https://github.com/desihub/desispec into …

    …resol_fix
    Stephen Bailey authored and Stephen Bailey committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    01eee32 View commit details
    Browse the repository at this point in the history
  8. fix test

    Stephen Bailey authored and Stephen Bailey committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    ac1b9c1 View commit details
    Browse the repository at this point in the history
  9. fix the last failing tests

    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    8c2f367 View commit details
    Browse the repository at this point in the history
  10. fix tests comments

    add test for coadd_cameras bits
    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    388dee4 View commit details
    Browse the repository at this point in the history
  11. fix the bug in the original implementation and the new

    one when the comismic maskign would mask completely unrelated
    pixels when one of the spectra in the set is fully masked
    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    7031fb8 View commit details
    Browse the repository at this point in the history
  12. a bit of cleanup.

    Add camera option to mask_cosmics for printing
    Also put the check for the minimum number of spectra needed for cosmic masking
    inside the function to avoid having it in two places
    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    78acd70 View commit details
    Browse the repository at this point in the history
  13. a bit more cleanup

    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    7bc6b68 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    6382c00 View commit details
    Browse the repository at this point in the history
  15. add a final test of the coadd_cameras with cosmics

    (motivated by desire of having all the lines of coadd()/coadd_cameras()
    test-covered)
    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    5b2056e View commit details
    Browse the repository at this point in the history
  16. hopefully final commit.

    Move the chi2 threshold into a function
    also get rid of harcoded value of 3 for min_cosmic rejection
    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    3d17312 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    7b2b5ab View commit details
    Browse the repository at this point in the history
  18. insert space into print

    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    84e1183 View commit details
    Browse the repository at this point in the history
  19. put some protection against coadding of spectra that

    deviate clearly more than prescribed by variance vector.
    In that case I ignore the chi^2 stats and just try to
    empirically estimate the threshold. It will never work
    particularly well, but it is better than doing what we
    are doing
    segasai committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    0ffcc4a View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. add more comments

    make a much cleaner interface for mask_cosmics as it does not need to get
    all the spectra + indices + mask
    Now it will just accept the spectra that needs masking and it will return
    the boolean mask of cosmics.
    That will also make it easier to store that information in the future
    segasai committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    987d359 View commit details
    Browse the repository at this point in the history
  2. update comments

    segasai committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    4c08364 View commit details
    Browse the repository at this point in the history
  3. docstring updates

    Stephen Bailey authored and Stephen Bailey committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    d8811d7 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c4d7d01 View commit details
    Browse the repository at this point in the history
  5. resolve whitespace merge conflict

    Stephen Bailey authored and Stephen Bailey committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    4d67246 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    a161a7d View commit details
    Browse the repository at this point in the history
  7. Merge branch 'resol_fix' of https://github.com/desihub/desispec into …

    …resol_fix
    Stephen Bailey authored and Stephen Bailey committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    1ae5f61 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2024

  1. docstring formatting and whitespace

    Stephen Bailey authored and Stephen Bailey committed Oct 5, 2024
    Configuration menu
    Copy the full SHA
    f3bc12b View commit details
    Browse the repository at this point in the history