Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consider splitting up the merged catalogs generated with each release #165

Open
moustakas opened this issue Jan 3, 2024 · 0 comments
Open
Labels
might-do Might be addressed.

Comments

@moustakas
Copy link
Member

The merged catalogs we generate with each release are getting unwieldy for the main survey. E.g.,

% du /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main*
68M /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-backup.fits
29G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-bright.fits
45G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-dark.fits
73G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main.fits

With ~18M rows (https://fastspecfit.readthedocs.io/en/latest/iron.html#summary-statistics), the fastspec-*-main.fits file is probably going to become too large to be useful, especially in Y3 and Y5.

So we need to think about either splitting this up into small-nside healpixels (like we do for the lsdr9 VAC of potential targets; see here) or maybe even stop producing this file and just provide wrapper code.

Other suggestions are welcome.

@moustakas moustakas added the might-do Might be addressed. label Jan 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
might-do Might be addressed.
Projects
None yet
Development

No branches or pull requests

1 participant