Alternative output types with better compression and cross platform : .csv.gz & .parquet

[dfalster]

our current plain text output in AusTraits 4.2.1 is 260Mb, with most 230Mb is traits.csv. We use an *.rds filetype for smaller distribution, which compresses to just 12Mb. But this only works for R users.

@cboettig pointed out two alternative types comprised types that are possibly better: .csv.gz and .parquet

Both are cross platform and offer comparable compression to .rds. E.g.

• traits.csv which is 230Mb could be compressed to 10.8Mb as traits.csv.gz, or 10.2Mb as traits.parquet

this will be particularly important when/if exporting wide format (see #19), which is

• 4.2GB as .csv
• 72Mb as .csv.gz
• 26Mb as .parquet

Apache .parquet format (see https://en.wikipedia.org/wiki/Apache_Parquet) is rapidly emerging as new standard, accessible via the arrow package.

[dfalster]

Note also, parquet can be read just like csv:

x <- read_parquet("export/as_wide/austraits_wide.parquet")
x %>%
  filter(taxon_name == "Banksia serrata") %>%
  collect() -> y

Can also filter before read, It's slower to read but saves RAM by only loading the set you want

x <- arrow::open_dataset("export/as_wide/austraits_wide.parquet")

x %>%
  filter(taxon_name == "Banksia serrata") %>%
  collect() -> y

can do this on csv too

x <- arrow::open_dataset("export/as_wide/austraits_wide.csv.gz", format = "csv")