BES-3.21.0-46 for Hyrax-1.17.0

News for 3.21.0-46

Reworked the way we implement the BES's singletons (or more of them at least)
so that they use the Meyers Singleton pattern. This includes the EffectiveUrlCache
and TheBESKeys classes.

We have made significant advances to supporting HDF5 files in the DMR++ builder
and interpreter. Our support is close to complete with only a very few features
not supported. This support now includes HDF5's Varying-length string arrays. The
software now also supports scalar strings in all their forms. Note that we also
added support for fixed length string arrays in the DMR++ builder and interpreter.

This version of the BES includes a new feature, dependent on the newest version
of libdap, that reduces the time needed to build NetCDF4 responses by a factor
of 10 or more when the variables in the response are not spatially subset (that
is, when all variables in the response are returned in their entirety). It is still
possible to subset at the variable level and get this performance boost. This is
the 'Direct I/O' feature added ti libdap. NOTE: To use this feature, you must
rebuild DMR++ files using the version of the DMR++ builder associated with this
version of the BES (or get_dmrpp tool from the hyrax-1.16.8-356 build).

We improved the performance of finding the effective URL for a data item when it
is accessed via a series of redirect operations, the last of which is a signed
AWS URL. This is a common case for data stored in S3.

We have added generic Memory and File caching, tailored specifically toward the
cases that arise when serving data from S3 using the DMR++ system.

The BES now has much better support for response size and time limits. Users are
warned about responses that are too large before they are built and the BES
now exits gracefully when a response takes too long to build. The limits are
configurable.

We replaced a home-grown HTTP connection pool with a scheme provided by libcurl.
The two have equivalent performance, but the libcurl version is much easier to
maintain and might offer room for improvement in the future.

C99 compatibility improvements for the GCTP code in/used-by the HDF5 handler.
Thanks ti Florian Weimer for those fixes.

We have added a BES module that can work with S3 using the DMR++ system. This
provides a data flow that is similar to the one we provide for Hyrax in the Cloud
as developer for NASA, but this new module does not make use of the NASA/ESDIS
CMR system to resolve 'NASA Granules' to URLs. This will enable other groups to
use the DMR++ system to serve data from S3.

We have adopted C++-14 as aggressively as we can, resulting in fewer lines of code,
better memory management and more efficient code.

We have moved more of the handlers to build/use DAP4 as the default response format
and build DAP2 responses from those. This reduces the amount of code we have to
maintain.

The BES can sign S3 URLs using the AWS V4 signing scheme. This uses the Credentials
Manager system.

The bundled grid() and geogrid() server side functions now support DAP4.

BES version 3.20.13 for Hyrax-1.16.8

Requires libdap-3.20.11

• Improved error messages for the response is too big, and response tool too
  much time to marshall.

BES version 3.20.12 for Hyrax-1.16.7

Requires libdap-3.20.10

• Patched bug in HDF5Array.cc introduced by the std::vector refactor
• Added time.h header to ppt/SocketUtilities.cc

BES version 3.20.11 for hyrax-1.16.6

Requires libdap-3.20.10

General

• Support for RHEL8
• Refactored get_dmrpp, application. Some features still broken but core
  functionality is working now.
• Enhanced the fileout Covjson handler to support GES DISC AIRS level 3 and GLDAS level 4 products
• Refactored project so that all of the code that depends on the gdal library
  is in a single module, modules/gdal_module.
• Retired use of auto_ptr.
• Refactored timeout implementation and dropped the use of SIGALRM therein.
• Added the support of shuffling filter in the fileout_netcdf handler.
• Modified fileout_netcdf handler to allow netcdf-3 responses to be up to 4GB
  in size. This behavior can be reverted by setting FONc.NC3ClassicFormat=true
  in the BEs configuration (aka /etc/bes/site.conf file)

DMR++ Improvements

• Various improvements for supporting FillValue in dmr++ lifecycle.
• Various improvements for supporting FillValue in dmr++ lifecycle.
• Improved support for arrays of type String.
• Fixed trusted url bug in DMZ parser.
• Added support for "empty" valuyes scalars with associated _FillValue metadata.

get_dmrpp Improvements

• Added support for S3 hosted granules to get_dmrpp
• Added regression test suite for get_dmrpp

HDF5 handler and HDF4 handler Improvements

• Continued general migration to C++11 coding norms.
• Added the DAP4 Coverage support to the CF option of the HDF5 handler.
• Improved the CF option’s dimension name handling so that it can successfully handle  NSIDC SMAP polar grid level 3 products.

Version 3.20.10 for Hyrax 1.16.5

Requires libdap-3.20.9

DMR++ Improvements

• Added support for the HDF5 filter Fletcher32 to the dmr++ creation and
  processing code.

• Implemented lazy evaluation of dmr++ files. This change greatly
  improves efficiency/speed for requests that subset a dataset that
  contains a large number of variables as only the variables requested
  will have their Chunk information read and parsed.

• Added version and configuration information to dmr++ files built using
  the build_dmrpp and get_dmrpp applications. This will enable people to
  recreate and understand the conditions which resulted in a particular
  dmr++ instance. This also includes a -z switch for get_dmrpp which
  will return its version.

• Performance improvement: By patching Chunk::add_tracking_query_param() so
  that it doesn't do anything if no parameter is submitted we eliminated a
  very time costly regular expression evaluation that was being performed
  during the read operation for every Chunk. This improved read performance
  by a factor of 2-3 orders of magnitude!

Handler Updates

• Added new netcdf_handler configuration parameter:
  NC.PromoteByteToShort
  Which, when set to true will cause signed 8-bit
  integer values to be promoted to Int16 (Because the DAP2 data
  model does not support signed 8-bit integers)

• By default the NetCDF Fileout feature will ship with
  FONc.ClassicModel=false

• Added new configuration option to the HDF5 handler:
  EnableCFDMR=true which will allow the generation of CF compliant DMR output.

General Housekeeping

• Code organization changes to facilitate development
  and compilation on different platforms.
  • Moved functions into modules directory
  • Improved Test scripts to handle version string updates.
  • Changed libdap include paths to "libdap/*.h" to prevent
    include file collisions.
  • Started migrating to C++11 regex implementation (away from
    older less efficient GNU implementation) This is controlled
    by a compile time switch for now.

Version 3.20.9 for Hyrax 1.16.4

Requires libdap-3.20.8

Release Notes

NGAP & DMR++ Improvements

• Trusted cmr

  • Modifying things into shape to use http::url instead of std::string
  • Replaced strings with http::url objects
  • Moved AllowedHosts to http
  • Fixed implementations of http::url::is_expired()
  • Switch RemoteSource constructor to shared_ptr
  • Changed the way that http::url interprets no protocol urls/
  • Fixed concurrency issues in EffectiveUrlCache

• Corrected usage statement for get_dmrpp

• Handle the "missing data" files in the NGAP system

• Update NgapApiTest to reflect changes in CMR holdings.

• Dropped useless call to Chunk.inflate() and added a state check to protect us
  from a CONTIGUOUS variable that is marked as compressed.

• Rewrote read_contiguous() with std::async() and std::future dropping the SuperChunk idea.

• First implementation of the new restified path with two mandatory and one optional path components.

DAP4

• DAP4 doesn't support DAP2 Grid. The code that handles the DAP2 Grid coordinates
  will cause some DAP4 coordinate variables under different groups to be ignored.
  So this fix ensure the code NOT to call the code that handles the DAP2 Grid coordinates
  for the DAP4 case.

General

• Added GitHub Actions to bes.

• Stop parser EffectiveUrl resolution activity

• Fonc refactor

  • Added support for streaming netCDF3 files that do not contain Structures
  • Fix a small memory leak in the history attribute code at the transmitter
  • Add history attribute is added to dap4
  • Add NC.PromoteByteToShort=true in the configuration file. This makes it consistent with nc.conf.in
    Also it make sure the value of signed 8-bit integer to be correctly represented in DAP2
  • Remove unused getAttrType function from FONcArray.cc
  • Dropping throttle from Fonc_transmiter

• Added throttle to BESUtil::file_to_stream()

• Ensure the data value correctness for the classic model

  • When encountering a mismatch for the data type mapping mismatches,an error will be generated
  • For the classic model, ensure the _fillvalue datatype to be the same as the variable datatype

• Server handler refactor

• Fixing duplicate CF history entries

• Perform comprehensive check of datatype match and implementation of ensuring _FillValue attribute
  type the same as the variable type

• Added new implementation of temp file transfer code for fileout_netcdf

• Added config param Http.UserAgent

• Fixed netCDF-4 and compression information is missing when A DAP2 grid maps to three netCDF variables

• Adds call to the ftruncate() function in the update cache files activity, unit tests for string replace_all()

Version 3.20.8 for Hyrax 1.16.3

Requires: libdap-3.20.7

Release Notes

NGAP & DMR++ Improvements

• The dmr++ production chain: get_dmrpp, build_dmrpp, check_dmrpp, merge_dmrpp, and
  reduce_mdf received the following updates:

  • Support for injecting configuration modifications to allow fine tuning of the
    dataset representation in the produced dmr++ file.
  • Support for HDF5 COMPACT layout data.
  • Optional creation and injection of missing (domain coordinate) data as needed.
  • Endian information carried in Chunks
  • Int64 support
  • Updated command line options and help page.

• Improved S3 reliability by adding retry efforts for common S3 error responses that
  indicate a retry is worth pursuing (because S3 just fails sometimes and a retry
  is suggested).

• Improved and more transparent error handling for remote access issues.

• Migrated the service implementation making parallel requests using multi-cURL to
  the c++11 std:async and std:future mechanism.

• Added caching of S3 “effective” URLs obtained from NGAP service chain.

• Implemented support for EDL token chaining.

• New implementation of ngap restified path parser that is (almost) impervious to
  the the key value content in the path.

• Implemented the SuperChunk optimization for mass acquisition of required, consecutive
  chunks.

BALTO

• Updated JSON-LD content of the server’s Data Request Form pages so that it is (once
  again) in keeping with the (evolving) rules enforced by the Rich Results page which
  is part of Google’s Dataset Search

DAP4

• AsciiTransmit supports DAP4 functions
• Group support in fileout netcdf-4

General

• End Of Life for CentOS-6 Support - It’s been a long road CentOS-6, but NASA has
  given us the OK to drop support for you just days before your nominal end of
  life. On to the next std::future.

• Dropped the “longest matching” Whitelist configuration key in favor of a multiple
  regular expressions configuration using the new AllowedHosts key.

• Consolidation of internal HTTP code and caching for all services. This means more
  consistent behavior and error handling everywhere the server has to reach out for
  something.

• Introduced log message types: request, error, info, verbose, and timing which all
  log to BES.LogName/. Each type is identified in the log and has a “fixed” format
  that can be reliably parsed by downstream software.

• Updated GDAL handler.

• SonarCloud and /or Snyk is now a blocking step for all Hyrax component PRs

• Our Docker images have been updated to utilize ncWMS-2.4.2 which is compatible
  with current Tomcat security measures. This means ncWMS2 is working again…

• Dynamic Configuration - This feature is currently a proof-of-concept idea and is
  disabled with a compiler macro. To become an actual feature it will need to be
  implemented in a much more thoughtful and efficient manner. Which we will be happy
  to do so if there is sufficient interest!

Version 3.20.7 for Hyrax 1.16.2

Requires: libdap4-3.20.6

Release Notes

• Hyrax can generate signed S3 requests when processing dmr++
  files whose data content live in S3 when the correct credentials
  are provided (injected) into the server.

• For configurations that require Hyrax to authenticate to access various
  remote resources, Hyrax can now utilize ~/.netrc (or a netrc file may
  be specified in site.conf) and Hyrax will use the appropriate credentials,
  if they are present.

• The get_dmrpp script for dmr++ production has been enhanced with better
  configuration options and built in QC tests for inventory along with
  a simple value test.

• The get_dmrpp script will default to using a template string for the value
  of the dmrpp:href URL when a URL is not provided with a -u switch. This
  allows the URL to be injected later.

• We fixed a problem with fill values by converting the FillValue type to
  the data type of the associated variable.

• Fileout NetCDF has a new configuration key, FONc.NoGlobalAttrs that can
  be used to suppress global attributes.

hdf5_handler

• The CF option support is enhanced.

• Added support for correctly generating dmr++ files.

• Enhanced support to handle netCDF-4 like HDF5 files and general two
  dimensional latitude and longitude files.

See the modules/hdf5_handler/ChangeLog for details.

Version 3.20.6 for Hyrax 1.16.1

Released in conjunction with libdap4-3.20.5

Release Notes

Performance Improvements

• No DAS construction for data responses.

Bugs Fixed

• Memory leak fixed and unneeded copy removed in awsv4 code
• CovJson patch.
• ncml_handler: When an NCML aggregation named a dataset that the server could not read... crsh. Fixed..
• Server cache stabliszation
• Added (alpha) support support for server authentication credentials:
• • Environment injection support for besstandalone and besd
• • A credentials configuration file for multiple sets, in which credentials are asscoaied with URL prefixs and multiple creds can be defined in a bes.conf environment.
• Combined Logs: The server can now be configured to add the OLFS request log content to the BES log.
• hdf4_handler: Fixed memory leaks
• hdf5_handler: Fixed memory leaks

Version 3.20.5 for Hyrax 1.16.0

BES version 3.20.5

Requires:
libdap version 3.20.4

Improved and fixed bugs in the the CovJSON response format
(contributed by Corey Hemphill hemphilc@oregonstate.edu).

Added support for c++11 and c++0x builds including tests in
configure that look for those options and use c++11 over c++0x.
This works with the stock compiler on CentOS 6 .

We have added much more support for S3. Data can now be served
from S3 without transferring whole files (Hyrax supports subset
in-place) and without reformatting the data. This version of Hyrax
supports configuration of data already in S3 as well as files
still on spinning disk. See the README.md file in
bes/modules/dmrpp_handler/data

Performance improvements and bug fixes to the XML parsers.

Added a fix from Aron.Bartle@mechdyne.com.
Netcdf file responses were not compressed when they should have been
if the dataset had Structures. Fixed by patching FONcStructure as Aron
suggested.

We have added experimentatl support for STARE (Spatio Temporal
Adaptive-Resolution Encoding) as part of our work on NASA ACCESS
grant 17-ACCESS17-0039.

We have added support for Dataset crawler/indexer systems that
use JSON-LD as part of our work on NSF grant #1740704.

The FreeForm handlers now support regular expression matching
when looking for format file.