Proposal for Generated Filter Code

[DennisHeimbigner]

Proposal for Generated Filter Code

I am starting work on a new netcdf utility that takes a simplified filter specification and uses it to generate a complete HDF5 filter wrapper [2] plus an NCZarr filter wrapper.

The raison d'etre is that the process of building an HDF5 filter wrapper [2,3] from scratch is complex, time-consuming and error prone. Using a code generator is likely to simplify this process. At the very least, it will produce base code that a filter builder can modify to build the desired wrapper.

This program is analogous to, say, the yacc parser generator that converts an annotated BNF to a full blown parser.

What I need: I have a simple prototype working, but I need some community input on this idea. Would anyone use it? Is the proposed specification (Appendix A) reasonably simple to construct?

Specification Overview

The filter specification is written in JSON, although it is highly stylized. It was derived from the NumCodecs [4] format but with significant extensions to support the Netcdf-4/HDF5 wrapper format.

A couple of visible extensions with respect to JSON are:

1. Single line comments are supported beginning with '#'.
2. An alternate string delimiter is provided using the '`' character; chosen because occurrences of that delimiter in C code is very uncommon.

The basic specification is a JSON dictionary with very specific keys that are used to control code generation.

A draft example for specifying the zstandard filter wrapper is shown in Appendix A. The various dictionary keys provide filter information.

• "id" -- specifies the NumCodecs name (Zstd) and the HDF5 assigned
  identifier (32015); it also specifies an alternate preferred name.
• "parameters" -- a dictionary whose keys are the parameter names
  as specified by NumCodecs, and the value is a keyword
  indicating the type of the corresponding parameter.
  The allowable types are "integer" or "float". or an enumeration
  (not described here).
• "initialize" -- the value is a piece of code to initialize the filter before use.
• "finalize" -- the value is a piece of code to shutdown the filter after all use is complete.
• "prefix" -- arbitrary code to insert at the front of the filter wrapper; typically used to include filter library specific headers.
• "suffix" -- arbitrary code to insert at the end of the filter wrapper; typically used to include filter library specific utility functions.
• "encode" -- a function name plus the code for a user-provided function to invoke the filter's encoding/compression capability; this has a very specific signature.
• "decode" -- a function name plus the code for a user-provided function to invoke the filter's decoding/decompression capability; this has a very specific signature.

References

[1] HDF5 Filter Specification

[2] Registered HDF5 Filter Plugins

[3] User Contributed Filters

[4] NumCodecs

Appendix A. Zstandard Draft Example

{
    "id": {"zstd": 32015, "preferred": "zstandard"},
    "parameters": [{"level": "integer"}]
    "encode": ["name": "zstd_compress",
               "code": # The signature is standardized
        `
        size_t zstd_compress(size_t srclen, void* srcbuf, size_t* dstlenp, void** dstbufp, size_t cd_nelmts, const unsigned int* cd_values)
        {
            int ret = NC_NOERR;
            size_t dstlen;
            void* dstbuf;
            dstlen = (size_t)ZSTD_compressBound(srclen);
            if(ZSTD_isError(dstlen)) {ret = NC_EFILTER; goto cleanup;}
            /* Prepare the destination buffer. */
            if((dstbuf = malloc(dstlen))==NULL) {ret = NC_ENOMEM; goto cleanup;}
            dstlen = ZSTD_compress(dstbuf, dstlen, srcbuf, srclen, /*level*/cd_values[0]);
            if(ZSTD_isError(dstlen)) {ret = NC_EFILTER; goto cleanup;}
            if(dstlenp) *dstlenp = dstlen;
            if(dstbufp) *dstbufp = dstbuf;
        cleanup:
            return dstsize;
        }`],

    "decode": ["name": "zstd_decompress",
               "code": # The signature is standardized
        `...`]
    "prefix": `...`,
    "suffix": `...`
}

[edwardhartnett]

I think this is very much in line with ncgen, so fits as a netcdf-c utility.

I will certainly try this out next time I need to wire in a filter.

[lucasvr]

Hi, @DennisHeimbigner! Thanks for putting together this proposal. I have a couple of questions:

1. How do you intend to invoke the "finalize" callback? HDF5 filters don't really know when an application has finished feeding data or reading data from a compressed dataset. At best, one could register an atexit handle that would be called when the process exits, but I'm afraid that's not going to provide what you're looking for.
2. I notice that users often struggle to determine a reasonable chunk size for the datasets they're about to compress. Is your filter specification going to request that information from the user or are you planning on determining a good enough chunk size on-the-fly?

[DennisHeimbigner]

1. The filter finalization is invoked by the nc_finalize function. Currently that function must be explicitly invoked by the user, but we have experimented with attaching it to he atexit() functionality of Windows and Linux.
2. The filters are independent of the user-chosen chunksize. So it will not be part of his effort. As an aside, I note that we have a grant proposal that, among other things, helps users choose chunk sizes.

[lucasvr]

Got it. I thought you wanted to invoke the finalization callback upon a call to H5Dclose() or similar. Thanks for the clarification.