Cooling-off period and test files for CF conventions changes

[JonathanGregory]

Topic for discussion

Dear all

In conventions issue 369 I am reviving the proposal that we should merge CONTRIBUTING in the conventions repo and rules in the website repo, because both of them are about how to change the conventions. There are a couple of other open conventions issues relevant to that process: #151 on moderation, and #440 on labels. Those discussions would inform the content of the merged document.

Here, I am seeking opinions on two other aspects of the current rules which perhaps should be changed.

• The rules apparently require (at least) two three-week periods to pass with no comments for a proposal to be agreed. In practice, we often require only one, and I am not sure that two was really what the rules intended. I think one such cooling-off period is enough. It should begin when the moderator of the proposal judges that there is sufficient support (i.e. satisfying the rules) for whatever form the proposal has reached during the discussion, and when there are no outstanding objections. At that point, the moderator should state this position on the issue, and should invite further comments within the next three weeks. If there are none, it means consensus has been achieved and the proposal, as it current stands, is accepted.

• The rules require a test netCDF file to be presented when new conventions are added, in order for the proposal to be implemented in the conventions. We have had this rule, in some form, since the first release of CF twenty years ago, but as far as I recall it has never been enforced, and hardly ever mentioned in practice. In my opinion, the fact they we have always completely ignored this rule means that we have tacitly decided we do not need it. If enforced, it would add further indefinitely long delays to agreement. So far, there has only once been a flaw in a change made to the conventions, and it's not obvious that a test file would have made that flaw apparent. Therefore I suggest we delete this rule, and instead add a requirement in the constitution that the CF committee should ensure that there is at least one "reference" open-source software implementation of every aspect of the current CF conventions (not necessarily all aspects in the same software) publicly available within one year of each release of the conventions document. By "reference" I mean that the committee believes the implementation is trustworthy and has been tested on files containing all relevant CF features. If the implementation reveals that there is a flaw in the convention, we will follow the rules to rectify it, but experience shows that this is likely to be only a rare occurrence.

What are your views on these points?

Best wishes

Jonathan

[larsbarring]

Dear Jonathan,

Thanks for moving this forward! :-)) I have been thinking about these different issues, but never managed to wrap my head around them to form a consistent picture.

On the first point I fully agree.

On the second point I am more reluctant to delete the requirement altogether. But I see and agree with your argument that it may severely delay acceptance or simply put off people wanting to suggest an enhancement. On the other hand, as the conventions become more and more technically advanced and complex I think that the actual need to have test -- or reference -- files becomes increasingly visible (which obviously wasn't the case in the past). I believe there are at least one presentation at the upcoming CF2024 workshop that touch on this. Perhaps this particular point can be a topic for conversation.

Kind regards,
Lars

[taylor13]

Thanks for thinking about this. Regarding the second point, I think that tasking the conventions committee with responsibility for ensuring a reference application making use of a new convention is a realistic expectation. What if the year goes by and nothing happens? Do we rescind the convention change? Probably not. How does this differ from the original rule that we never enforced (except for replacing "before" with "within a year")?

[mwengren]

As one of the providers of a CF compliance checker software package, I'd just like to add example/test files are especially helpful to have in order to verify proper function of compliance tests.

The IOOS Compliance Checker has not fully implemented CF 1.9 yet, mostly due to not being able to obtain a file to test the Chapter 8.3 Lossy compression by coordinate subsampling rules with.

There's an open issue in our repo describing the steps to implement the conformance guidelines for Chapter 8.3 here, with a comment by @benjwadams mentioning the need for example files. So far we've come up emptyhanded in efforts to locate such files.

So, from our perspective, including test files alongside, or sometime after release of, new CF versions is beneficial for the community, however you decide the best way to encourage that is.

At least as far as CF checker support goes, test files will help to facilitate the development of one type of 'reference open source software implementation' you referred to above.

[JonathanGregory]

Dear @mwengren

Thanks for your comment. Since I started this discussion, the conventions committee has discussed it. On the test files, the committee took the view that for many changes to the conventions document a test file is not needed. A test file is not needed, for instance, if the change alters the text of the document but not the meaning of the convention. However, it should be part of the discussion about any proposed conventions change to decide whether a test file is needed for the particular issue concerned. If it is, it would have to be provided before the change was agreed (as the rules say).

Many changes to the conventions document add snippets of CDL to illustrate them, or CDL examples (which are in boxes and numbered). Most of these examples are not complete CDL files; often, for example, they show definitions of variables without data. If the CDL examples were made available as complete netCDF files, would that meet the need for test files,do you think?

Best wishes

Jonathan

[benjwadams]

Many changes to the conventions document add snippets of CDL to illustrate them, or CDL examples (which are in boxes and numbered). Most of these examples are not complete CDL files; often, for example, they show definitions of variables without data. If the CDL examples were made available as complete netCDF files, would that meet the need for test files,do you think?

Complete CDL files alone are sufficient, as properly formed ones can generate netCDF files. Some of the examples aren't quite valid CDL but can be made so with a couple minor modifications, e.g. enclosing the CDL examples with netcdf { }, so in my opinion, if you had a link to the individual CDL in the top right corner of CDL examples, it'd be helpful.

I also want to point out that the various CF checker implementations that are open source also often contain test files or code that generates netCDF in their unit tests, often derived from the conformance requirements documents. These could be adapted with some review to meet gaps in the existing NetCDF files for the current requirement to supply.

On the test files, the committee took the view that for many changes to the conventions document a test file is not needed.

I agree that it's not actually needed, but it makes CF checker implementation maintainers' work much easier and less prone to misinterpretation of the standard.