Composite/array values in string valued global attributes

[benjwadams]

Hi, I'm getting a number of questions on whether string valued global attributes such as "references" can be composed of multiple values in https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#description-of-file-contents. Should such attributes only consist of one value, or are two or more values considered acceptable as well?

Referencing issue (among others):

ioos/compliance-checker#1093

[JonathanGregory]

Do you mean, can they be arrays of strings?

[benjwadams]

Do you mean, can they be arrays of strings?

Yes.

My understanding if I have read the NetCDF data model UML correctly is that attributes are arrays anyhow: https://docs.unidata.ucar.edu/netcdf-c/current/netcdf_data_model.html

Assigning dataset.some_attribute = ["value"] and then querying dataset.some_attribute will return "value" instead of a list in NetCDF4 Python. Obviously, that is library specific, but it occurs from time to time.

Other than the common case above which is a single element, arrays of two or more elements allowed in these attributes?

[JonathanGregory]

I don't remember that we have considered this question. The text was formulated before strings were introduced. Arrays of strings previously could only be represented by two-dimensional character arrays. The document does not mention that two-dimensional character arrays might be expected in these global attributes, so I guess that no-one thought they would be needed. That's not the same thing as prohibiting them, but I imagine that existing programs which access these attributes will expect them to contain a single string. If the program was written using netCDF4 Python, it would expect a string to be returned, and receiving a list of strings would probably lead to some error.

Therefore I'm inclined to think that we should recommend that these attributes (history, references, etc.) not to be arrays of strings, or multidimensional character arrays, unless there is a persuasive need for them - is there? The text notes that you can embed newlines in them, in fact recommends so when appropriate. What do you and others think?

[taylor13]

I agree that we should explicitly say that when a CF attribute value is supposed to be a string, it should not be an array of strings. Of course, a compelling use case might justify relaxing this rule, but, as Jonathan noted, that would likely break some existing software.

[ChrisBarker-NOAA]

Agree on all counts -- there's a lot of precedent for using strings to represent multiple items via whitespace, delimiters, etc. As long as you can embed newlines, then there should be no need for an array.

[DocOtak]

Pretty long discussion in cf-convention/cf-conventions#141

The discussed use cases were for the history attribute, anything in CF that is "space separated" e.g. coordinates, flag definitions, etc...

[pp-mo]

FWIW my understanding of the CF spec is indeed that all attributes essentially are a 1-D "array" of values, but this only "somewhat like" an array, since it is 1-dimensional at most.
In the Python interface, at least, a single element is also treated as a scalar
-- i.e. crudely, it reads and writes as "x" not ["x"]. And if ["x"] is written, "x" will be read back.

However, my further understanding is that attributes containing multiple strings are not fully supported at least in the Python interface.
From experiment, string array attributes are entirely unsupported for NetCDF 3 files
-- that is, you get an error if you attempt to create one.
So, it has long been practice to encode multiple strings as a single "\n"-separated character array.

Also from experiment, multi-element string arrays can now be assigned to attributes, but these are encoded as variable-length strings, not an array of 'ordinary' strings. So the datatype of a single string and a string array are effectively different.
( The python interface now also has "setattr_string", which is specific to the variable-length string type. )
For demonstration:

>>> ds = nc.Dataset("tmp.nc", "w")
>>> ds.setncattr("q", ["one string"])
>>> ds.setncattr("a", "abc def\nfgh")
>>> ds.setncattr("x", ["y", "z"])
>>> ds.setncattr_string("a2", "abc def\nfgh")
>>> ds.setncattr_string("x2", ["y", "z"])
>>> 
>>> print(repr(ds.getncattr("q")))
'one string'
>>> print(repr(ds.getncattr("a")))
'abc def\nfgh'
>>> print(repr(ds.getncattr("x")))
['y', 'z']
>>> 
>>> ds.close()
>>> os.system("ncdump -h tmp.nc")
netcdf tmp {

// global attributes:
		:q = "one string" ;
		:a = "abc def\nfgh" ;
		string :x = "y", "z" ;
		string :a2 = "abc def\nfgh" ;
		string :x2 = "y", "z" ;
}

I have almost no experience of other language APIs, but I can see that in the C interface, nc_put_att_string and nc_get_att_string pass a "** char", analogous to the "* int" of nc_put_att_int and nc_get_att_int. And we are instructed to use nc_inq_attlen to determine how many elements there are.
So, that does not rule out multiple "regular" strings, and possibly that limitation is confined to the Python library.
Possibly, also, this was essentially different in the NetCDF3 API ?

HTH, and I'd be keen to hear the situation in other languages from someone less Python-centric.

[benjwadams]

Also from experiment, multi-element string arrays can now be assigned to attributes, but these are encoded as variable-length strings, not an array of 'ordinary' strings. So the datatype of a single string and a string array are effectively different.
( The python interface now also has "setattr_string", which is specific to the variable-length string type. )
For demonstration:

Yes, this issue is more or less about > 2 element variable length strings. While the question mentions global attributes, it nonetheless should be extended to cover places where string/char attributes in CF are present in general.

[DocOtak]

@pp-mo what you are seeing there are differences between the netCDF classic and the netCDF enhanced data models. The classic data model has no support for variable length data types (aka strings). The only slightly unusual thing I've seen that might be python specific, is the netCDF4 python library will force the data type of attributes containing character arrays that have any chars outside the ASCII code point range into a string vlen type rather than a char array.

[JonathanGregory]

Like for the global attributes, we don't have statements in the convention about arrays of strings for variable attributes, since they didn't exist when much of the text was written. The CF string-valued attributes are all expected to contain a word or a blank-separated list of words. I'm not aware of any reason for needing to use an array of strings, and this would certainly not work with some existing software.

From the above discussion and cf-convention/cf-conventions#141, I'm inclined to think we should insert explicit statements in the convention text and correspondingly in the conformance document to say that any CF string-valued attributes must be either a 1D character array or a scalar string. Would anyone disagree with that? As with all aspects, this could be reconsidered in future.

[taylor13]

Consistent with my earlier comment, I support Jonathan's comment immediately above.

[DocOtak]

Arrays of vlen strings as attributes have been a thing in netCDF for about 16 years now. It has been 6 years since cf-convention/cf-conventions#141 was started, and 4 years since that conversation went dormant. I think the future to consider supporting these types of values is now (or soon). For my own work, having to parse some CF custom string syntax (see e.g. cell_methods) or mangling my flag definitions has caused more issues that just dealing with features that have been in netCDF itself for my entire professional career. I'd support keeping the Conventions attribute as a 1d char array attribute, since that is what you would need to read to know if you might encounter more "advanced" features in a file.

I guess the above is just asking, if not now, when?

[ChrisBarker-NOAA]

Hmm -- is there any talk of CF 2.0?

In which we could start expecting "modern" netcdf, and stop supporting old stuff from COARDS (at least when talking about it in the docs, etc?)

We wouldn't want to do any major breaking changes, but a chance to remove some of teh cruft of the past would be nice, and then we wouldn't need to have these debates ....

[DocOtak]

My feeling is that CF 2.0 gets talked about in a "one day" sense that just stops the conversation. In CF 1.11 right now netCDF4 groups and the vlen string dtype for variables are supported so it's already "modern" in some ways. Groups can be especially breaking if programs aren't expecting them. I know when I first started working with CF, outdated (dare I say false) statements like "dtype x aren't supported in netCDF" that used to be in the document made me suspicious that the conventions were not being updated or maintained.

Aside, I think for a CF 2.0, it would be neat to break it up into different parts that are "independent" which are then opt into via the Conventions attribute: e.g. "CF-2.0 CF-DSG" for a data file that is following the "core CF conventions" with the discrete sampling geometry extension or perhaps some externally defined standard like "CF-2.0 CF-Radial" for the radar data.

[JonathanGregory]

@DocOtak pointed out that it's been six years since cf-convention/cf-conventions#141 was started. I hope we can finish it now! I'm on a mission to bring ancient issues to conclusion, admittedly rather slowly.

I propose that we split the preamble of 2.6 on Attributes into two paragraphs, since it's quite a long paragraph, with a new sentence to begin the second paragraph, shown in bold in the following. The rest is all existing text. Also, I show in italic a sentence which I have moved from the very end to near the start and updated it to recognise there's now more than one list.

This standard describes many attributes (some mandatory, others optional). See Appendices A "Attributes", F "Grid Mappings", and K "Mesh Topologies" for lists of attributes defined by this standard. A file may also contain non-standard attributes. Such attributes do not represent a violation of this standard. Application programs should ignore attributes that they do not recognise or which are irrelevant for their purposes. Conventional attribute names should be used wherever applicable. Non-standard names should be as meaningful as possible. Before introducing an attribute, consideration should be given to whether the information would be better represented as a variable. In general, if a proposed attribute requires ancillary data to describe it, is multidimensional, requires any of the defined netCDF dimensions to index its values, or requires a significant amount of storage, a variable should be used instead.

Any global or variable string-valued attribute described by this standard may be stored either as a netCDF scalar string or as a netCDF 1D character array; arrays of strings are not allowed for CF attributes. When this standard defines string attributes that may take various prescribed values, the possible values are generally given in lower case. However, applications programs should not be sensitive to case in these attributes. Several string attributes are defined by this standard to contain "blank-separated lists". Consecutive words in such a list are separated by one or more adjacent spaces. The list may begin and end with any number of spaces.

In section 2.6 of the conformance document, I propose that we insert a new requirement: String-valued attributes defined by CF must be scalar strings or 1D character arrays; arrays of strings are not allowed.

Enough support has already been expressed for making this change in principle. What do you all think about the above proposals?

[ChrisBarker-NOAA]

Could we compromise by: ...

I would rather just keep it simple, and not allow arrays of strings -- it's really hard for me the see the benefit.

But I won't put up a fight if others really want them.

One comment on that -- it may be unique to Python, but one of the major still remaining type issues in Python is that an atomic string is also a Sequence of strings (i.e. a Sequence of length-one strings) -- this means you sometimes have to write convoluted code to make sure you don't confuse them -- so I'd really rather that confusion weren't an option when reading CF data.

[sethmcg]

Late to the party, but I am in agreement with the general stances held by @DocOtak and @larsbarring. "Older software doesn't support it" is a chicken-and-egg problem, where nobody uses the feature because it isn't supported and it's not supported because nobody uses it. It's a valid consideration, but I think we should give it very limited weight.

We don't have an urgent need for string arrays at this moment, but I think we are on the verge of wanting them very badly. Whitespace- or newline-delimited free text attributes were a good solution for the early days when netCDF file histories and suchlike were simple. As time has progressed, however, we have started generating more complex derived data products from longer processing chains, and have started recording more and more provenance metadata to document the it. As a result, the "stuff everything into a single string and separate the elements with whitespace/newlines" approach is getting uglier and uglier, and I think we're not far from it becoming unmanageable in some cases.

Over in discussion #305, we've been talking about how to represent climatologies and other statistical processing, and while using a single delimited string works, I think that a solution that used string arrays would dramatically improve human interpretability and reduce the potential for confusion and error. Likewise, in #244, we're converging on a solution for the localization of attributes in different languages that is effectively creating an ad-hoc array of attribute strings by appending a suffix to long_name. It'll work, but it creates a messy namespace. (So that's a reason for making it an array.)

I think the compromise position proposed by @JonathanGregory is a good one; the default is that string arrays are by prohibited for attributes with prescribed values and allowed for free-text attributes, but there may be exceptions (such as long_name (for the moment, anyway)), and we give a recommendation with respect to the potential impacts on older software.

[ChrisBarker-NOAA]

"Older software doesn't support it" is a chicken-and-egg problem

well, yes, it certainly is :-(

However, it's not just that that makes me wary of this -- I'm wary because it's a bit tricky to correctly write code that will work with either a single string or an array of strings, particularly if all the test case a user is looking at are only one of those.

I think we are on the verge of wanting them very badly.

OK -- fair enough so there's a need.

What I think would be best would be a clear line, e.g.:

"This attribute is always an array of strings -- even if it is a length-1 array" (for CF version > something)

However, I'm not sure how we could get from here to there without breaking even more workflows....

but there may be exceptions (such as long_name)

Ahh yes -- perhaps we DO define, for every controlled attribute, whether or not it can be (or must be?) an array of strings.

And can we disallow multidimensional character arrays please?

[sethmcg]

Ahh yes -- perhaps we DO define, for every controlled attribute, whether or not it can be (or must be?) an array of strings.

And can we disallow multidimensional character arrays please?

I am in favor of both of those ideas.

[DocOtak]

Are 2D (or more Ds) even allowed by the netCDF API itself?

[JonathanGregory]

Dear @ChrisBarker-NOAA

My suggestion is that we say in the preamble of Sect 2.6 "Attributes" that string-valued attributes must not be arrays of vlen strings except where otherwise stated, and in Sect 2.6.2 that comment history institution references source title are allowed to be arrays of strings, as an exception to the general rule. We could add other exceptions if the benefit justifies the cost in future. It would also be sensible to indicate the attributes that can be arrays of strings in Appendix A. Yes, I agree, we can prohibit 2D character arrays for attributes. They have not previously been needed, so there's no backwards incompatibility. We do already allow 2D character arrays as string-valued auxiliary coordinates, though, serving as arrays of strings.

Could you agree to this?

Cheers

Jonathan

[ChrisBarker-NOAA]

That all sounds good to me -- thanks!

Though one thought: should we be more cautious?

Maybe not include title ?

All the rest you list I can see that as a file makes it way through a workflow, there would be a need to add another "entry" at each step.

[ChrisBarker-NOAA]

Is it off the table to require an array of strings (even if of length-1) for these?

If I were starting from scratch, that's what I'd do.

Or maybe suggest that a length-1 array be used for these?

[JonathanGregory]

@DocOtak is right that netCDF itself does not allow multidimensional attributes. NUG says "Another difference between attributes and variables is that variables may be multidimensional. Attributes are all either scalars (single-valued) or vectors (a single, fixed dimension)." That's useful to know, thanks - it removes one of @ChrisBarker-NOAA's concerns. 😃

What do others (@benjwadams @DocOtak @sethmcg) think of not allowing title to be a vector of strings, as @ChrisBarker-NOAA suggests? title is one of the attributes defined by the NUG, originally as "A global attribute that is a character array providing a succinct description of what is in the data set." (The Appendices of the current NUG are broken links, so I can't check what it says now.)

In answer to your last question, Chris, I don't think we can prohibit scalar strings; that would be even more backward-incompatible than allowing arrays of strings. We could add explicit advice to software-writers, as well as data-writers, to be aware that the affected attributes could be either scalars or arrays.

[DocOtak]

Restricting title to not be a vector of strings is probably fine.

[ChrisBarker-NOAA]

I don't think we can prohibit scalar strings; that would be even more backward-incompatible than allowing arrays of strings.

Fair enough -- I'll shut up now.

We could add explicit advice to software-writers, as well as data-writers, to be aware that the affected attributes could be either scalars or arrays.

We certainly. should do that. But I'd take in one step farther for data-writers: suggest that those attributes be written, initially, as len-1 arrays of strings, rather than a single string. The idea being that they may well be added to in the future.

[davidhassell]

Jonathan wrote:

My suggestion is that we say in the preamble of Sect 2.6 "Attributes" that string-valued attributes must not be arrays of vlen strings except where otherwise stated, and in Sect 2.6.2 that comment history institution references source title are allowed to be arrays of strings, as an exception to the general rule. We could add other exceptions if the benefit justifies the cost in future. It would also be sensible to indicate the attributes that can be arrays of strings in Appendix A. Yes, I agree, we can prohibit 2D character arrays for attributes. They have not previously been needed, so there's no backwards incompatibility. We do already allow 2D character arrays as string-valued auxiliary coordinates, though, serving as arrays of strings.

I would be happy with this.

[larsbarring]

This comment contains two independent points:

Firstly:
I wonder how much of this discussion is primed by a python-centric perspective. I get the feeling that is is the case (and not only in this particular discussion.) Nowadays I am almost only using python and I have to deliberately remove my python perspective when thinking about CF because I do not want see the CF Conventions to become python only. [If however this is community intention it needs to be specifically discussed.]

To my (not very deep) knowledge all relevant languages have the capability to test for variable/attribute type, so it should not be a problem to handle distinguish between and an array of chars or a scalar vlen string, and an array of vlen strings. In python type tests may not be very 'pythonic' but that does not carry much weight to my mind.

Continuing down the python road, despite what I first argued for, @pp-mo mentioned that python do not distinguish between ["some text"] and "some text" that is saved to a netCDF file and then read back. If I have not misunderstood something this means that making all arrays of chars or scalar vlen strings into length-1 arrays of vlen strings as @ChrisBarker-NOAA suggests would not help much.

@sethmcg gave a couple of examples where arrays of vlen strings could be very useful, and wrote

We don't have an urgent need for string arrays at this moment, but I think we are on the verge of wanting them very badly.

I fully agree with this!

Secondly:
I have a hard time understanding why we should explicitly not allow (i.e. *prohibit) arrays of vlen strings. If these were allowed (but recommended against) for the attributes discussed above, this will only occur in new files following CF-1.X. As mentioned above, this may break existing workflows if they are not updated. But so may a number of other enhancements that have been introduced in recent versions without much discussion, or are under consideration. What it however does not mean is that allowing such arrays will somehow invalidate existing files.

[ChrisBarker-NOAA]

Agree about not using a Python-Centric perspective. But dealing with two possible types in a attribute strikes me as a harder to deal with, rather than easier, with a statically typed language such as C (or even more so, Fortran) -- but I'd like to hear from someone that writes tools to read CF files in other languages if they have any opinion on this.

[ChrisBarker-NOAA]

Continuing down the python road, despite what I first argued for, @pp-mo mentioned that python do not distinguish between ["some text"] and "some text" that is saved to a netCDF file and then read back.

Darn -- IMHO, that's a bug in the Python lib :-(.

I may test and submit a PR.

[DocOtak]

@ChrisBarker-NOAA

This might not be python specific. I managed to create, using hdf5, two netCDF4 files where one has a string attribute as a SCALAR and the other as a 1D array with a single element. I've included them in the attached zip should they be useful. You'll need to use h5dump to see the difference (or check the bytes), the cdl output of e.g. ncdump only differs in the filename.

files.zip

[sethmcg]

They look identical when read into R using the ncdf4 library.

[JonathanGregory]

Dear Lars

If [arrays of vlen strings] were allowed (but recommended against) for the attributes discussed above [comment history institution references source, and perhaps title], this will only occur in new files following CF-1.X. As mentioned above, this may break existing workflows if they are not updated. But so may a number of other enhancements that have been introduced in recent versions without much discussion, or are under consideration. What it however does not mean is that allowing such arrays will somehow invalidate existing files.

That is all correct, I believe. My proposal is to allow arrays of strings for those attributes, but recommend against them if the data might be read by software that doesn't support CF 1.>=X.

Best wishes

Jonathan

[davidhassell]

My proposal is to allow arrays of strings for those attributes, but recommend against them if the data might be read by software that doesn't support CF 1.>=X.

Do we need the recommendation conditional on software support, since we don't do that for other features for which we think some software might not support (e.g. groups and strings)? It wouldn't be a recommendation that could be checked, in any case.

[ChrisBarker-NOAA]

I agree -- there should be a blanket statement somewhere that certain features (put them in a list all in one place) are not compatible with older software. But we don't have to put that everywhere.

[larsbarring]

I think that we need to be a bit more specific than writing ".... software ...."? That is, isn't the key software component what version of the netCDF library as such is used, and which version of the netCDF data model this some software implements. Or did I miss something here?

Irrespective of this, the conversation has now diverged from the question posed in the initial post. Maybe we should continue in a new discussion with a better title?

[ChrisBarker-NOAA]

isn't the key software component what version of the netCDF library as such is used

key, yes, but I think tha's fairly obvious -- you can't use nc4 features without at an nc4 aware lib.

But client software also needs to be written to expect wither a single string of an array array of strings -- they have to be treated differently.

but yes -- not really the same topic.

[JonathanGregory]

Dear all

Originally I wrote something longer

Recommending that arrays of strings should not be used if software written before CF 1.12 might be used to process the data (because the software might not be able to interpret them, or might abort)

David commented

Do we need the recommendation conditional on software support, since we don't do that for other features for which we think some software might not support (e.g. groups and strings)? It wouldn't be a recommendation that could be checked, in any case.

It's true, it couldn't be checked, and we don't always make such remarks. For strings, we say in 2.2, "The string type is only available in files using the netCDF version 4 (netCDF-4) format," from which you can infer that it won't work for programs assuming classic netCDF. There isn't such a remark for groups, that I can find.

Perhaps a recommendation is too strong in this case. A recommendation would cause a warning to be emitted by the cf-checker. Nonetheless, I think the remark is worth making, as we did for strings. As a compromise on my compromise, I suggest that we don't include a recommendation in the conformance document, but we do state in the text that, because arrays of strings were not permitted in atributes before CF 1.12, software written before CF 1.12 might fail if it encounters them.

How is that? If we agree on this, I will open an issue to propose the text changes.

Cheers

Jonathan