Returning null values for required metadata attributes in Federation API

[asafieva]

This is a discussion related to Federation API returning null values for required metadata attributes. In components,  these are entities with names models.<...>.Metadata as models.subject.Metadata.
Even syntactically these definitions are OK with OpenAPI specification, semantically it is questionable. 

When these values are null it is unclear how we can search by these value using API endpoints search parameters. 

   models.subject.Metadata:
      type: object
      description: Metadata associated with a subject.
      required:
      - sex
<...>
      properties:
        sex:
          allOf:
          - $ref: '#/components/schemas/field.unowned.subject.Sex'
          nullable: true

We have multiple conversations about nullable+required metadata attributes including in Slack. This was one of the discussions with Clay's responses: https://uc-cri.slack.com/archives/C05GL22240Y/p1710528469367179

CCDI Hub Explore uses specific values as

• Unknown
• Not Reported
• Unspecified
• Not Allowed to Collect

If a value is not present in selected CDE's PVs, it is possible to request caDSR curators to add it.

Is it feasible avoiding null values for required harmonized attributes in API returned entities?

[asafieva]

Slack conversations related to this discussion:
https://uc-cri.slack.com/archives/C05FFPKLQ79/p1708455661414349?thread_ts=1707938351.836389&cid=C05FFPKLQ79
https://uc-cri.slack.com/archives/C05FFPKLQ79/p1694705453225319

[claymcleod]

There are a few points to consider here.

First, on the point of whether we need a sentinel value at all (whether that be null, false, -1, or -999) or whether we can simply omit the keys should be addressed.

In the API, we've taken the philosophy of "if a field is expected and not present, the absence of that field should be explicit to reassure users that the lack of data is not due to a bug in the API or the API being out of date". This is based on best practices of scaling and growing an API into the future. To illustrate the kinds of issues that can occur if we are not explicit in our omission of values, consider the issues that would occur if we decided to simply omit the keys when metadata was missing:

• When we add new metadata elements to, say, v1.1 of the API, clients will not be able to distinguish between the cases where (a) the information is missing because the client hasn't yet updated to v1.1, meaning that the metadata element is not yet supported and (b) the information is missing because it's simply not available. The most straightforward way to avoid that case is to explicitly annotate when fields are missing. Sure, we could require clients to check the server version, keep track of which metadata elements were introduced in what versions, and handle things accordingly, but that is quite a bit of hassle and will drive lower adoption of the API—especially when contrasted with the best practices of RESTful API design whereby all of the information needed to interpret a response is included within the response itself.
• Further, when filtering entities, how would a user specify that they want to select for values that are unset in this scenario? Without the introduction of a sentinel value like null, the omission of a filtering parameter key overloads the concepts of desiring to select entities where the field is unset vs. ignoring the field altogether when filtering. In this case, a sentinel value is required to distinguish these two scenarios. And, if it is required for query parameters to support a sentinel, it appears to me a better practice to also explicitly delineate these cases in the responses itself due to the consistency it introduces.

Now, as to the question of what the sentinel value should actually be, the case I would make is a pretty straightforward: the concepts we're wrangling with here are not new, and the designers of the language itself introduced null specifically for the purpose we're describing.

As described by the official JavaScript documentation:

The null value represents the intentional absence of any object value. It is one of JavaScript's primitive values and is treated as falsy for boolean operations.

Though one may exist, I cannot think of a compelling reason to contradict the language's chosen strategy here. Further, buying in to the language's type created specifically for this purpose has a few major benefits:

• Without any prior knowledge, clients (and users at large) are going to immediately understand what null means in this context, simply because this is the way the concept is handled at the language level. In fact, they're going to expect this, and not conforming to this standard is going to be something the API would have to actively fight against. In my opinion, not adding to the mental burden required to grok the API is a key consideration for whether it will be adopted at large.
• Existing clients that parse responses from OpenAPI specifications are also going to be built on the assumption that null will be used in these cases, meaning that you get the value of translating that concept to each language-specific construct where someone might implement a client for free.
• The value does not disqualify other totally valid values, such as -999, arbitrarily.

Selecting another other special value, such as false, -1, or -999, has downsides in each of these cases:

• Users are going to have to read the documentation and understand this special value's purpose.
• Any value that could have been added by external libraries handling null is going to be wiped out, and it will be the responsibility of the programmers writing the clients to remember to catch -999 in all cases. This is going to be a intermediate headache and a source for error.
• This value can no longer be used as a valid value in the API, even though the reason for that is arbitrary.

As has been discussed in the conversations on Slack, sometimes tools or dependencies don't properly support the specification: whether that be a third-party library not handling null correctly according to the OpenAPI specification or even clients that may not interpret the API we're designing here correctly. In these cases, my philosophy is that the tools need to be updated to be compliant with the spec, not the other way around. Anything else is simply not sustainable at scale: adopting dependencies that are not exactly compliant with the specifications they purport to implement or influencing the design of a specification to handle some quirk of a particular implementation is a slippery slope that can quickly degrade the effectiveness of the specification as a whole.

Last, I'd just like to address the idea of null versus the various values provided by the CDEs that sometimes overlap this concept. Natalia pointed this out in this comment:

CCDI Hub Explore uses specific values as

Unknown
Not Reported
Unspecified
Not Allowed to Collect

If a value is not present in selected CDE's PVs, it is possible to request caDSR curators to add it.

Is it feasible avoiding null values for required harmonized attributes in API returned entities?

As described by the "Incomplete Data and Nullity" section of the specification, the concept of null and Not Reported/Unknown/other permissible values do not usually represent the same thing and must be able to be communicated distinctly.

• For example, that sections describe how a participant answering Unknown to a demographics question represents something very different than, say, a source server not having access to what the participant answered for the question.
• Further, the definitions of Unknown/Not Reported are not consistent amongst CDEs—some group the two distinct situations above in the same Unknown category while others only address the former situation. The fact that CDEs are generally designed to communicate information collected and not to communicate concepts like missing data is a key reason why null needs to be introduced here.
• As such, the API needs a straightforward way to communicate all these different situations, and the changes we made in #81 do just that.

Related to this discussion, there was a comment in our discussion last week regarding how null and strings should be parsed separately when filtering in the API. I think that's a valid criticism, and I've introduced #98 to address that by now requiring that parameters be valid JSON values.