1.13.0 Appears to have introduced new SSA error behaviours

[perlboy]

Description

In the release of version 1.13.0 the following section was added on October 2021 by @JamesMBligh that was not present in the Register specification:

Additional statements regarding these endpoings:
During registration management requests, Data Holders MUST validate that the scope of access tokens provided includes cdr:registration
Registration requests and responses must conform to the specification in the DCR APIs section.
Any fields the Data Holder does not support MUST be ignored without error.
Registrations MUST only be updated via a PUT operation on the registration endpoint
POST and PUT operations MUST accept the SSA payload
Update (PUT) operations are to be used for one of two scenarios:

1. The client has updated their registration details on the CDR Register and updates this information to the data holder brands
2. A new version of the SSA has been released and the client updates this information to the data holder brands

This is despite Biza.io raising the potential of altering existing specification and the reassurance from the CDR API Stream that there was no such intent.

Despite these assurances the following statements have resulted in direct modifications using binding RFC2119 language to current implementations without notice:

1. Any fields the Data Holder does not support MUST be ignored without error
   Fields contained within the SSA, if ignored, would result in the incorrect outcome for the registration OR an invalid SSA. An example of this is the sector_identifier_uri field which was introduced as an optional field in July 2021 with no specific future dated obligation. By ignoring such a field a software product would be non-functional as the redirect uris would have multiple hostnames.
2. Update (PUT) operations are to be used for one of two scenarios:
   The documented scenarios do not handle the possibility of unknown scopes being introduced into the SSA and as such, as the scenarios are explicitly defined, many Holders currently default to rejecting SSAs containing malformed scopes.

Area Affected

Register Standards
Change Management regarding the Standards

Change Proposed

1. Reverse the presence of the statements and appropriately consult on their content
2. Clarify how, despite assurances and not for the first time, members of the DSB saw fit to introduce unvalidated and unannounced RFC2119 statements and provide guidance to the community on how this may be avoided in the future

[CDR-API-Stream]

I have referenced issue #443 as the impacts raised here overlap with the expectations set out in SSA change management.

[CDR-API-Stream]

This issue has been flagged as a candidate for MI 11

[CDR-API-Stream]

Let’s talk to each concern individually:
 

1. Any fields the Data Holder does not support MUST be ignored without error
Fields contained within the SSA, if ignored, would result in the incorrect outcome for the registration OR an invalid SSA. An example of this is the sector_identifier_uri field which was introduced as an optional field in July 2021 with no specific future dated obligation. By ignoring such a field a software product would be non-functional as the redirect uris would have multiple hostnames.

The previous registration design articulated this requirement as:

Any fields the data holder brand does not support are not persisted. However, registration responses must conform to the DCR API specification

The change in language was not intended to change the interpretation of the Standards.

However, this concern raises the following question.
Should the registration process enforce the downstream technical or business requirements of the CDR?

Using the sector_identifier_uri example, if a data holder does not support this field, after the obligation date has passed, the creation of a registration would still occur but the sector_identifier_uri would return unpopulated. If the data recipient requires the sector_identifier_uri, this registration does not facilitate the Standards, however, if sector_identifier_uri support is not required, data sharing can continue as normal.

There are various reasons that a data holder may not have met their obligation. They may be late, the ACCC can provide exemptions. There is benefit in having flexibility in the process if obligation dates are not met.

It’s also worth considering that if an older registration already exists but is not updated when there is the addition of the sector_identifier_uri support, the data recipient will not benefit from the sector_identifier_uri but the data holder will continue to share consumer data. Therefore, new registrations are impacted more significantly than current registrations.

If a data holder is to return an error on receipt of a registration request with fields they do not support, the data recipient will have no opportunity to request consumer data. The benefit of this approach is that any consequences of the lack of support are less likely to occur.

---

2. Update (PUT) operations are to be used for one of two scenarios:
The documented scenarios do not handle the possibility of unknown scopes being introduced into the SSA and as such, as the scenarios are explicitly defined, many Holders currently default to rejecting SSAs containing malformed scopes.

Unsupported scopes are scopes that are defined in the Standards, but the data holder does not support them.
This issue has been in discussion through MI 10 & 11 and is being addressed through issues #486, #488, and #507
 

---

3.     Change management process.

The merging of the Register design into the Consumer Data Standards never intended to alter the Register design. During the merge, the language was tightened up to follow the requirement-centric language of the Standards.

This exercise was a significant piece of work and this issue you raise is an unintended consequence.

As part of the maintenance iteration process, we have retrospectives to capture issues and opportunities for improvement. When the MI 11 retrospective occurs, we can raise this issue for discussion.
 

[perlboy]

Using the sector_identifier_uri example, if a data holder does not support this field, after the obligation date has passed, the creation of a registration would still occur but the sector_identifier_uri would return unpopulated. If the data recipient requires the sector_identifier_uri, this registration does not facilitate the Standards, however, if sector_identifier_uri support is not required, data sharing can continue as normal.

Except pairwise identifiers would (should...) be different when the Holder does add sector_identifier_uri support resulting in the ADR having a mass identifier change if they choose to use this support.

3.     Change management process.
The merging of the Register design into the Consumer Data Standards never intended to alter the Register design. During the merge, the language was tightened up to follow the requirement-centric language of the Standards.

Within 206 Biza.io specifically requested a clarifying statement to be added to the Standards to explicitly state no technical change was intended. The DSB even went so far as to state:

Due to the complexity of the merge process we will consult on the specific changes but the intention is that this will be done via specific technical changes in the standards-staging repository

And yet the additional statements were added without such consultation. Nothing in DP206 provided the DSB with a license to "tighten" language and yet it did it anyway.

As per the first sentence of this ticket:

In the release of version 1.13.0 the following section was added on October 2021 by @JamesMBligh that was not present in the Register specification

The Chair never actually approved the changes made as highlighted and now the consequences are playing out.

As part of the maintenance iteration process, we have retrospectives to capture issues and opportunities for improvement. When the MI 11 retrospective occurs, we can raise this issue for discussion.

Feedback on this has been given numerous times. The DSB needs to walk the walk rather than twisting its own processes to suit the favourable narrative at the time.

[CDR-API-Stream]

@perlboy Thanks for your input.

To help elaborate on the sector_identifier_uri component of the problem, I have put together the following are four scenarios where adoption or migration of the sector_identifier_uri may occur

#	Description	redirect_uris	Old sector_identifier_uri	New sector_identifier_uri	Prev sector_identifier	New sector_identifier	Impact
1	Valid adoption of new sector_identifier_uri	https://example.com/redirect_1 https://example.com/redirect_2	none	https://example.com/sector_identifier.json	example.com	example.com	Pairwise identifiers remain intact
2	Invalid adoption of new sector_identifier_uri	https://old_example.com/redirect_1 https://old_example.com/redirect_2	none	https://new_example.com/sector_identifier.json	old_example.com	new_example.com	Pairwise identifiers broken
3	Valid migration of sector_identifier_uri	N/A	https://example.com/old_sector_identifier.json	https://example.com/new_sector_identifier.json	example.com	example.com	Pairwise identifiers remain intact
4	Invalid migration of sector_identifier_uri	N/A	https://old_example.com/sector_identifier.json	https://new_example.com/sector_identifier.json	old_example.com	new_example.com	Pairwise identifiers broken

For these examples, sector is determined as per section 8.1 in OIDC Core 1.0.
Impacts are only relevant if the data holder is using the sector_identifier as an input into their pairwise calculations

The data recipient is in control to drive the adoption/migration through updating their registration via the PUT DCR operation. Currently, the standards DO NOT constrain the scenarios where identifiers are broken.

If an invalid adoption or migration were to occur, identifiers would break and the data the ADR has previously collected would lose referential integrity.

What we will attempt to answer this maintenance iteration is: Should the registration process enforce the downstream technical or business requirements of the CDR?

[CDR-API-Stream]

The analysis of this issue initially focussed on the sector_identifier_uri value and the consequence of this changing over time. However, the problem space is much more significant and therefore, analysis should include the technical and business impact of changes in the registration metadata.
 
The DSB will conduct further analysis on changes to registration metadata. This information will then be used to help inform further collaboration conducted through a future maintenance iteration. This issue will therefore be deferred from maintenance iteration 11.

[biza-io]

We note this issue has been added to the intended issues for discussion on MI12. Due to resource constraints associated with multiple future dated obligations and Energy sector activation Biza is unable to provide the further comments, evaluation or elaboration we feel will be necessary to appropriately resolve this issue.

As a consequence we request this issue is deferred until a later maintenance iteration.