Form data: Remove hidden values when submitting

[olemartinorg]

Description

The backend already has RemoveHiddenDataPreview, but this feature should be moved to app-frontend-react instead, and enhanced with a few improvements.

When the last known component bound to a path in the data model is hidden, we should:

1. Reset the data back to what we originally got when loading the data model from the backend. This will preserve prefill data, etc, but remove most data (as most data does not exist when we load the data model). We should also revert any effects of ProcessDataWrite, as they will either be re-applied or they'll possibly leave unwanted side-effects.
2. Remove entire rows from hidden repeating groups, not empty leaf values (see RemoveHiddenDataPreview with hidden repeating group sets row values to null instead of deleting rows app-lib-dotnet#321).
3. Preserve the data in-memory such that if the user un-hides the component the data will re-appear (and be sent to the backend again).

To demonstrate the point of this functionality, let's say pretend we have this form:

I have pets:
(x) Yes    ( ) No

My pets (repeating group):

| Name     | Type   | Age  |             |
|----------|--------|------|-------------|
| Fido     | Dog    | 3    | Edit Delete |
| Whiskers | Cat    | 1    | Edit Delete |

(Add new row)

And the data model looks something like this:

{
  "SF_someHelper": "someValue",
  "havePets": true,
  "pets": [
    {
      "SF_verified": true,
      "SF_type": "child",
      "name": "Johnny",
      "type": "human",
      "age": 5
    },
    {
      "SF_verified": false,
      "SF_type": "pet",
      "name": "Fido",
      "type": "dog",
      "age": 3
    },
    {
      "SF_verified": false,
      "SF_type": "pet",
      "name": "Whiskers",
      "type": "cat",
      "age": 1
    }
  ]
}

In this example, the repeating group in the UI is using the hiddenRow functionality to only show rows of SF_type = pet (SF in this context means shadow field). Let's pretend this persons children are fetched from official and reliable sources, but the pets are added manually (and are not verified). Verification and other shadow fields are added by ProcessDataWrite.

As soon as the user changes their selection to "No" for having pets, app-fronted-react should remove the two pets from the data model, and revert the data model to the original state (i.e. removing the shadow fields along with the pets if they were added by ProcessDataWrite, or, if the rows were already present in the data model when we loaded it, just remove
the values the user typed in, along with the ones we got from the backend). And if the user changes their selection back to "Yes", the pets should re-appear as if nothing happened.

The purpose of this feature is to make it possible to use hiding of components to make the form more dynamic, without having to worry about the data model getting out of sync with the UI. It is useful for validation, as validation code can make some more assumptions about the data model being in a somewhat valid state (i.e. not having pets when the user says they don't have pets), and it also becomes a privacy feature as the user can hide data they don't want to share with the receiver of the form.

Additional Information

• New validation extension app-lib-dotnet#369 (comment)
• RemoveHiddenDataPreview with hidden repeating group sets row values to null instead of deleting rows app-lib-dotnet#321
• https://github.com/Altinn/app-lib-dotnet/pulls?q=is%3Apr+sort%3Aupdated-desc+RemoveHiddenDataPreview+is%3Amerged

[olemartinorg]

Relevant discussion with examples of readOnly fields that should also 'reset' data back to prefill:

• https://altinn.slack.com/archives/C02EJ9HKQA3/p1705650576694759

[olemartinorg]

Relevant discussion about deleting uploaded files when the uploader itself has been hidden:

• https://altinndevops.slack.com/archives/C045EB3JA9X/p1706006244808469?thread_ts=1706006182.137749&cid=C045EB3JA9X

We could perhaps do that before we go to process/next, i.e. while the user is waiting.

[RonnyB71]

Relevant input, RemoveHiddenData also deletes data you don't want to remove. One example is a hidden id on a repeating group which doesn't provide any user value and hence is hidden, but should definitely be stored.

[olemartinorg]

Relevant input, RemoveHiddenData also deletes data you don't want to remove. One example is a hidden id on a repeating group which doesn't provide any user value and hence is hidden, but should definitely be stored.

Yup, I believe that's covered by the first point: Reset the data back to what we originally got when loading the data model from the backend. Meaning, we only reset the changes the user made. The example here was a Dropdown that is always hidden, but used to display a text value in the repeating group table, and if that's always hidden (but maybe the backend sets data on the repeating group) there is no user changes for us to reset. However, if the user actually deletes the whole row, we have no choice to just tell the backend the user deleted the row (by removing it from the form data). If the app developer does not want it to be possible to delete entire rows, they must disable the delete button (which is already an option for repeating groups).

[olemartinorg]

Removing fe-v4 label on this one, as it's not critical for that release.

[jhhssb]

Some additional information that we see in V4:
If you set "Remove hidden data" to ON, hidden prefill-fields (with value) that are not shown in the app (visible to the user) will not appear in the XML sent in. This is a blocking issue I think.

[olemartinorg]

@jhhssb Yes, that's a known issue with the current implementation (in the backend), and something we're planning to 'fix' with this suggestion to implement a solution (in the frontend instead). It's possible to work around the issue in apps, for example by duplicating data (when displayed in a purposely hidden component) or hiding the component using ruleHandler based ("older-style") dynamics (which the backend doesn't know about).

[StianVestli]

Jeg trudde den gamle dynamikken ikke var støttet i v4 @olemartinorg ?

[olemartinorg]

Den er fortsatt støtta - og vi har ingen konkrete planer om å fjerne støtten enda (men heller ikke særlig med forbedringer, utenom at helper-objektene i RuleHandler.js nå er valgfri i v4).

[olemartinorg]

Relevant (eksempel på noe som kan automatisk testes):

• https://altinn.slack.com/archives/C02EJ9HKQA3/p1712138823961559
• Ugyldig verdi på skjult felt gir uløselig feilmelding ved innsending av skjema #2011