Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/4908 json registration backend #4956

Draft
wants to merge 58 commits into
base: master
Choose a base branch
from

Conversation

viktorvanwijk
Copy link
Contributor

@viktorvanwijk viktorvanwijk commented Dec 19, 2024

Closes #4908

Changes

Add registration backend plugin that converts submitted form data to JSON and sends it to a specified API endpoint

Checklist

Check off the items that are completed or not relevant.

  • Impact on features

    • Checked copying a form
    • Checked import/export of a form
    • Config checks in the configuration overview admin page
    • Problem detection in the admin email digest is handled
  • Release management

    • I have labelled the PR as "needs-backport" accordingly
  • I have updated the translations assets (you do NOT need to provide translations)

    • Ran ./bin/makemessages_js.sh
    • Ran ./bin/compilemessages_js.sh
  • Dockerfile/scripts

    • Updated the Dockerfile with the necessary scripts from the ./bin folder
  • Commit hygiene

    • Commit messages refer to the relevant Github issue
    • Commit messages explain the "why" of change, not the how

Copy link

codecov bot commented Dec 19, 2024

Codecov Report

Attention: Patch coverage is 97.72727% with 2 lines in your changes missing coverage. Please review.

Project coverage is 96.66%. Comparing base (a950635) to head (9840e1e).

Files with missing lines Patch % Lines
src/openforms/registrations/contrib/json/plugin.py 95.74% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4956   +/-   ##
=======================================
  Coverage   96.66%   96.66%           
=======================================
  Files         761      766    +5     
  Lines       25943    26031   +88     
  Branches     3392     3399    +7     
=======================================
+ Hits        25078    25164   +86     
- Misses        601      602    +1     
- Partials      264      265    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@viktorvanwijk viktorvanwijk force-pushed the feature/4908-json-registration-backend branch from 0eaf0b3 to 1b81a5c Compare December 31, 2024 08:39
Plugin name did not make sense before
API endpoint is relative, name should reflect that
It should match the corresponding property of the serializer (which is 'service' and not 'service_select')
Not relevant anymore or duplicates
Wrong place and doesn't look great
…n plugin

Need to include all form variables listed in the options to a values dictionary
The content of the attachment is now added to the result
Now includes a check on the generated values dictionary for easy testing
This is a requirement of the plugin
The user might want to send it to just the api root of the service
Makes more sense to keep static variables dict close to where it's used
The previous size was not big enough to show all elements without scroll bars
Required for simulating a receiving service
It is sufficient to only pass the relative endpoint to the .post method
This check if redundant, as the for loop will never be executed anyway if the query set is empty
It is closely related to the serializer
This will raise a KeyError if a form variable is not available in the submitted data nor in the static variables. This is better than passing is silently
…e service

The instructions on how to start it is now in line with the other docker compose services
The configuration options could be saved without specifying any form variables to include. This does not make much sense. Added a minimum length of 1 for the form variables list in the serializer (Thanks to Robin for the hint)
Will check the connection to the configured service
…n editor

The field 'formVariables' is read-only, so need an extra const to modify its value. Note, this didn't show up in storybook testing, only in manual functional testing
…lugin

Just as an example of what will be implemented in #4980
* Use VCR instead of mock now
* Added check on the response
* Added check on attachment encoding
* Use VCR instead of mock now
* Add test for invalid response from API endpoint
@viktorvanwijk viktorvanwijk force-pushed the feature/4908-json-registration-backend branch from 1b81a5c to 707ef37 Compare January 2, 2025 10:12
@viktorvanwijk viktorvanwijk force-pushed the feature/4908-json-registration-backend branch from 8549ba7 to 9840e1e Compare January 2, 2025 11:31
src/openforms/conf/base.py Outdated Show resolved Hide resolved
src/openforms/registrations/contrib/json/models.py Outdated Show resolved Hide resolved
Comment on lines 24 to 64
def register_submission(self, submission: Submission, options: JSONOptions) -> dict:
values = {}
# Encode (base64) and add attachments to values dict if their form keys were specified in the
# form variables list
for attachment in submission.attachments:
if attachment.form_key not in options["form_variables"]:
continue
options["form_variables"].remove(attachment.form_key)
with attachment.content.open("rb") as f:
f.seek(0)
values[attachment.form_key] = base64.b64encode(f.read()).decode()

# Create static variables dict
static_variables = get_static_variables(submission=submission)
static_variables_dict = {
variable.key: variable.initial_value for variable in static_variables
}

# Update values dict with relevant form data
all_variables = {**submission.data, **static_variables_dict}
values.update(
{
form_variable: all_variables[form_variable]
for form_variable in options["form_variables"]
}
)

# Generate schema
# TODO: will be added in #4980. Hardcoded example for now.
schema = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"static_var_1": {"type": "string", "pattern": "^cool_pattern$"},
"form_var_1": {"type": "string"},
"form_var_2": {"type": "string"},
"attachment": {"type": "string", "contentEncoding": "base64"},
},
"required": ["static_var_1", "form_var_1", "form_var_2"],
"additionalProperties": False,
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can grab the static variables from the submission variables state too, and build up what needs to be processed before doing the processing.

state = submission.load_submission_value_variables_state()

all_values: JSONObject = {
    **variables_state.get_static_data(),
    **state.get_data(),  # dynamic values from user input
}
values_to_register = {
    key: value 
    for key, value in all_values.items()
    if key in options["form_variables"]
}

For the attachment processing, there's a bug if you have multiple file uploads - now the previous uploads get overwritten. So, depending on if the component is multiple True/False, you need to handle an array of uploads or just a single upload. I'd recommend setting up component-specific processing like in the Objects API:

submission.registration_result = result = {}
with build_client(service) as client:
result["api_response"] = res = client.post(
options.get("relative_api_endpoint", ""),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this requires some run-time validation just to be sure :)

src/openforms/registrations/contrib/json/plugin.py Outdated Show resolved Hide resolved
src/openforms/registrations/contrib/json/plugin.py Outdated Show resolved Hide resolved
PR feedback: global config checks are not really useful for this plugin right now.
A bit more descriptive and avoids possible naming conflicts
formData will be an empty object if we add a new object, which means the default values for the configuration fields are missing
* Add required and noManageChildProps to ServiceSelect and FormVariableSelect because they are required and need to prevent some Field props being passed down to the children
* Remove max length of FormioVariableKeyField: it is based on a text field which has no length constraints
* Remove id field from RelativeAPIEndpoint TextInput: is managed automatically
* Remove Required from JSONDumpOptions: required is the default for TypedDict
* Add default value for relative_api_endpoint: guarantees that the key is always present, which gives a strict type definition
* Remove content type headers: is set automatically when using the json kwargs
* Replace 'Included/Not Included' text with icons: makes it easier for the user to see whether it is included.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a registration backend that sends the form/submission variables as JSON to some API endpoint
2 participants