API Validation Rules/Patterns: Standardizing Validation Error Responses in API Endpoints

[defstat]

Hi team,

I would like to start a discussion on how our API endpoints should handle and return validation errors. On the work on the Invitations API, there is a need to ensure that validation errors are returned in a consistent and informative manner, distinguishing between

• UI field validations,
• generic validations, and
• exceptions

We should consider a standardized and documented way of returning such errors which will lead to more clarity and consistency in how errors are handled on the front-end and back-end.

There is already a way to return errors (see for example this). I can't see that documented though so that it is clear for the API developers and consumers on how to process different kinds of errors.

Proposal

1. UI Field Validations:
   • These are errors that are specific to individual fields on the UI.
   • Example: A required field is missing or an input does not meet specific criteria (e.g., email format, password strength).
   • HTTP Status Code: 400 Bad Request or 422 Unprocessable Entity
   • Response Structure:

     {
         "status": "error",
         "type": "UI_VALIDATION",
         "field_errors": {
             "field_name": "Localized error message for the field"
         }
     }

2. Generic Validations:
   • These are errors that are not tied to specific UI fields but are related to the overall data or request.
   • Example: Business logic validation such as "end date must be after start date."
   • HTTP Status Code: 422 Unprocessable Entity
   • Response Structure:

     {
         "status": "error",
         "type": "GENERIC_VALIDATION",
         "messages": [
             "Localized first possible error message",
             "Localized second possible error message"
         ]
     }

3. Exceptions:
   • These are unexpected errors that occur during the processing of the request.
   • Example: Database errors, unhandled exceptions, or external service failures.
   • HTTP Status Code: 500 Internal Server Error
   • Response Structure:

     {
         "status": "error",
         "type": "EXCEPTION",
         "ui_message": "User specific and localized exception error message",
         "message": "Systemic exception error message",
         "details": "Additional details about the exception"
     }

Benefits

• Clarity: Differentiating between types of validation errors helps the front-end handle errors appropriately.
• Consistency: A standardized structure ensures that all API responses are uniform.
• Debugging: Clear distinctions between error types can aid in debugging and troubleshooting.

Further Questions

• Should we include additional metadata for each error type?
• How do we handle nested field errors in complex data structures? (is something like field.subfield sufficient)
• Are the proposed HTTP status codes appropriate for each type of validation error?
• How should we handle partial successes?
• Should we consider a pattern that ties backend and frontend field names somehow?
• Should we consider Entity specific errors (rather than field specific errors) in cases that such a concept applies and if so should we consider that a different kind of validation?
• Should we merge cases 1.UI Field Validations and 2.Generic Validations above into one response case?

[jonasraoni]

I think this is very important, we definitely need have a standard way to return errors (here I'm not sure if we have many different patterns in place).

I think the errors should be validated/propagated following this order:

• HTTP status defines the type of error (bad request, not authorized, unexpected error, etc), we don't even need a "type" on the response.
• Input validation (field missing a value, start date bigger than end date, etc)
• Business logic validation

So, I'd merge the items "2" and "3" from your list.

[jonasraoni]

About the questions:

Should we include additional metadata for each error type?

Hmm, not sure what type of additional metadata you mean here. In the past I was using a validation tool that returned extra information, something like this:

{
  "password": {
    "invalidCharacter": {"accepted": "a-z"},
    "minLength": {"minimum": 20}
  }
}

How do we handle nested field errors in complex data structures? (is something like field.subfield sufficient)

I think we might need to represent arrays properly too, but not sure here, in case we have .

Are the proposed HTTP status codes appropriate for each type of validation error?

I think the 400, 403, 404, 500 are enough. The 422 is ok if we're willing to add such "business logic" errors, but I think it can be merged with the 400 to simplify things.

How should we handle partial successes?

Do we have such condition? For me an operation should either work or fail.

As an end-user, I expect that a partial success will eventually turn into success later.
Otherwise, I should be warned that even though my request was accepted, things can still go wrong, then I should look for a status frequently or be notified.

Submitting an article is a partial success in this sense, it may fail to be published, and the user must interact/check the status.

Should we consider a pattern that ties backend and frontend field names somehow?

Depending on how we're creating the forms it might not be possible (with Vue field names might not be that important) and maps might be needed (e.g. use authorName on the UI instead of name, to avoid conflicts with another name field).

Should we consider Entity specific errors (rather than field specific errors) in cases that such a concept applies and if so should we consider that a different kind of validation?

I didn't understand.

Should we merge cases 1.UI Field Validations and 2.Generic Validations above into one response case?

In my opinion, simple validations should happen before we start validating the business logic (anything that requires database access, extra processing, calling external services, etc.). Would be nice to avoid duplicated code on the client/server, but that will require using a framework.

But nothing against merging both. We can have the field errors together with a generic "errors".

[jardakotesovec]

Difference between different inputs

I think its fine to keep 2 & 3 together. It would be hard for API creator to think about whats field input and whats some other user input. Its all input :-).

Nested fields

From what I can we use Laravel validation, so I think that guide us how the nested fields will be presented - https://laravel.com/docs/11.x/validation#validating-nested-array-input

Error type

Agree with Jonas that there is not much value of adding type. That can be covered with http code.

500

For 500 errors - these are unexpected errors - so unlikely to be able to communicate accurate error to the user. Not sure if was in past was considered to expose stack trace, so users have something to share on the forum. This is often considered as bad security practice revealing internals - but with opensource software thats maybe not concern.

Current errors and consistency

Of course everyone agree on consistency. But I suspect someone would need to investigate how good/bad are we with consistency with existing API.

Couple of endpoints I checked for validation error returns 400 with the object of errors (Example). Which I don't have any problem at all. It has all the information I need on frontend.

Shape of errors for multilingual fields

One thing thats specific for OJS/OMP/OPS are multilingual fields. Don't have much insights at this point, apart from that we might need to communicate validation errors multilingual as well for multilingual fields. I can explore bit more with Devika, how we might want to communicate validation errors in the future forms if that would help.

Validation errors translations

Thats something I need to understand more - how that has been approached so far.

[withanage]

I also prefer with HTPP error codes as types. ORCID is using 500 Error codes to return different types of errors extensively and is very helpful in testing.

https://github.com/ORCID/ORCID-Source/blob/main/orcid-api-web/tutorial/api_errors.md

[jardakotesovec]

@ewhanson Hi! Just tagging you, as you might help to conclude with some guidance.

[ewhanson]

Thanks, having a look now.

[defstat]

For the invitation API I am proposing the following validation strategy:

1. The invitation class has a set of validation rules. They follow the Laravel Validation rules and look like

'givenName' => [
   'sometimes',
   'string',
   'max:255',
   function ($attribute, $value, $fail) {
      if (is_null($this->getUserId()) && empty($value) && $this->getStatus() == InvitationStatus::PENDING) {
         $fail(__('invitation.validation.required', ['attribute' => $attribute]));
      }
    }
 ],
'userGroupsToAdd' => [
    'sometimes',
    'array',
    function ($attribute, $value, $fail) {
        $userGroupIds = array_column($value, 'userGroupId');
        if (count($userGroupIds) !== count(array_unique($userGroupIds))) {
            $fail(__('invitation.userRoleAssignment.validation.error.addUserRoles.duplicateUserGroupId'));
        }
    }
],
'userGroupsToAdd.*' => [
    'array',
    function ($attribute, $value, $fail) {
        $allowedKeys = ['userGroupId', 'masthead', 'dateStart', 'dateEnd'];
        $unexpectedKeys = array_diff(array_keys($value), $allowedKeys);
        if (!empty($unexpectedKeys)) {
            $fail(__('invitation.userRoleAssignment.validation.error.userRoles.unexpectedProperties', [
                'attribute' => $attribute,
                'properties' => implode(', ', $unexpectedKeys),
            ]));
        }
    }
],
'userGroupsToAdd.*.userGroupId' => [
    'required',
    'integer',
    function ($attribute, $value, $fail) {
        // Custom validation logic for userGroupId
        $userGroup = Repo::userGroup()->get($value);

        if (!isset($userGroup)) {
            $fail(__('invitation.userRoleAssignment.validation.error.addUserRoles.userGroupNotExisting', 
                [
                    'userGroupId' => $value
                ])
            );
        }
    }
],
'userGroupsToAdd.*.masthead' => 'required|bool',
'userGroupsToAdd.*.dateStart' => 'required|date',

It leaves room for already defined validation rules and custom ones. OJS already has a Validator defined PKP\validation\ValidatorFactory that can be used to validate a set of data before the backoffice code processes them.

$reqInput = $illuminateRequest->all();

$rules = $this->invitation->getValidationRules('refine');

$validator = ValidatorFactory::make(
    $reqInput,
    $rules,
    []
);

if ($validator->fails()) {
   // Do something with `$validator->errors()`
}

2. Each request is first validated against the afore mentioned rules. The further process of the request is stopped if the validation fails for at least one rule, and, if so, the errors are returns along with an http response 422: HTTP_UNPROCESSABLE_ENTITY

if ($validator->fails()) {
    return response()->json([
        'errors' => $validator->errors()
    ], Response::HTTP_UNPROCESSABLE_ENTITY);
}

3. The validation process expects only the properties that should be processed by the back office. If anything else besides those is send along with the request, the request should get a validation error. (we could ignore those, but I choose to follow the validation error approach for security reasons).

4. If the validation process does not give any errors, then the backoffice is allowed to process the request. Right now I am trying to contain any errors that I can predict into validation errors, so that they can be used better from the UI. When we will have a better mechanism for managing exceptions, maybe we should consider cases where exceptions make more sense than validation errors. (like when we are finalizing the an invitation and something in the invitation data is not right)

This is still a work in progress, but I think it makes sense to establish some ground rules regarding the strategy we should follow, so any thoughts and considerations are helpful and welcome.

[ewhanson]

This all sounds really well thought out. Here are some of my initial thoughts:

In general, I think sticking as close to Laravel conventions as possible will serve us well (which we do to a large degree already). It provides a well thought-through structure and will allow us to benefit from additions and enhancements in the future.

For UI Field validation responses, I'd propose we follow the Laravel convention outlined in the XHR Request and Validation and Validation Error Response Format section of the Laravel docs. This would standardize a format of the following for responses with a situation-specific message key and an array of errors as described in the docs:

{
	"message": "The team name must be a string. (and 4 more errors)",
	"errors": {
		"team_name": [
			"The team name must be a string.",
			"The team name must be at least 1 characters."
		],
		"authorization.role": [
			"The selected authorization.role is invalid."
		],
		"users.0.email": [
			"The users.0.email field is required."
		],
		"users.2.email": [
			"The users.2.email must be a valid email address."
		]
	}
}

and responding with 422 Unprocessable Entity.

Echoing the others, I also prefer letting the HTTP status code convey the semantics of the response rather than further details in the JSON. @jonasraoni mentioned the other response codes aside from 422 could be enough, but I don't feel too strongly about it. I primarily see using 422 as a way to standardize around Laravel's practices. The exact handling of these in a way that's useful for the frontend is the type of thing that can be collected by the fetch helper in the UI library.

Using this format for more generic errors that aren't tied to fields would work as well, there just wouldn't be any fields with field-specific "errors" in the errors array and just a message in the message key, e.g.

{
	"message": "Localized error message"
}

In @defstat's example of this, there's an array of business logic validation errors. I'm curious if you had a specific use case where it makes sense to return multiple errors like this that are not tied in any way to specific fields.

I'll spend a bit of time looking through the existing endpoints with this kind of approach in mind and see if any obvious issues/pain points pop up as well.

[jardakotesovec]

One scenario we forgot about where we do have somewhat nested structure are multilingual fields.

Which we currently returns consistently with how our multilingual fields are expressed, with nested objects:

{
    "files": [
        "You must upload at least one Article Text file."
    ],
    "abstract": {
        "en": [
            "This field is required."
        ]
    }
}

Are options are

1. Either any nesting handle via dots

      "user.givenName.en":[
         "This is not a valid string."
      ],

2. Or keep the multi lingual structure

      "user.givenName": {
           "en":[
                "This is not a valid string."
            ]
      }

Benefit of 2) is backward compatibility with our forms, but on the other hand it would not be too difficult have (until we are consistent) compatibility layer in Form components..

@defstat @ewhanson
So really interested to hear which approach you would like to prefer from backend side of things, because there will already quite a lots of form specific things to generate the 2) structure.

[defstat]

My opinion, which is also reflected in the existing code for invitations api (see #10459, specifically https://github.com/pkp/pkp-lib/pull/10472/files#diff-daa0a2489ef1d62ae0b46c12a6949c32ed68f9fbe1385b0fb9140277087870d9R68) is that we should follow the

"user.givenName.en":[
   "This is not a valid string."
]

approach.

A localized field is considered an array of values, with the locale being the key of the array and the localized value is the value corresponding to this locale. So we can utilize the laravel array validation rules in those cases.

The result in cases of validation errors will be something like

"user.givenName.{locale}":[
   "This is not a valid string."
]

For backward compatibility, I suggest investigating a solution where we could manipulate the above validation errors format to fit the needs of older forms. I think that a straight forward transformation from

"user.givenName.{locale}":[
   "This is not a valid string."
]

to

"user.givenName": {
   "{locale}":[
      "This is not a valid string."
    ]
}

could do the trick.

We should consider if there are any edge cases or complex nested structures that could complicate this transformation. As long as these cases are minimal or can be handled effectively, I believe this approach strikes a good balance between following Laravel's approach and legacy support.

[jardakotesovec]

Submission wizard logic

Just looking more what strategy the submission wizard process has taken. Thats more complex use case, as there is multiple api endpoint to interact with, to include all the data, but still useful to explore that.

Whats interesting that for example title&abstract are sent to publication endpoint. And that skips the validation as it consider 'submissionProgress' . Therefore it relies on the client side to enforce the title at that moment.

But it does the validation of 'everything' on last 'Review' page. Using the POST http://localhost:7003/index.php/publicknowledge/api/v1/submissions/22/submit with _validateOnly: true, and provides all the validations error at the end. So some of the validations are enforced just by setting 'isRequired' on form fields by wizard. But there are other things - like uploading Article Text File, which are not enforced by client during the process and it complains at the end if user skips it.

Creating invitation

Context

Whats @Devika008 has been asking from UX perspective is to provide more instant feedback if something is missing, not at the end of the wizard. So we want to make sure that every page is properly filled before moving user to the next.

And its not just about fields being required. For example when editor is entering role and the start/end date. These dates needs to be validates not just have sensible format, but also sensible dates. So we did not really attempt to do these validations on client side, as api needs to validate these things regardless so it can't be bypassed. Therefore implementing such validation on client would be simply duplicated effort, which might slightly improve UX and make it even more instant, but providing the feedback from API when user attempts to go to the next page I think is good balance between complexity and UX (@Devika008 feel free to chip in on this).

What I have been advocating for is that the 'update' endpoint for validation always returns all things that are still wrong. And because frontend knows which fields are being currently presented - it shows relevant subset of the errors, ideally collocated with actual fields.

I think what @defstat described - it currently works like that, right? I remember initially you attempted to validate only fields that are included in the request - but now you are validation everything, correct?

Question to explore

Only outstanding thing from my side that I wanted to ask you about is the 'partial updates'. For invitations its probably not big deal - but it could be slight UX improvement if user gets pass certain screen (so the API endpoint is hit) - that it would save the data that are passing validation (in case he abandons the process and opens the invitation later on to finish it).

Theoretically should be possible - that whatever field pass validation gets saved. But wanted to ask whether such approach creates some pain points on api implementation? Also if you think its bad idea for some reason, please share these thoughts.

[defstat]

I think what @defstat described - it currently works like that, right? I remember initially you attempted to validate only fields that are included in the request - but now you are validation everything, correct?

All the validation rules are defined in one function in the specific invitation. Then, in the case of the populate and refine endpoints, the data that we are passing through the validation process, are only the ones that are send to the request. Right now, the validation rules that we are applying for the specific Invitation uses mainly the sometimes laravel validation keyword, meaning that the specified validation rule for the property will be applied only if it is present in the request. This ensures that the "current" step will not get passed if something is missing or invalid.

In the case of invite and finalize, the backoffice validates selected data from the object, without which the process should not be continued. In those cases, again the API response returns the respective validation errors if any.

For example: let's say that the request includes the "givenName" property. Then this property can not be null or empty. But if the request does not include that property, then the validation of this property is skipped.
If we need the "givenName" to have a value by the send, of the acceptance of the invitation, namely the invite and the finalization endpoints, then we passing the current value of this field through the validation process, insuring that the invitation will not go through the invite or the finalization stages with invalid data.

In all cases, if during the request a validation error occurs, the API returns the validation errors, with an HTTP 422 code (Response::HTTP_UNPROCESSABLE_ENTITY) indicating that the request failed.

I would argue that if we attempt to use "partial updates" over only the properties that are valid, could complicate things:

1. Returning HTTP codes that indicate some errors, generally indicate that the request has stopped its process (including the DB updates). This, in my opinion, provides a clear and straight forward pattern of what has been done in the backoffice and how to handle the response. This can also help documenting the API for a more broad audience/consumers.
2. The way the steps are designed in the UI provides a clear separation of data groups. So, for example, let's say that the invitation corresponds to a new user - and you want to create a username that is already taken. This will not affect the Country, Affiliation, GivenName, FamilyName for example (you have not given them yet if they are in a later step, or you have already given them and they are already saved in the DB if they are in a previous step). That means to me that, right now, we may not find cases that partial updates will help the user with "difficult" steps that are rich with data that he may need to enter again if something fails along the way.
3. We can still opt for "partial updates" with the code as it is now by applying the following mechanism (I am not sure I would go this way, but it is technically possible):
   • Each field can be sent separately, calling its populate or refine endpoint only with the data for a specific property (except from username and password properties that the validation rules want the to be send together).
   • When the async request returns, if the response code is 200, it means that this specific field is OK (and, for example, a ✅ can be appeared next to it) and saved in the DB.
   • When the async request returns, if the response code is 422, it means that this specific field is "NOT VALID", and the validation error can appear.

[jardakotesovec]

Thanks @defstat for elaborating, thats helpful. And I do like your proposal more compared to what I was proposing.

In general when we are creating new object (like new contributor) it make sense to require everything at once, but in this case where we have process of creating invitation, than refining and submitting - I think what you describe is better fit.

Will try to summarise:

• for populate/refine will client send only the fields that are presented on given step
• API validates these and once its successful it gets saved to the database (therefore we get partial saves per step)
• invite/finalise make sure that everything has been provided in the process and returns validation error if not. But if the client is correctly posting relevant fields for each step - at the end of this process its certain that it should pass with 200. And validation here is more to ensure that the client is working correctly, than to actually still provide feedback to the user.

@ewhanson Can you just double check this from your point of view? Once you give it green light, @ipula could start working on client side adjustments.

[ewhanson]

Thanks @jardakotesovec and @defstat. That all sounds conceptually good to me, and @jardakotesovec your summary fits with my understanding as well. If we do go for any "partial updates," I'd like to have a look once it's minimally working to make sure we're all on the same page for implementation, but otherwise 👍

[defstat]

invite/finalise make sure that everything has been provided in the process and returns validation error if not. But if the client is correctly posting relevant fields for each step - at the end of this process its certain that it should pass with 200. And validation here is more to ensure that the client is working correctly, than to actually still provide feedback to the user.

@jardakotesovec @ewhanson, just a small comment here: performing a "validity" check at these terminal steps of the invitation process is not merely a technicality for the following reasons:

Unvalidated Properties: There may be properties that were never sent to the populate or refine calls and therefore have not undergone any validation. These properties could be essential during the invite/finalize stages. For instance, consider the User Groups changes for an invitation that assigns a user to specific groups. The property defining which user groups are being assigned is required for the invite action. However, we cannot tag this property as required initially, because doing so would force us to provide this information during every populate action, making it the first step in the wizard.

Time-sensitive Validations: Some properties are validated against rules that may yield different results over time. A good example is the "email" property in an invitation, which highlights the distinction between validation errors and other exceptions. When finalizing the invitation, the system might need to create a new user. The email associated with the "Invite" stage may have already been checked for uniqueness against the Users table. However, if the invitation is finalized several days later, the email's uniqueness might need to be rechecked. In the initial version of the backoffice code, this check returned an exception if it failed. However, I changed this to a validation error so that the UI can display a user-friendly message.

[jardakotesovec]

@defstat Good point with time-sensitive validations - so we still have to be able to display errors from invite/finalise stages even these should be rare.

But not sure if I follow the example with User Groups. My expectation would be that once client is on the screen where user needs to fill in User Groups - it would include userGroups property in the payload and therefore it could be validated - so there is at least one userGroup and the values make sense etc?

[defstat]

@jardakotesovec It seems like there might be an assumption that the API will only be accessed through the PKP UI (and not by any third-party consumers) and that the UI will always be free of bugs, such as misspelled fields (e.g., userGroupToAdd instead of userGroupsToAdd).

However, it's important to ensure that the back office is designed to function independently of the UI.

[jardakotesovec]

@defstat Agree, thats the case I mentioned above.

And validation here is more to ensure that the client is working correctly, than to actually still provide feedback to the user.

[withanage]

Hi @jardakotesovec @defstat @ewhanson

I fully agree with the above discussion.#10190 (comment)
Specially the idea of the populate/refine Endpoints for properties that has to be explicitly validated and invite/finalize for full-validation.

if we only want to see for some alternative ideas, we can put in the payload for populate/refinesomethin like e.g. validate_all_properties=true and may reduce the number of end-point typess e.g. finalize . It is just another idea, but finalize also makes sense and more easy to understand.

[Devika008]

I completely agree with the above discussion #10190 (comment), and also the limitations described by @defstat

I think this aligns really well with the error validation logic I mentioned earlier.