Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.msg file upload #4961

Merged
merged 6 commits into from
Dec 30, 2024
Merged

.msg file upload #4961

merged 6 commits into from
Dec 30, 2024

Conversation

robinmolen
Copy link
Contributor

Closes #4795

Changes

The sdk cannot determine what content type should be used for .msg files, which results in the content type "". When you try to submit the form, the fileSerializer doesn't recognizes the content type and returns an error.

To allow the submission of .msg files, the content type should be allowed to be empty.

  • Turned type property on FileSerializer optional
  • Added validation for .msg files to the MimeTypeValidator

Checklist

Check off the items that are completed or not relevant.

  • Impact on features

    • Checked copying a form
    • Checked import/export of a form
    • Config checks in the configuration overview admin page
    • Problem detection in the admin email digest is handled
  • Release management

    • I have labelled the PR as "needs-backport" accordingly
  • I have updated the translations assets (you do NOT need to provide translations)

    • Ran ./bin/makemessages_js.sh
    • Ran ./bin/compilemessages_js.sh
  • Dockerfile/scripts

    • Updated the Dockerfile with the necessary scripts from the ./bin folder
  • Commit hygiene

    • Commit messages refer to the relevant Github issue
    • Commit messages explain the "why" of change, not the how

@robinmolen robinmolen marked this pull request as draft December 19, 2024 16:40
@robinmolen
Copy link
Contributor Author

The removal of the FileSerializer type blank message conflicts with #4496. Some more work is required to ensure users receive useful error messages, when needed

@robinmolen robinmolen force-pushed the bug/4795-msg-file-upload branch from 6833a2c to 4908172 Compare December 23, 2024 08:58
@robinmolen
Copy link
Contributor Author

The MimeTypeValidator now checks if the file has an extension. If it doesn't, an validation error is raised

@robinmolen robinmolen force-pushed the bug/4795-msg-file-upload branch from 4908172 to 2d11d04 Compare December 23, 2024 09:01
Copy link

codecov bot commented Dec 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.64%. Comparing base (28754fd) to head (f3bf841).
Report is 7 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4961   +/-   ##
=======================================
  Coverage   96.64%   96.64%           
=======================================
  Files         761      761           
  Lines       25904    25908    +4     
  Branches     3385     3386    +1     
=======================================
+ Hits        25036    25040    +4     
  Misses        604      604           
  Partials      264      264           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@robinmolen robinmolen force-pushed the bug/4795-msg-file-upload branch 2 times, most recently from cc51188 to 5e77364 Compare December 23, 2024 10:50
@robinmolen robinmolen marked this pull request as ready for review December 23, 2024 10:51
@sergei-maertens sergei-maertens force-pushed the bug/4795-msg-file-upload branch from 5e77364 to 061dce1 Compare December 27, 2024 10:17
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please exclude backend translations from PRs? They get sorted out during the release preparation.

They're a frequent source of merge conflicts otherwise.

src/openforms/formio/tests/test_validators.py Outdated Show resolved Hide resolved
src/openforms/formio/components/vanilla.py Outdated Show resolved Hide resolved
src/openforms/formio/api/validators.py Outdated Show resolved Hide resolved
src/openforms/formio/api/validators.py Outdated Show resolved Hide resolved
src/openforms/formio/api/validators.py Outdated Show resolved Hide resolved
@sergei-maertens sergei-maertens force-pushed the bug/4795-msg-file-upload branch from 061dce1 to 647522e Compare December 30, 2024 10:12
@sergei-maertens sergei-maertens force-pushed the bug/4795-msg-file-upload branch 2 times, most recently from 071c30b to d82f035 Compare December 30, 2024 11:40
robinmolen and others added 5 commits December 30, 2024 12:56
The SDK cannot reliably determine which content type belongs to a .msg
file, most notably on Linux and MacOS because the extension is not in
the mime type database. This manifests as a file being uploaded with empty
content-type.

To allow these files to go through, the serializer must allow empty
values for the 'type' field which contains the detected content type,
and the backend must perform additional processing to determine the file
type. We can do this by falling back to the generic case of 'binary
file' (application/octet-stream) content type, and let libmagic figure
out which extensions belong to the magic bytes, i.e. we look at the
magic bytes to figure out what kind of file was provided, and we check
the provided file extensions against the list of valid extensions for
the detected file type.
The validator now rejects (temporary) file uploads that don't have an
extension, as this prevents validating that the extension and content
type match. This used to pass and would then be caught later when
linking the file upload component and temporary file upload.
@sergei-maertens sergei-maertens force-pushed the bug/4795-msg-file-upload branch from 1ac9576 to f3bf841 Compare December 30, 2024 11:56
@sergei-maertens sergei-maertens added the needs-backport Fix must be backported to stable release branch label Dec 30, 2024
sergei-maertens pushed a commit that referenced this pull request Dec 30, 2024
The SDK cannot reliably determine which content type belongs to a .msg
file, most notably on Linux and MacOS because the extension is not in
the mime type database. This manifests as a file being uploaded with empty
content-type.

To allow these files to go through, the serializer must allow empty
values for the 'type' field which contains the detected content type,
and the backend must perform additional processing to determine the file
type. We can do this by falling back to the generic case of 'binary
file' (application/octet-stream) content type, and let libmagic figure
out which extensions belong to the magic bytes, i.e. we look at the
magic bytes to figure out what kind of file was provided, and we check
the provided file extensions against the list of valid extensions for
the detected file type.

Backport-of: #4961
sergei-maertens pushed a commit that referenced this pull request Dec 30, 2024
The SDK cannot reliably determine which content type belongs to a .msg
file, most notably on Linux and MacOS because the extension is not in
the mime type database. This manifests as a file being uploaded with empty
content-type.

To allow these files to go through, the serializer must allow empty
values for the 'type' field which contains the detected content type,
and the backend must perform additional processing to determine the file
type. We can do this by falling back to the generic case of 'binary
file' (application/octet-stream) content type, and let libmagic figure
out which extensions belong to the magic bytes, i.e. we look at the
magic bytes to figure out what kind of file was provided, and we check
the provided file extensions against the list of valid extensions for
the detected file type.

Backport-of: #4961
@sergei-maertens
Copy link
Member

Backports:

@sergei-maertens sergei-maertens merged commit 5e67644 into master Dec 30, 2024
26 of 28 checks passed
@sergei-maertens sergei-maertens deleted the bug/4795-msg-file-upload branch December 30, 2024 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-backport Fix must be backported to stable release branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

.msg files cannot be uploaded
2 participants