Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Append Transformation Edge Cases #595

Merged
merged 3 commits into from
May 15, 2024

Conversation

miles-grant-ibigroup
Copy link
Contributor

Checklist

  • Appropriate branch selected (all PRs must first be merged to dev before they can be merged to master)
  • Any modified or new methods or classes have helpful JavaDoc and code is thoroughly commented
  • The description lists all applicable issues this PR seeks to resolve
  • The description lists any configuration setting(s) that differ from the default settings
  • All tests and CI builds passing
  • The description lists all relevant PRs included in this release (remove this if not merging to master)
  • e2e tests are all passing (remove this if not merging to master)

Description

The append transform wasn't handling newlines correctly. Newlines in the middle of a file would cause datatools to stop processing right there, causing problems.

I'm not entirely sold on the solution I've found here of creating a second file to store the non-newlined text contents. My thinking was this was more stable than relying on memory (imagine an >100mb text file, which happens!)

Copy link
Contributor

@br648 br648 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor items to address. I think given how large these files can get, this is probably the best approach. Unit test fails on Windows OS. I may address this one day.

@br648 br648 assigned miles-grant-ibigroup and unassigned br648 May 15, 2024
Copy link

@josh-willis-arcadis josh-willis-arcadis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@miles-grant-ibigroup miles-grant-ibigroup merged commit 18dbbc3 into dev May 15, 2024
6 checks passed
@miles-grant-ibigroup miles-grant-ibigroup deleted the fix-append-transform-edge-cases branch May 15, 2024 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants