Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PLUGIN-1293] Gcs plugin Bug for buckets having delete multi part upload policy. #1307

Merged

Conversation

Shubhangi-cs
Copy link
Contributor

Bug:
Pipeline fails with warning and error for the GCS buckets with delete multi part upload policy.

Bugfix:
Plugin should support the buckets with this policy enabled in it. Bumped up the library version to fix this.

https://cdap.atlassian.net/browse/PLUGIN-1293

@Shubhangi-cs Shubhangi-cs added the build Trigger unit test build label Sep 25, 2023
@Shubhangi-cs Shubhangi-cs force-pushed the gcs_fix_upgradation branch 2 times, most recently from 1bac32f to 5be6d23 Compare September 26, 2023 09:03
@Shubhangi-cs Shubhangi-cs force-pushed the gcs_fix_upgradation branch 4 times, most recently from c66693d to 52b0645 Compare September 28, 2023 05:09
Copy link
Member

@itsankit-google itsankit-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an e2e test.

@itsankit-google
Copy link
Member

There are also two required e2e tests failures, ptal: https://github.com/data-integrations/google-cloud/actions/runs/6334790063/job/17216255525?pr=1307

[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 52.683 s <<< FAILURE! - in GCS source - Validate GCS plugin output schema for different formats
[ERROR] GCS source - Validate GCS plugin output schema for different formats.GCS Source output schema validation for blob, parquet, avro and text format  Time elapsed: 14.287 s  <<< FAILURE!
java.lang.AssertionError: Schema displayed on UI should match with the expected Schema

[ERROR] GCS source - Validate GCS plugin output schema for different formats.GCS Source output schema validation for blob, parquet, avro and text format  Time elapsed: 12.998 s  <<< FAILURE!
java.lang.AssertionError: Schema displayed on UI should match with the expected Schema

@Shubhangi-cs
Copy link
Contributor Author

Please add an e2e test.
e2e test case can be added further and separate PR can be raised for the same, till then we can go ahead merging this PR.

Copy link
Member

@itsankit-google itsankit-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code changes lgtm, but we'll have to make sure we do pretty comprehensive testing. Browsing/sampling, service account json vs auto-detect, on dataproc vs on k8s, etc.

@vikasrathee-cs vikasrathee-cs merged commit dc01190 into data-integrations:develop Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Trigger unit test build
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants