Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Add a test to prove how to use protobuf nullability #11712

Closed
wants to merge 5 commits into from
Closed

[DO NOT MERGE] Add a test to prove how to use protobuf nullability #11712

wants to merge 5 commits into from

Conversation

gortiz
Copy link
Contributor

@gortiz gortiz commented Sep 29, 2023

This PR adds a test to verify the correct nullability read on ProtoBufRecordExtractor.

The PR is not yet ready to merge because for some reason the descriptor is using v2 instead of 3.

cc @swaminathanmanish

@gortiz gortiz marked this pull request as draft September 29, 2023 11:44
@codecov-commenter
Copy link

codecov-commenter commented Sep 29, 2023 •
edited
Loading

Codecov Report

Merging #11712 (202e5b1) into master (54a1ca0) will decrease coverage by 63.05%.
Report is 3 commits behind head on master.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master   #11712       +/-   ##
=============================================
- Coverage     63.05%    0.00%   -63.05%     
+ Complexity     1118        6     -1112     
=============================================
  Files          2342     2266       -76     
  Lines        125787   122066     -3721     
  Branches      19335    18780      -555     
=============================================
- Hits          79314        6    -79308     
- Misses        40813   122060    +81247     
+ Partials       5660        0     -5660
Flag Coverage Δ
integration <0.01% <ø> (ø)
integration1 <0.01% <ø> (ø)
integration2 0.00% <ø> (ø)
java-11 0.00% <ø> (-63.02%) ⬇️
java-17 <0.01% <ø> (-62.91%) ⬇️
java-20 0.00% <ø> (-62.93%) ⬇️
temurin <0.01% <ø> (-63.05%) ⬇️
unittests ?
unittests1 ?
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1990 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

swaminathanmanish
swaminathanmanish reviewed Sep 29, 2023
View reviewed changes
...obuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordExtractor.java Outdated
@@ -62,7 +62,7 @@ private Object getFieldValue(Descriptors.FieldDescriptor fieldDescriptor, Messag
// Note w.r.t proto3 - If a field is not declared with optional keyword, there's no way to distinguish
// if its explicitly set to a proto default or not been set at all i.e hasField() returns false
// and we would use null.
if (fieldDescriptor.isRepeated() || !fieldDescriptor.isOptional() || message.hasField(fieldDescriptor)) {
if (fieldDescriptor.isRepeated() || !fieldDescriptor.hasOptionalKeyword() || message.hasField(fieldDescriptor)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exactly what my PR changed - from hasOptionalKeyword to isOptional

#11682

API is deprecated
protocolbuffers/protobuf@d6157f7

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is deprecated in latest versions of protobuf, but:

  • It is not deprecated in the version we use
  • There is no alternative to this method in the latest protobuf version.

Theoretically we should is isPresent, but it is not honoring its javadoc. You can verify that by changing this to isPresent and see that it returns false for the test added here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems isPresent is not bug, but I was unable to correctly configure protobuf model programmatically.

@gortiz
Copy link
Contributor Author

gortiz commented Oct 2, 2023 •
edited
Loading

I've changed the code to:

  • Recover the logic in master (use isOptional)
  • Add ProtoBufRecordExtractorLowLevelTest which runs tests using the same protocol definition used by ProtoBufRecordExtractorTest but doesn't include all the error prone magic in AbstractRecordExtractorTest.

I think ProtoBufRecordExtractorLowLevelTest is easier to read than the Simple test I've added in previous PRs. There are 4 test cases:

  1. When the field is set to some non default value
  2. When the field is set to the default value
  3. When the field is not set
  4. When the field is clear (which should be semantically equal to 3, but I wanted to be sure.

It is easier and more useful to read the DataProviders than the tests.

As you can see, both ProtoBufRecordExtractorProgrammaticTest and ProtoBufRecordExtractorLowLevelTest have some test that fail. All these fails can be fixed in two ways:

  • By recovering Shen's code where we used hasOptionalKeyword instead of isOptional. We know isOptionalKeyword is deprecated
  • By using hasPresence, as it has been done in Proto null fix #11723

@gortiz gortiz mentioned this pull request Oct 2, 2023
Merged
@gortiz gortiz closed this Oct 4, 2023
@gortiz gortiz deleted the proto-null branch October 4, 2023 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@swaminathanmanish swaminathanmanish swaminathanmanish left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@gortiz @codecov-commenter @swaminathanmanish