Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix StreamUtil to execute the map function for each batch in the thread pool #548

Merged
merged 1 commit into from
Feb 14, 2024

Conversation

xiaoyu-yang-gh
Copy link
Collaborator

methods like filter and map on stream are lazy. They just create a new stream. Traversal of the pipeline source does not begin until the terminal operation of the pipeline is executed.

The StreamUtil wants the map method executed for each batch in the thread pool Asynchronously, so we should make the stream pipeline executed in the thread-pool for the reason mentioned above.

Here the collect is the terminal operation that triggers the stream execution.

@codecov-commenter
Copy link

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (5249a34) 46.19% compared to head (1debdf1) 46.18%.

Files Patch % Lines
...om/linkedin/avroutil1/builder/util/StreamUtil.java 0.00% 4 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master     #548      +/-   ##
============================================
- Coverage     46.19%   46.18%   -0.01%     
+ Complexity     4546     4544       -2     
============================================
  Files           407      407              
  Lines         28442    28443       +1     
  Branches       4642     4642              
============================================
- Hits          13138    13137       -1     
- Misses        13730    13731       +1     
- Partials       1574     1575       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@karthikrg
Copy link
Contributor

LGTM. This is a subtle behavior with flatMap that changed in Java 10.
https://blog.jooq.org/are-java-8-streams-truly-lazy-not-completely/

@xiaoyu-yang-gh xiaoyu-yang-gh merged commit 51ff79b into linkedin:master Feb 14, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants