Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Kafka exporter and receiver configuration #5703

Merged
merged 30 commits into from
Jul 24, 2024
Merged

Add Kafka exporter and receiver configuration #5703

merged 30 commits into from
Jul 24, 2024

Conversation

joeyyy09
Copy link
Member

@joeyyy09 joeyyy09 commented Jul 3, 2024 •
edited
Loading

Which problem is this PR solving?

Description of the changes

  • Add config files for replacement of jaeger-collector and jaeger-ingester.
  • Add the architecture for the integration tests for kafka.
  • Add e2e integration tests for Kafka.

How was this change tested?

  • Manually ran the collector, ingester configs to check if the data is being pushed and pulled the right way and verified the architecture.

Checklist

@joeyyy09
Add configs for collector and ingester
f02e2fc 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@codecov Codecov
Copy link

codecov bot commented Jul 3, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.66%. Comparing base (d8b2110) to head (d7c70cf).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #5703      +/-   ##
==========================================
+ Coverage   96.65%   96.66%   +0.01%     
==========================================
  Files         342      342              
  Lines       16515    16515              
==========================================
+ Hits        15962    15964       +2     
+ Misses        363      362       -1     
+ Partials      190      189       -1
Flag Coverage Δ
badger_v1 8.05% <ø> (ø)
badger_v2 1.81% <ø> (ø)
cassandra-3.x-v1 16.61% <ø> (ø)
cassandra-3.x-v2 1.74% <ø> (ø)
cassandra-4.x-v1 16.61% <ø> (ø)
cassandra-4.x-v2 1.74% <ø> (ø)
elasticsearch-6.x-v1 18.77% <ø> (-0.02%) ⬇️
elasticsearch-7.x-v1 18.83% <ø> (-0.02%) ⬇️
elasticsearch-8.x-v1 19.02% <ø> (ø)
elasticsearch-8.x-v2 1.81% <ø> (?)
grpc_v1 9.52% <ø> (ø)
grpc_v2 7.14% <ø> (ø)
kafka 9.74% <ø> (ø)
memory_v2 1.81% <ø> (ø)
opensearch-1.x-v1 18.89% <ø> (ø)
opensearch-2.x-v1 18.88% <ø> (-0.02%) ⬇️
opensearch-2.x-v2 1.81% <ø> (ø)
unittests 95.08% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

yurishkuro
yurishkuro reviewed Jul 3, 2024
View reviewed changes
Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, but we need e2e tests

@yurishkuro yurishkuro added storage/kafka v2 changelog:exprimental Change to an experimental part of the code labels Jul 3, 2024
@joeyyy09
Copy link
Member Author

joeyyy09 commented Jul 4, 2024

Lgtm, but we need e2e tests

Yeah, will get started on it!

@joeyyy09
Update collector config
a728c2e 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Update ingester config
fc7da59 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Add initial kafka_test.go
53173c5 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Add debug logs in the config
b2133cc 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Update conig files for UI
2ccff53 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Update conig files
c9c32cc 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Add kafka e2e setup
802413a 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
yurishkuro
yurishkuro reviewed Jul 18, 2024
View reviewed changes
cmd/jaeger/internal/integration/kafka_test.go Outdated Show resolved Hide resolved
cmd/jaeger/internal/integration/kafka_test.go Outdated Show resolved Hide resolved
cmd/jaeger/internal/integration/kafka_test.go Outdated Show resolved Hide resolved
@joeyyy09
Add kafka e2e setup
819ee21 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Add Readme for Kafka
d02136e 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09 @yurishkuro
Update cmd/jaeger/internal/integration/README.md
3cf0151 
Co-authored-by: Yuri Shkuro <yurishkuro@users.noreply.github.com>
Signed-off-by: Harshith Mente <109957201+joeyyy09@users.noreply.github.com>
@joeyyy09
fix merge conflicts
6ab1285 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
fix merge conflicts
66ef1f9 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Modify E2EStorageIntegration struct
4d4a882 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Fixes
7cc6a6f 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Fixes
29581a4 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Update Lint
580bdc9 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Fix Lint
fea62c6 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
yurishkuro
yurishkuro reviewed Jul 20, 2024
View reviewed changes
Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to add this new test to the gh workflow? I don't see how it's getting run now.

cmd/jaeger/internal/integration/kafka_test.go Outdated Show resolved Hide resolved
@joeyyy09
Copy link
Member Author

Do you need to add this new test to the gh workflow? I don't see how it's getting run now.

Yes, I'll be doing that

@joeyyy09
Add e2eCleanup in e2eInitialize
284acc9 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Fixes
4ba5917 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
@joeyyy09
Lint fixes
06e68ac 
Signed-off-by: joeyyy09 <menteharshith@gmail.com>
yurishkuro
yurishkuro reviewed Jul 24, 2024
View reviewed changes
Copy link
Member

@yurishkuro yurishkuro left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

yurishkuro and others added 2 commits July 24, 2024 16:09
@yurishkuro
fix ui
07f903e 
Signed-off-by: Yuri Shkuro <github@ysh.us>
@yurishkuro
Merge branch 'main' into kafka-config
46121f5
@yurishkuro yurishkuro marked this pull request as ready for review July 24, 2024 20:10
@yurishkuro yurishkuro requested a review from a team as a code owner July 24, 2024 20:10
@dosubot dosubot bot added the enhancement label Jul 24, 2024
@yurishkuro
Copy link
Member

I think we should merge this is the build is green. It's not running the e2e test, but we can add that in a separate PR.

@yurishkuro yurishkuro enabled auto-merge (squash) July 24, 2024 20:11
@yurishkuro
schedule clean-up after reader/writer are created
4d624d6 
Signed-off-by: Yuri Shkuro <github@ysh.us>
@yurishkuro
remove duplicate call to cleanup
d7c70cf 
Signed-off-by: Yuri Shkuro <github@ysh.us>
@yurishkuro yurishkuro merged commit e437971 into jaegertracing:main Jul 24, 2024
44 checks passed
@joeyyy09
Copy link
Member Author

I think we should merge this is the build is green. It's not running the e2e test, but we can add that in a separate PR.

Yeah sure, i'll open another PR for the e2e test

@joeyyy09 joeyyy09 deleted the kafka-config branch July 29, 2024 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yurishkuro yurishkuro yurishkuro approved these changes

@albertteoh albertteoh Awaiting requested review from albertteoh albertteoh is a code owner automatically assigned from jaegertracing/jaeger-maintainers

Assignees
No one assigned
Labels
changelog:exprimental Change to an experimental part of the code enhancement storage/kafka v2
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Jaeger v2] Add Kafka exporter and receiver
2 participants
@joeyyy09 @yurishkuro