Kafka Connect & MongoDB Database #74
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ran into consumer performance problems where consumers could't subscribe to topics properly.
|
Requires this PR to be merged first: #71 |
1 task
|
Got a small conflict with the README |
Okay, that's fixed now! |
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Details
Description
This PR includes an additional docker-compose file named: "docker-compose-mongo.yml" which stands up a MongoDB database with a kafka connect instance to sink topics specified in the ./jpo-s3-deposit/mongo-connector/connect_start.sh script.
This PR also makes a change to the new kafka instance to use a docker managed kafka volume rather than a bind mount docker volume. This improves the performance of kafka (especially for windows users) by keeping everything in the docker file system. This is also helpful as now the kafka data can be reset if you attach the
-vtag when usingdocker compose down -vcommand.Related Issue
No related issue.
Motivation and Context
The move to have this functionality in the core ODE environment comes from combining requirements from the jpo-conflictmonitor and the jpo-cvmanager. When deploying in a local environment both of these will require kafka connect and a MongoDB instance to work from.
How Has This Been Tested?
If you stand up the mongo environment with the command:
docker compose -f .\docker-compose-mongo.yml up -d, the standard ODE environment will be stood up along with the Mongo and kafka connect container. There is also a MongoDB configuration container that provisions users for RBAC. Please make sure to update your.envfile with the new MongoDB required environmental variables from thesample.envTypes of changes
Checklist:
ODE Contributing Guide