feat: Add kafka to docker compose

[bmtcril]

This PR is preliminary work on combining the Kafka and redis implementations in one Tutor plugin, and this branch will be shared for feedback on that idea. If it passes community feedback we'll rename this repository and maintain this as a first class plugin.

Current State

• Obviously very rough, it's just a proof of concept
• The login event is configured and enabled, and there is a default LMS consumer that will start and read those events as they come in (though I don't think there event receivers that do anything with it, so it's a no-op).
• Kafka is configured as the default while we test things, and required services are able to be run as they are configured in devstack, with Zookeeper, Schema Registry, and Control Center. Almost nothing is configurable for them currently.
• Tutor dev and local both work, k8s has not been implemented yet.
• redis works

Testing

Install this repo in your Tutor environment and enable it:

• git clone https://github.com/openedx/tutor-contrib-event-bus-redis
• cd tutor-contrib-event-bus-redis
• git checkout bmtcril/add_kafka
• Activate your Tutor virtual environment
• pip install -e .
• tutor plugins list to make sure "event-bus-redis" is installed
• tutor plugins enable event-bus-redis

Rebuild the openedx image to get all of the dependencies:

• tutor images build openedx --no-cache (or if you are in dev tutor images build openedx-dev --no-cache)

Start the services, Kafka is currently enabled by default:

• tutor local start -d
• Wait a little while, then check that kafka and schema-registry actually started clean and are still running, sometimes they just die on startup and will auto restart a few times before becoming stable.
• You should be able to monitor Kafka using the "kafka control center", it's on by default at port 9021
• Navigate to "Topics" in the control center and add one named "dev-user-login" and save with the defaults, or you can just log into LMS and log out again to create the topic automatically (again presuming kafka is healthy)

Test:

• Log in to the LMS
• The logs for the login-consumer container should now say [ Responses of the Open edX Event <org.openedx.learning.auth.session.login.completed.v1>](consumer.py:393 - Message from Kafka processed successfully)
• Take a look at the control center UI to see what's happened under Topics -> dev-user-login -> messages you can look at the messages sent by setting the offset to 0 in the search box.

Test redis:

• tutor local stop if you're still running the kafka version
• Change your Tutor settings:
  • EVENT_BUS_BACKEND = "redis"
  • RUN_KAFKA_SERVER = False
• tutor config save
• tutor local start -d
• You should not see any of the Kafka services running
• Log in to the LMS
• The logs for the login-consumer container should say [ Responses of the Open edX Event <org.openedx.learning.auth.session.login.completed.v1>](consumer.py:393 - Message from redis processed successfully)

Future work

If we want to pursue this direction , there are a lot of things we could do:

• Create configured Kafka topics on init
• Make Kafka integration configurable
• Generally clean up the configuration
  • Especially producer / consumer config
• Build out and maintain a list of known bus-enabled events
• Build / run consumers for each event, and have them start based on config
• Make k8s work
• Make the event configuration / images / consumers extensible
• Support externally hosted Kafka
• (Optionally) create a 2nd redis service just for the bus

[Ian2012]

can you pin this one?

[Ian2012]

Can you remove this variable?

[Ian2012]

Tested locally

[bmtcril]

@navinkarkera can you take a look and let me know what you think about this direction?

[navinkarkera]

@bmtcril Sure. I'll go through it tomorrow.

[navinkarkera]

@bmtcril I remember our discussion, we came to the conclusion that tutor plugin for redis event bus is not really helpful. Our reasoning was

that event_bus_redis actually needs to be installed in the host environment instead of running as an IDA. So if we want to publish events from edx-platform, event_bus_redis needs to be installed as a python package in its environment and if discovery wants to consume events the same is required there as well, event_bus_redis needs to be installed as python package. Creating a plugin can only install in one host as tutor does not have an option to inject extra deps to multiple IDA's (not that we are aware of) and even without a plugin, the user just needs to add this package as extra dep and 4-5 django settings.

Has something changed in tutor which allows us to install extra deps in multiple IDA's now? I can see that this can be helpful in regards to kafka as it is not included in openedx by default.

[bmtcril]

@navinkarkera the idea is to let the plugin handle the complicated configuration bits, especially with Kafka, but also with running consumers. I think that we can work to add plugin introspection to find other plugins and extend them as necessary / use their configuration if they exist / update their configuration to use the event bus here. The hope is that with one plugin to manage the versions of things and some of the complexity of running consumers (especially multiple consumers) in k8s in one spot.

I'm definitely open to hearing this is a bad idea, that's why I wanted to shop it around for feedback early. I'm not entirely convinced this will work the way I'd hope. Especially with cases like custom IDAs that this plugin wouldn't know about.

[navinkarkera]

@bmtcril Makes sense, especially for kafka and running consumers. I don't think it is a bad idea 😄 .

Especially with cases like custom IDAs that this plugin wouldn't know about.

Yes, that will be hard. Maybe we can contribute (if possible) a feature to tutor where it will allow us to patch dependencies in all IDA's.