feat: allow to route events in sync

[Ian2012]

Description

This PR creates a handler to process tracking log from the event bus.

• Update the event router to include a field to define when to process the events in celery or in the same thread.
• Update the setting to use the event bus by default.

Testing instructions

• Install openedx-events from the mentioned PR
• Update your EVENT_TRACKING_BACKENDS settings to use the engine eventtracking.backends.event_bus.EventBusRoutingBackend and update the caliper and xapi backends to use: event_routing_backends.backends.sync_events_router.SyncEventsRouter
• Install event-tracking from this PR feat: add event bus backend event-tracking#246
• Configure the event bus
• Run: tutor dev exec lms ./manage.py lms consume_events -t analytics -g event_routing_backends --extra '{"consumer_name": "aspects"}'
• Install, enable and initialize tutor-contrib-aspects
• Verify the xAPI statements are sent to Ralph

[openedx-webhooks]

Thanks for the pull request, @Ian2012! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[Ian2012]

@ziafazal do you have any ideas to update the settings based on the event bus is enabled? If not use celery

cc @bmtcril

[ziafazal]

@ziafazal do you have any ideas to update the settings based on the event bus is enabled? If not use celery
@Ian2012 I'm not getting you. Could you please explain a bit what you are trying to achieve?

[Ian2012]

I can imagine some people that would like to still use the celery backend instead of the event bus. Before the settings were connected to the Async one, and I've changed it to the EventBus. How can we automagically update our own settings to define when to use one or the other?

[ziafazal]

@Ian2012 if you don't want to process EventBus events asnchronously you can use this engine
'ENGINE': 'eventtracking.backends.routing.RoutingBackend',

[Ian2012]

We are implementing a new backend for these reasons:

• Reduce the latency of Aspects by sending events through the event bus
• Scale the Aspects sync service independently of the LMS-workers
• Using the RoutingBackend by default will send all the events synchronously and will cause slowdowns in the LMS

I'm in the process of refactoring the EventRouter so that there are two classes to send events. One for using celery and another one for sending events in the same thread.

@bmtcril The current implementation will process and load events to the backends in the same thread but I'm not sure if that's what we really want.

I see some advantages in performing everything in the same thread and defining an "IDA" in Aspects, but then we will lose the retry capabilities and we will have to perform more operator tasks related to the event bus.

[bmtcril]

Still trying to wrap my head around it all, but here are some initial comments

[Ian2012]

@bmtcril added new test cases and updated the docs

[Ian2012]

@bmtcril added batching capabilities. Hopefully, we can implement a management command to trigger the consumption of events every N seconds, but I'm worried about race conditions unless we only send events every N seconds and not every X event.

I'm using a Redis list to save the events to be sent to the backends (xAPI, Caliper) then those are processed and sent in batch to the backends

[Ian2012]

@ziafazal This is ready for re-review. Especially if you have any concerns about the settings changes or the refactoring commit: cf59bfa

[ziafazal]

@Ian2012 Thanks for making these changes. I have a question could you please answer that?

[Ian2012]

@ziafazal @bmtcril This is finally ready for review, it's mainly a refactor however we should release and tag this to allow people to configure the event bus backend.

Would you agree to not change the default settings here but to provide a flag in tutor-contrib-aspects that allows to easily switch from celery to the event-bus?

I'm thinking on ASPECTS_ENABLE_EVENT_BUS_CONSUMER which automatically configures the xapi and celery backends to use the event bus backends and the sync router

[ziafazal]

LGTM 👍

[openedx-webhooks]

@Ian2012 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.