Benchmark performance for xAPI over Kafka bus to Ralph

[bmtcril]

Tests for various Kafka bus configurations.

Remaining tests:

• Multiple workers with batching

[bmtcril]

Test 1 (e9d952) - 1k rows, no batching

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe, tutor-plugin-event-bus-redis @ PR 5
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: true
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: False
  • EVENT_BUS_BACKEND: kafka

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend kafka_bus
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 1000 --sleep_time 0 --tags kafka 1k local novector nobatch

Data captured for results:

• Length of run
  • Event generation: 0:00:28.648144
  • Monitoring / total run length: 0:07:26.738801
• Sleep time: 0
• Events: 1000
• Raw stats attached: e9d952_stats.txt

Findings:

• The consumer could not keep up, ClickHouse was 396 seconds behind on a 425 second run at the end
• Kafka queue grew to 950 pending events before the generation stopped and then took 395 seconds to catch up
• Inserted rows per second: 2.4 - 2.6

[bmtcril]

Test 2 (2eed44) - 1k rows, batch size 10

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe, tutor-plugin-event-bus-redis @ PR 5
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: true
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_BUS_BACKEND: kafka
  • EVENT_ROUTING_BACKEND_BATCH_SIZE: 10
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: True

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend kafka_bus
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 1000 --sleep_time 0 --tags kafka 1k local novector batch10

Data captured for results:

• Length of run
  • Event generation: 0:00:32.620124
  • Monitoring / total run length: 0:01:27.334202
• Sleep time: 0
• Events: 1000
• Raw stats attached: 2eed44_stats.txt

Findings:

• The consumer was not quite able keep up, ClickHouse lag grew to 35 seconds on a 65 second run
• Maximum kafka queue size was 621
• Inserted rows per second was 14-18, with a maximum of 22

[bmtcril]

Test 3 (708fb0) - 10k rows, batch size 100

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe, tutor-plugin-event-bus-redis @ PR 5
  • 1 consumer
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: true
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_BUS_BACKEND: kafka
  • EVENT_ROUTING_BACKEND_BATCH_SIZE: 100
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: True

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend kafka_bus
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 10000 --sleep_time 0 --tags kafka 10k local novector batch100

Data captured for results:

• Length of run
  • Event generation: 0:03:28.310482
  • Monitoring / total run length: 0:03:44.309783
• Sleep time: 0
• Events: 10,000
• Raw stats attached: 708fb0_stats.txt

Findings:

• The consumer was able keep up, ClickHouse was never more than 6 seconds behind, usually 2-3.
• Maximum kafka queue size was 19, which is to be expected on a batch size of 100.
• Inserted rows per second was consistently 40 in the middle of the test, though never higher

[bmtcril]

What we've found in #202 and earlier tests is that insert performance exceeds our ability to generate events up to that ~55 / sec line. Once we have more production information from partners we can determine if we should test to a higher threshold, but for now I'm closing these tasks out.