Identus Cloud Agent Scalability - Background Jobs

[essbante-io]

This Epic aims to investigate ZIO Stream and ZIO Kafka as a replacement for the mechanism behind the execution of background jobs.

Currently, each background job is a recurrent ZIO effect created using ZIO#repeat that processes a configurable number of records in parallel during each iteration. This will inevitably lead to conflicts and race conditions as soon as we run two instances of the cloud agent in parallel, as the same record will be processed by those two agents sharing the same DB. The related flows will be executed multiple times, and the same DIDComm message will be sent several times to the DIDComm peer.

This can be solved using a message queueing system like Kafka and leveraging Kafka's consumer groups capability to concurrently distribute the load among multiple agent instances.

At a later stage, using a queuing mechanism like Kafka would allow us to better control - and allocate resources to - the execution of the resource-intensive tasks that put pressure on the system (e.g., generation of AnonCreds credential definitions via the Rust library).

[bvoiturier]

Implementation completed and merged into the Identus Cloud Agent main branch via the following PR.