Training api loses connection to rabbit

[sushidelivery]

Hello @faymanns !

Here is the issue with losing connection.

2024-01-11 16:18:39.376809+00:00 [warning] <0.2217.0> closing AMQP connection <0.2217.0> (1XX.1X.0.X:XXXXX -> 1XX.1X.0.X:XXXX, vhost: '/', user: 'ttoolai'): 
2024-01-11 16:18:39.376809+00:00 [warning] <0.2217.0> client unexpectedly closed TCP connection

Rabbit expects the acknowledgments regularly, but the training takes so long, so it doesn't send it for a long time. Could you please have a look on it?

Thank you so much!

[sushidelivery]

Hello !

Just wanted to let you know that I've put in a quick fix for now. You can check it out in this commit: 3bbd41a.

So RabbitMQ is looking for heartbeat signals from the services to make sure they're up and running. For now, I've switched off the heartbeat thing, so RabbitMQ won't be waiting for those signals anymore.

But hey, this is just a quick fix. If you've got a better way to handle it, feel free to implement it!