Best practice for communication failure with file-based queue

[dineshbvadhia]

I have a multi-threaded file-based queue. The Producer is a remote file. The remote file locations and connections vary.

Using the multi-threaded test_queue as the example,
https://github.com/peter-wangxu/persist-queue/blob/46451510228447358e997bc9d7365b874a2c4a05/persistqueue/tests/test_queue.py#L133
... what is the best practice in case of communication failure?

[peter-wangxu]

Any more information?

[dineshbvadhia]

Through an api, a user kicks-off a remote sftp file transfer. The sftp connection details are passed to the persistqueue Producer.

The sftp communication could fail completely or it could slow down or other issues.

In case of partial or total communications failure, what is the best practice wrt persistqueue?

[dineshbvadhia]

Using test_multi_threaded() as the example, one solution is to wrap the calling function (to test_multi_threaded()) with a retry decorator which re-executes it if it fails. Does this make sense?

[dineshbvadhia]

Solved problem using sqlackqueue and propagating exception handling.