Occasional inaccessible socket with simultaneous calls

[jamesturner246]

Hi drunc devs. Myself and @cc-a Ran into a bit of a bump today whilst working with the drunc process manager Docker image.

Occasionally the socket is inaccessible with error 500 returned. The issue occurs sporadically when two or more gRPC calls occur in quick succession. It's not disrupting our work, and it's an opportunity to make things a bit mroe robust on our side, but thought I'd bring it up in any case.

Cheers.

Stack trace:

❯ docker compose exec app python scripts/talk_to_process_manager.py & docker compose exec app python scripts/talk_to_process_manager.py
[4] 2463345
[4]  + 2463345 suspended (tty output)  docker compose exec app python scripts/talk_to_process_manager.py
                                                                                                        Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/drunc/utils/shell_utils.py", line 204, in send_command_aio
    response = await cmd(request)
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/grpc/aio/_call.py", line 327, in __await__
    raise _create_rpc_error(
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Socket closed"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Socket closed", grpc_status:14, created_time:"2024-10-11T13:58:59.404908282+00:00"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/app/scripts/talk_to_process_manager.py", line 59, in <module>
    val = asyncio.run(main())
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/src/app/scripts/talk_to_process_manager.py", line 54, in main
    await create_session(pmd)
  File "/usr/src/app/scripts/talk_to_process_manager.py", line 31, in create_session
    return [
           ^
  File "/usr/src/app/scripts/talk_to_process_manager.py", line 31, in <listcomp>
    return [
           ^
  File "/usr/local/lib/python3.11/site-packages/drunc/process_manager/process_manager_driver.py", line 175, in dummy_boot
    yield await self.send_command_aio(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/drunc/utils/shell_utils.py", line 207, in send_command_aio
    self.__handle_grpc_error(e, command)
  File "/usr/local/lib/python3.11/site-packages/drunc/utils/shell_utils.py", line 92, in __handle_grpc_error
    rethrow_if_unreachable_server(error)
  File "/usr/local/lib/python3.11/site-packages/drunc/utils/grpc_utils.py", line 222, in rethrow_if_unreachable_server
    raise ServerUnreachable(grpc_error._details) from grpc_error
drunc.utils.grpc_utils.ServerUnreachable: ('Socket closed', 14)

[plasorak]

Hi James, regarding this, would you be able to modify: https://github.com/DUNE-DAQ/drunc/blob/develop/src/drunc/process_manager/interface/process_manager.py#L59C18-L59C35 to

import futures
grpc.aio.server(futures.ThreadPoolExecutor(max_workers=10))

And check whether the problem persists?