Unable to load model in the object-detector-python sample reference app

[saicharithpasula]

Hello everyone,
I am trying to run the object-detector-python app in Axis camera for architecture aarch64 with a few changes to the application, but the reference MobileSSD model is failing to load in the container. I have attached the error when running docker. Can someone help me with this issue. I tried rebooting the docker daemon and also reinstalling it.

docker-compose --tlsverify --host tcp://%DEVICE_IP%:%DOCKER_PORT% --env-file ./config/env.%ARCH%.%CHIP% up
[+] Running 6/6

• Network object-detector-axis_default Created
• Volume "object-detector-axis_inference-server" Created
• Volume "object-detector-axis_acap_dl-models" Created
• Container object-detector-axis-inference-server-1 Created
• Container object-detector-axis-acap_dl-models-1 Created
• Container object-detector-axis-object-detector-python-1 Created
  Attaching to object-detector-axis-acap_dl-models-1, object-detector-axis-inference-server-1, object-detector-axis-object-detector-python-1
  object-detector-axis-acap_dl-models-1 | COPYRIGHT
  object-detector-axis-acap_dl-models-1 | coco_labels.txt
  object-detector-axis-acap_dl-models-1 | ssd_mobilenet_v2_coco_quant_postprocess.tflite
  object-detector-axis-acap_dl-models-1 | ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite
  object-detector-axis-acap_dl-models-1 exited with code 0
  object-detector-axis-inference-server-1 | ERROR in Inference: Failed to load model ssd_mobilenet_v2_coco_quant_postprocess.tflite (Could not load model: Unable to run warmup job: Failure when invoking interpreter)
  object-detector-axis-inference-server-1 exited with code 1
  object-detector-axis-object-detector-python-1 | object-detector-python connect to: unix:///tmp/acap-runtime.sock
  object-detector-axis-object-detector-python-1 | <_InactiveRpcError of RPC that terminated with:
  object-detector-axis-object-detector-python-1 | status = StatusCode.UNAVAILABLE
  object-detector-axis-object-detector-python-1 | details = "failed to connect to all addresses"
  object-detector-axis-object-detector-python-1 | debug_error_string = "{"created":"@1679679623.341809380","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3217,"referenced_errors":[{"created":"@1679679623.341806660","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":165,"grpc_status":14}]}"

[Corallo]

Hello @saicharithpasula

This doesn't seem a firmware problem, but just in case, check that the firmware in your camera is the latest (11.3). You can do that from the gui, going on the maintenance tab.
Then, if that doesn't solve the problem, ssh into the camera and run journalctl -u larod, we can get more information about the problem that way.

[saicharithpasula]

Hi Corallo, thank you for your help.

The firmware of the camera is 11.3.57. The journalctl -u larod command gives me the following error

Mar 24 10:12:50 axis-b8a44f6cba26 larod[293289]: Created a new session ID: 1, cl ient: :1.1887
Mar 24 10:12:51 axis-b8a44f6cba26 sh[293289]: WARNING: Fallback unsupported op 3 2 to TfLite
Mar 24 10:12:52 axis-b8a44f6cba26 sh[293289]: ERROR: Failed to run graph
Mar 24 10:12:52 axis-b8a44f6cba26 larod[293289]: Node number 99 (Vx Delegate) fa iled to invoke.
Mar 24 10:12:52 axis-b8a44f6cba26 larod[293289]: Session 1: Unable to run warmup job: Failure when invoking interpreter
Mar 24 10:12:52 axis-b8a44f6cba26 larod[293289]: Session 1 killed since client's (:1.1887) connection has been lost
Mar 24 10:40:17 axis-b8a44f6cba26 larod[293289]: Created a new session ID: 2, cl ient: :1.1892
Mar 24 10:40:17 axis-b8a44f6cba26 sh[293289]: WARNING: Fallback unsupported op 3 2 to TfLite
Mar 24 10:40:17 axis-b8a44f6cba26 sh[293289]: ERROR: Failed to run graph
Mar 24 10:40:17 axis-b8a44f6cba26 larod[293289]: Node number 99 (Vx Delegate) fa iled to invoke.
Mar 24 10:40:18 axis-b8a44f6cba26 larod[293289]: Session 2: Unable to run warmup job: Failure when invoking interpreter
Mar 24 10:40:18 axis-b8a44f6cba26 larod[293289]: Session 2 killed since client's (:1.1892) connection has been lost
Mar 24 12:32:35 axis-b8a44f6cba26 larod[293289]: Created a new session ID: 3, cl ient: :1.1972
Mar 24 12:32:35 axis-b8a44f6cba26 sh[293289]: WARNING: Fallback unsupported op 3 2 to TfLite
Mar 24 12:32:36 axis-b8a44f6cba26 sh[293289]: ERROR: Failed to run graph
Mar 24 12:32:36 axis-b8a44f6cba26 larod[293289]: Node number 99 (Vx Delegate) fa iled to invoke.

[saicharithpasula]

I have already tried removing the volumes using docker-compose down. It didn't work...
I am getting the same error

docker-compose --tlsverify --host tcp://%DEVICE_IP%:%DOCKER_PORT% --env-file ./config/env.%ARCH%.%CHIP% down --volumes
[+] Running 6/6

• Container object-detector-axis-acap_dl-models-1 Removed 0.1s
• Container object-detector-axis-object-detector-python-1 Removed 0.2s
• Container object-detector-axis-inference-server-1 Removed 0.1s
• Volume object-detector-axis_inference-server Removed 0.0s
• Volume object-detector-axis_acap_dl-models Removed 0.0s
• Network object-detector-axis_default Removed 0.2s

C:\Users\Labcam\Desktop\Dwell_time\scout_cv_dwell\object-detector-axis>docker-compose --tlsverify --host tcp://%DEVICE_IP%:%DOCKER_PORT% --env-file ./config/env.%ARCH%.%CHIP% down --volumes -v
[+] Running 2/0

• Volume object-detector-axis_inference-server Removed 0.0s
• Volume object-detector-axis_acap_dl-models Removed 0.0s

C:\Users\Labcam\Desktop\Dwell_time\scout_cv_dwell\object-detector-axis>docker-compose --tlsverify --host tcp://%DEVICE_IP%:%DOCKER_PORT% --env-file ./config/env.%ARCH%.%CHIP% up
[+] Running 6/6

• Network object-detector-axis_default Created 0.1s
• Volume "object-detector-axis_acap_dl-models" Created 0.0s
• Volume "object-detector-axis_inference-server" Created 0.0s
• Container object-detector-axis-acap_dl-models-1 Created 1.2s
• Container object-detector-axis-object-detector-python-1 Created 1.3s
• Container object-detector-axis-inference-server-1 Created 1.3s
  Attaching to object-detector-axis-acap_dl-models-1, object-detector-axis-inference-server-1, object-detector-axis-object-detector-python-1
  object-detector-axis-acap_dl-models-1 | coco_labels.txt
  object-detector-axis-acap_dl-models-1 | ssd_mobilenet_v2_coco_quant_postprocess.tflite
  object-detector-axis-acap_dl-models-1 | ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite
  object-detector-axis-acap_dl-models-1 exited with code 0
  object-detector-axis-inference-server-1 | ERROR in Inference: Failed to load model ssd_mobilenet_v2_coco_quant_postprocess.tflite (Could not load model: Unable to run warmup job: Failure when invoking interpreter)
  object-detector-axis-inference-server-1 exited with code 1
  object-detector-axis-object-detector-python-1 | object-detector-python connect to: unix:///tmp/acap-runtime.sock
  object-detector-axis-object-detector-python-1 | <_InactiveRpcError of RPC that terminated with:
  object-detector-axis-object-detector-python-1 | status = StatusCode.UNAVAILABLE
  object-detector-axis-object-detector-python-1 | details = "failed to connect to all addresses"
  object-detector-axis-object-detector-python-1 | debug_error_string = "{"created":"@1679849518.623034088","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3217,"referenced_errors":[{"created":"@1679849518.623031368","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":165,"grpc_status":14}]}"

[Corallo]

Also after factory reset (and re-updating the firmware)?

[saicharithpasula]

Soft restarting the camera from the GUI resolved the issue. Do you think the issue is because the camera was not detecting the new volumes properly until it was restarted?

[Corallo]

It is not expected. As long as you run docker-compose .... down -v it should erase the old volume.
If you find a way to reproduce systematically this issue, let us know.

Otherwise, just make sure to run the down -v command after stopping your container.

[saicharithpasula]

Sure Corallo, I will let you know if I encounter this again or get a way to reproduce this.
Thank you for your help.