Devices occasionally lose the network

[OleksandrMasliuchenkoGL]

I have a zigbee network of ~40 devices, consisting mostly of aqara devices (about ~20 aqara wired switches QBKG12LM. a few motion sensors, temperature sensors, etc), plus a few Moes TRVs. Currently the network is not behaving robustly, and I am trying to understand why. Particularly, some of the devices occasionally stop communicating with the network.

Here is an attempt to catch one of the weird cases. I have a light switch and a motion sensor. There are a few automations in HA that turn lights on when a motion is detected. Today I noticed that lights no longer turn on. There were a few things I observed.

1. motion sensor did not send any messages, despite there was definitely a motion
2. When I click the switch it reports changed state normally

debug 2021-01-26 20:58:18: Received Zigbee message from 'vestibule_light', type 'attributeReport', cluster 'genOnOff', data '{"61440":58296064,"onOff":1}' from endpoint 1 with groupID 0
info  2021-01-26 20:58:18: MQTT publish: topic 'zigbee2mqtt/vestibule_light', payload '{"consumption":0.25,"energy":0.25,"linkquality":99,"power":0,"state":"ON","temperature":28}'
debug 2021-01-26 20:58:18: Received MQTT message on 'zigbee2mqtt/vestibule_light/set' with data 'ON'
debug 2021-01-26 20:58:18: Publishing 'set' 'state' to 'vestibule_light'
info  2021-01-26 20:58:18: MQTT publish: topic 'zigbee2mqtt/vestibule_light', payload '{"consumption":0.25,"energy":0.25,"linkquality":102,"power":0,"state":"ON","temperature":28}'
debug 2021-01-26 20:58:18: Received Zigbee message from 'vestibule_light', type 'attributeReport', cluster 'genOnOff', data '{"61440":117440562,"onOff":1}' from endpoint 1 with groupID 0
info  2021-01-26 20:58:18: MQTT publish: topic 'zigbee2mqtt/vestibule_light', payload '{"consumption":0.25,"energy":0.25,"linkquality":107,"power":0,"state":"ON","temperature":28}'
debug 2021-01-26 20:58:18: Received Zigbee message from 'vestibule_light', type 'attributeReport', cluster 'genMultistateInput', data '{"presentValue":1}' from endpoint 5 with groupID 0
info  2021-01-26 20:58:18: MQTT publish: topic 'zigbee2mqtt/vestibule_light', payload '{"action":"single","click":"single","consumption":0.25,"energy":0.25,"linkquality":113,"power":0,"state":"ON","temperature":28}'
info  2021-01-26 20:58:18: MQTT publish: topic 'zigbee2mqtt/vestibule_light', payload '{"action":"","consumption":0.25,"energy":0.25,"linkquality":113,"power":0,"state":"ON","temperature":28}'
info  2021-01-26 20:58:18: MQTT publish: topic 'zigbee2mqtt/vestibule_light', payload '{"click":"","consumption":0.25,"energy":0.25,"linkquality":113,"power":0,"state":"ON","temperature":28}'
info  2021-01-26 20:58:18: MQTT publish: topic 'zigbee2mqtt/vestibule_light/action', payload 'single'
info  2021-01-26 20:58:18: MQTT publish: topic 'zigbee2mqtt/vestibule_light/click', payload 'single'

3. But subsequent toggles were not reported to zigbee for some reason at all
4. Toggling the light via Home Assistant was not working properly. In ideal case light was toggled ~10 later, and the log shows this

debug 2021-01-26 20:58:42: Received MQTT message on 'zigbee2mqtt/vestibule_light/set' with data 'ON'
debug 2021-01-26 20:58:42: Publishing 'set' 'state' to 'vestibule_light'
error 2021-01-26 20:59:04: Publish 'set' 'state' to 'vestibule_light' failed: 'Error: Command 0x00158d0002113a74/1 genOnOff.off({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null}) failed (Timeout - 31111 - 1 - 51 - 6 - 11 after 10000ms)'
info  2021-01-26 20:59:04: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"Publish 'set' 'state' to 'vestibule_light' failed: 'Error: Command 0x00158d0002113a74/1 genOnOff.off({}, {\"timeout\":10000,\"disableResponse\":false,\"disableRecovery\":false,\"disableDefaultResponse\":false,\"direction\":0,\"srcEndpoint\":null,\"reservedBits\":0,\"manufacturerCode\":null,\"transactionSequenceNumber\":null}) failed (Timeout - 31111 - 1 - 51 - 6 - 11 after 10000ms)'","meta":{"friendly_name":"vestibule_light"},"type":"zigbee_publish_error"}'

And now the magic begins. A few seconds later I decided to re-connect the switch to the network, and magically both switch and motion sensor started working normally

debug 2021-01-26 21:00:58: Device 'vestibule_light' announced itself
info  2021-01-26 21:00:58: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"announce","meta":{"friendly_name":"vestibule_light"},"type":"device_announced"}'
debug 2021-01-26 21:00:59: Received Zigbee message from 'vestibule_light', type 'attributeReport', cluster 'genBasic', data '{"65281":{"100":0,"149":0.2533172369003296,"152":0,"154":16,"3":25,"5":34,"7":"0x0000000000000000","8":4383,"9":2816}}' from endpoint 1 with groupID 0
info  2021-01-26 21:00:59: MQTT publish: topic 'zigbee2mqtt/vestibule_light', payload '{"consumption":0.25,"energy":0.25,"linkquality":105,"power":0,"state":"OFF","temperature":25}'
debug 2021-01-26 21:01:00: Received Zigbee message from 'vestibule_light', type 'attributeReport', cluster 'genOnOff', data '{"61440":117440567,"onOff":0}' from endpoint 1 with groupID 0
info  2021-01-26 21:01:00: MQTT publish: topic 'zigbee2mqtt/vestibule_light', payload '{"consumption":0.25,"energy":0.25,"linkquality":105,"power":0,"state":"OFF","temperature":25}'
debug 2021-01-26 21:01:40: Received Zigbee message from 'vestibule_motion_sensor', type 'attributeReport', cluster 'msIlluminanceMeasurement', data '{"measuredValue":0}' from endpoint 1 with groupID 0
info  2021-01-26 21:01:40: MQTT publish: topic 'zigbee2mqtt/vestibule_motion_sensor', payload '{"battery":100,"illuminance":0,"illuminance_lux":0,"linkquality":107,"occupancy":false,"voltage":3025}'
debug 2021-01-26 21:01:40: Received Zigbee message from 'vestibule_motion_sensor', type 'attributeReport', cluster 'msOccupancySensing', data '{"occupancy":1}' from endpoint 1 with groupID 0
info  2021-01-26 21:01:40: MQTT publish: topic 'zigbee2mqtt/vestibule_motion_sensor', payload '{"battery":100,"illuminance":0,"illuminance_lux":0,"linkquality":107,"occupancy":true,"voltage":3025}'

Can someone explain what is happening, and how this odd behavior could be fixed?

BTW, Besides of described issues, I often notice message like Failed to execute LQI for 'vestibule_light' in the log. Not just for this device, but for other devices as well. Not sure if this is relevant, as usually these devices behave normally.

[DrTron]

Log the link quality over time and see how that behaves. The devices might simply have a weak connection.

[OleksandrMasliuchenkoGL]

Yes, I have it logged. But what should I pay attention at? What values could be considered as low?
Typically I have LQI >100 for most of the devices. Moreover these particular 2 devices are located within 3 meters from coordinator (though over the wall).

[OleksandrMasliuchenkoGL]

Here are graphs for these 2 devices.

There are a few glitches with LQI < 100, but in this particular case motion sensor was just disappeared at some point from the network

Situation for the light switch is slightly better, though still interesting. It has lost the network at 8AM, occasionally reporting that it alive, but got back to normal at 21:00 when I re-joined the switch to the network.

Unfortunately this information does not give me an understanding how I can avoid such glitches in future

[DrTron]

Everything above 30 should be good enough. Most of mine are around 50-80 and rock stable, so that can't be it...