Shelly BTHome Components support

[Misiu]

Shelly Gen 3 and Pro devices support BTHome Components - https://shelly.guide/bthome-components/
this way you can bind Shelly BLU devices directly to other devices and use the sensors in scripts or scenes, even without wifi.

I have a couple of LYWSD03MMC units flashed with your firmware, all set to BTHome v2, but the Shelly devices don't pick (find) them.
I'd like to ask for an additional format - Shelly BLU.
LYWSD03MMC reports temperature and humidity so as Shelly BLU H&T.
You can get LYWSD03MMC for 3€, and Shelly H&T costs around 22€, so the price difference is huge.

I'm going to order two Shelly units, so if I can help with providing logs or any other data I can gather from an Android phone or Windows 10 PC please let me know.

I'm not sure what is the difference between Shelly BTHome and BTHome v2 formats, maybe there aren't any differences, but I don't know how to check that and how to fix the issue.

The main idea is to use LYWSD03MMC directly with Shelly without internet access or any proxies or servers.

[pvvx]

Xiaomi (Mijia) also does not support unregistered devices working in the Mijia format. Perhaps Shelly also requires device registration (on Cloud?).

[Misiu]

@pvvx I'll wait for the Shelly H&T and will check that.
According to the docs, Gen 3 doesn't need wifi to read sensors.
If it will work without registering the device in the cloud and without wifi to pair the device then there is something else going on.
I'll update the issue as soon as possible.

[shaivadnai]

@Misiu , I have a Shelly 1 mini Gen3 that is picking up the Temp, Humidity and Battery sensors up from the LYWSD03MMC units with pvvx's custom firmware perfectly fine. They show up as "components" under the Shelly.

In order to pair the Shelly to the LYWSD03MMC, I needed to add the device as a component in the local Shelly Web Interface via the LYWSD03MMC's MAC. After a couple seconds/minutes (depending on advertising interval) it should appear on the Shelly device. (see screenshot below)

To view the LYWSD03MMC, you can look at the local Shelly Web Interface where it is listed as a device with sensors on the Dashboard and Components pages. (see below)

If you are looking at the Shelly Cloud interface, you need to navigate to the "components" submenu on the Shelly device you're connecting to. (see below)

At the moment, I'm working on checking if I can get the Shelly Cloud interface to thinking that the LYWSD03MMC is a Shelly BLU H&T so that it can be added as a "first class" Shelly Device with all the UI integrations that entails. As is, Shelly does not support Dynamic Components as "first class" devices, but plans on doing so (they have no eta, unfortunately the ticket I opened with them is not public).

Currently, our developers are working hard to implement the possibility to use a virtual component as a new device.

This will allow using those "new" devices (virtual components) via a scene with another device + thermostats, and basically have all the Cloud functions as a Shelly device.

However, this is very complex, and our developers will need more time to implement this innovative solution.

There is currently no ETA, however it will be announced in our Facebook group, forum and YouTube once its released.

Best Regards,
Shelly Support Team

[Misiu]

Thank you for the reply.
I thought I should use the Scan function and the device should be found automatically, but it didn't.
After adding the device via MAC I got this:

but after waiting for a while the values updated:

I'm waiting for Shelly BLE H&T and I'll check if the Scan button will pick it up. If yes, then maybe we can make changes, so that LYWSD03MMC is picked.

If I understand correctly, currently we can't use Shelly BLE H&T and LYWSD03MMC in thermostat functionality.

I want to mount Shelly Mini 1 Gen3 and LYWSD03MMC in my grandparents' home (actually 6 pairs of such devices) and use them as a thermostat controller. So I think I must wait for new firmware from Shelly.

Did you manage to use the LYWSD03MMC values in a Shelly script? This would be a temporary solution for now.

[shaivadnai]

I haven't tried to get the value from a BTHome Device, but I have a script which gets and sets the values on Virtual Components (the other kind of Dynamic Component) while parsing the values from raw Bluetooth packets.

Here's the gist

It should be a decent example of how to read events (temp changes) from a Dynamic Component, though you'll need to check the exact syntax for BTHome Sensors.

I'm trying to achieve the same goal as you, but for a larger set of people. The reason I'm moving away from the gist approach is because it requires users to go to a specific IP on their local network in order to change the temperature or need to navigate through a bunch of sub menus in Shelly Cloud.

If you're just building this thermostat for one household, it's likely more maintainable (arguable) and user friendly to set them up with home assistant - this is what I do for myself and it's worked great.

If I get Shelly to accept LYWSD03MMC as a first class component, I'll update this thread. Didn't have much progress yesterday - as you suspected, Shelly doesn't follow the entire BTHome standard and I ran into issues with the emulation. If you could share some of the advertisements and data beacon packets from the Shelly H&T you're purchasing, it would be appreciated. Thanks!

[Misiu]

@shaivadnai thanks for the update, just tell me what to do and I'll be more than happy to help.
The package just arrived and I added the Shelly BLE H&T using the scan functionality without any issues.

Here is the output from the Shelly BLE Debug app:

[13:33:24] 7C:C6:B6:57:CE:7D -- SBHT-003C's model determined as SBHT-003C and GEN:GBLE and NAME:ShellyBLUHT-7cc6b657ce7d, DPK:978185, ID:XB137192904576637
[13:33:24] [BLE] --- [00:packet_id:int] VALUE: BD, RESULT: 189 ---
[13:33:24] [BLE] --- [01:battery:int] VALUE: 64, RESULT: 100 ---
[13:33:24] [BLE] --- [2E:humidity:int] VALUE: 39, RESULT: 57 ---
[13:33:24] [BLE] --- [3A:button:int] VALUE: 01, RESULT: 1 ---
[13:33:24] [BLE] --- [45:temperature:short] VALUE: 00-E9, RESULT: 233 ---
[13:33:24] Device status: {"packet_id":"189.0","battery":"100.0%","humidity":"57.0%","button:0":"single_press","temperature":"23.3°C","raw":"00,BD,01,64,2E,39,3A,01,45,E9,00"}

Here raw data from BLE scanner app:

0201060D16D2FC4400E201642E3945ED000A08534248542D3030334310FFA90B0111000B03000A7DCE57B6C67C

Mode device information:

Hope this helps

[shaivadnai]

Are you, per chance, able to add it to the Shelly cloud directly instead of locally? Does Shelly cloud treat the H&T as a first class component while connected to the Shelly 1 Mini Gen 3 or the to the cloud via the S1MG3?

If not, then there's no difference in functionality between the Shelly H&T and the LYWSD03MMC as configured right now. Right?

[Misiu]

currently, I can add LYWSD03MMC as a Bluetooth (BTHome) device by manually entering the MAC address, while BLE H&T was found using Scan functionality.
Locally they look the same.
I'm trying to script a value change on the sensor, but I can't find an API for that. There is Virtual.getHandle but I don't see API for Bluetooth devices.

I didn't add BLE H&T to the cloud yet, I can do this, but it must be done via Shelly App. You enable Bluetooth on your phone, then you scan for a new device and it is somehow paired with your account. Then the device is available in the app as a separate device.
Shelly App isn't catching LYWSD03MMC, probably because they aren't Shelly devices :) I'm wondering if we can mimic BLE H&T somehow and use it as BLE H&T

[shaivadnai]

@Misiu, thanks for confirming it's possible to add H&T as a standalone cloud device - I have to log off for a bit, but I'm going to work on rewriting/adding to pvvx's firmware to be compatible with Shelly's standard so we can attempt to add it via the app.

Thanks for the additional details regarding the BLU H&T's payload, I can see some misunderstandings I had regarding their spec and will work on rectifying them when I have some more time. I'll push a fork when I have something that works so you can try testing.

@pvvx thanks for making this project so easy to work on - the Dev Container was awesome and after pulling the image I was up and running in seconds!

[shaivadnai]

I'm trying to script a value change on the sensor, but I can't find an API for that. There is Virtual.getHandle but I don't see API for Bluetooth devices.

@Misiu I haven't done this either, but try using the generic Shelly.addEventHandler() to see if some of the events coming through are related to BTHome Devices.

If that doesn't work, calling Shelly.getComponentStatus() in a loop will likely be able to retrieve the Sensor's status, though it's not particularly elegant

If none of the above works, Shelly.call() will likely allow you to use the BTHomeSensor.GetStatus RPC locally and pass it into a callback function of your choosing.

[Misiu]

@shaivadnai thank you for the help.
I've managed to create this script:

/**************** START CHANGE HERE ****************/
let CONFIG = {
    Hysteresis: 0.5,
    RoomTempSensorId: 201,
    TargetTempNumberId: 200,
    SwitchId: 0
};
/**************** STOP CHANGE HERE ****************/

let currentTemperatureSensorName = "bthomesensor:" + JSON.stringify(CONFIG.RoomTempSensorId);
let targetTemperatureNumberName = "number:" + JSON.stringify(CONFIG.TargetTempNumberId);
let switchName = "switch:" + JSON.stringify(CONFIG.SwitchId);
let targetTemperatureSlider = Virtual.getHandle(targetTemperatureNumberName);

function controlOutput(roomTemp, targetTemp) {
    var heatingStatus = Shelly.getComponentStatus(switchName).output;

    if (heatingStatus === false && roomTemp <= (targetTemp - CONFIG.Hysteresis)) {
        Shelly.call("Switch.Set", {
            id: 0,
            on: true
        });
        print("Room temperature: " + roomTemp + "°C, Heating turned ON");
    } else if (heatingStatus === true && roomTemp >= (targetTemp + CONFIG.Hysteresis)) {
        Shelly.call("Switch.Set", {
            id: 0,
            on: false
        });
        print("Room temperature: " + roomTemp + "°C, Heating turned OFF");
    }
}

Shelly.addStatusHandler(function(event, user_data) {
    if (event.component !== currentTemperatureSensorName) return;
    if (typeof event.delta === "undefined" || typeof event.delta.value === "undefined") return;

    var roomTemp = event.delta.value;
    var targetTemp = targetTemperatureSlider.getValue();

    controlOutput(roomTemp, targetTemp);
}, null);

Shelly.addStatusHandler(function(event, user_data) {
    if (event.component !== targetTemperatureNumberName) return;
    if (typeof event.delta === "undefined" || typeof event.delta.value === "undefined") return;

    var roomTemp = Shelly.getComponentStatus(currentTemperatureSensorName).value;
    var targetTemp = event.delta.value;

    controlOutput(roomTemp, targetTemp);
}, null);

it works, but I must test it before I use in in a "production" environment.

[shaivadnai]

I have bricked my device and seem to be unable to resurrect it via UART (which is what I have on hand).

If I can bring it back again, I'll continue working on it - but for now I'm probably going to take a break.

This is how I interpreted the packet you got from the Shelly H&T:

Raw Packet 0201060D16D2FC4400E201642E3945ED000A08534248542D3030334310FFA90B0111000B03000A7DCE57B6C67C

1. 020106 - BT Home Flags (always the same)
2. 0D - Length of BTHome Data (variable)
3. 16 - Length of UUID (always the same)
4. D2FC - UUID (always the same)
5. 44 - BTHome Device Information (always the same for this device)
6. 00 - packet id (always the same)
7. E2 - packet value (variable)
8. 01 - battery id (always the same)
9. E2 - battery value (variable)
10. 2E - humidity id (always the same)
11. 39 - humidity value (variable
12. 45 - temp id (always the same)
13. ED00 - temp value (variable)
14. 0A - length of name (always the same for this device)
15. 08 - local name type id (always the same for this device)
16. 534248542D30303343 local name text (always the same for this device)
17. 10 - power on boolean id
18. FF - power on value (always the same for this device)
19. A90B - power id with A9 suffix
20. 011100 - power_A9 value
21. 0B - power id
22. 03000A - power value
23. 7DCE57B6C67C -mac address

The existing BTHome firmware implementation uses a PDU function that adds a CRC at the end (and inserts the 0D in 2. for the length) - which is expected as per the BTHome spec. However, I can't see the CRC in the packet you sent.

In a poorly thought out attempt to keep the PDU function, but remove the CRC, I bricked my device.

If someone can get the device to output a packet which follows the above schema, you can probably add it to the shelly cloud app directly. I'd be curious to hear if it works.

[pvvx]

If someone can get the device to output a packet which follows the above schema, you can probably add it to the shelly cloud app directly. I'd be curious to hear if it works.

The program on the smartphone "nrf connect" can create arbitrary BLE advertising.

---

Raw Packet 0201060D16D2FC4400E201642E3945ED000A08534248542D3030334310FFA90B0111000B03000A7DCE57B6C67C

1. 020106 - BT Home Flags (always the same)
2. 0D - Length of BTHome Data (variable)
3. 16 - Length of UUID (always the same)
4. D2FC - UUID (always the same)
5. 44 - BTHome Device Information (always the same for this device)
6. 00 - packet id (always the same)
7. E2 - packet value (variable)
8. 01 - battery id (always the same)
9. E2 - battery value (variable)
10. 2E - humidity id (always the same)
11. 39 - humidity value (variable
12. 45 - temp id (always the same)
13. ED00 - temp value (variable)
14. 0A - length of name (always the same for this device)
15. 08 - local name type id (always the same for this device)
16. 534248542D30303343 local name text (always the same for this device)
17. 10 - power on boolean id
18. FF - power on value (always the same for this device)
19. A90B - power id with A9 suffix
20. 011100 - power_A9 value
21. 0B - power id
22. 03000A - power value
23. 7DCE57B6C67C -mac address

adv. structure:
structure size, id, data.

1. 02- structure size, 01 - ID: flags, 06 - data
2. 0D - structure size
3. 16 - ID: UUID16
   ...
4. 0A - structure size
5. 08 - local name type id
   ....
6. 10 - structure size
7. FF - ID: manufacturer data
8. .... manufacturer data

https://academy.nordicsemi.com/courses/bluetooth-low-energy-fundamentals/lessons/lesson-2-bluetooth-le-advertising/topic/advertisement-packet/

---

I can't see the CRC in the packet you sent.

There is no CRC in the PDU data.

The BTHome description shows the packet transmitted by the Linux driver. This is the RF packet data with additional information - watch description of "HCI". Many are mistaken on this.

There are still problems with parsing even a simple 8-bit AD.Flags in Linux (Bluez). Not to mention something else. Linux development in Bluetooth has remained at the level of 2014 (BT4.2 standard). Someone is forcibly holding back Bluetooth development in Linux (kernel, Bluez, Black).

BTHome only applies to the format of one structure in the PDU data of a packet:
size, id:16, UUID BThome, BTHome data. The rest is according to the BLE specification.

[Misiu]

@shaivadnai I'm sorry to hear that. I always thought that we can flash and restore the device with a cable connection.

The program on the smartphone "nrf connect" can create arbitrary BLE advertising.

Please let me know if I can use any Windows app or a web app (like your flasher) to help. I can gather additional packets.

I think that Shelly detects devices that are manufactured by Shelly.

After Scanning or manual adding of the Shelly BLU H&T I see that it is added with the manufacturer name, for LYWSD03MMC I got unknown.

@shaivadnai any updates from the Shelly team? Home Assistant supports virtual components, but the Shelly app has terrible support - I was able to add a virtual component to Shelly device, I see it in the app (but I must navigate to components), but I'm unable to add it to "the main view" of a component or to a room.

[Misiu]

I've just got a reply on my ticket.
I was asked to create a feature request to treat virtual components as standard devices.
Not sure if/when they will implement it, but...I think we must wait and see.

[pvvx]

After Scanning or manual adding of the Shelly BLU H&T I see that it is added with the manufacturer name, for LYWSD03MMC I got unknown.

In Shelly, the advertising packet in the PDU contains the manufacturer data (structure identifier: 0xff) and the device name. In typical BLE, the device name is transmitted as a response to an active scan request. That is, in an additional packet transmitted on request.

Custom firmware transmits the name in the main advertising package only in BT5.0 mode. Since in BT5.0 the size of the transmission package data is limited to kilobytes, and not 20..32 bytes, as in BT4.2.
Enable Bluetooth 5.0+ in the configuration. And check the ability of Shell to work with devices in the modern BLE standard. Currently, the BLE standard is version 5.4.

It is also advisable to check if the Shelly "LE Long Range" option works, which provides a 4x increase in the BLE communication range. This is available in the Bluetooth standard since version 5.0 (since 2016). If not, these devices should be moved to the trash bin as obsolete.