MQTli

MQTli is a multi-topic payload-converting MQTT cli client written in Rust.

It can be configured to automatically convert between different payload formats when reading input data for publish and outputting data for subscribe. The supported data formats and the conversion rules are listed under Supported Payload formats and conversion.

Features:

• automatically publish messages using triggers (periodically, messages on topics)

• subscribe to topics and output messages to console, to a file or to another topic

• support of different payload formats (json, yaml, protobuf, hex, base64, utf-8, raw)

• Seamlessly convert message payloads between input, payload, and output, if they differ (e.g. from protobuf to json or vice versa)

• Configuration via cli arguments and config file (yaml)

• MQTT v5 and v3.1.1

• TLS support (v1.2 and v1.3)

• Websocket support (unencrypted and TLS)

• Client authentication via username/password

• Client authentication via TLS certificates

• Last will

• QoS 0, 1, 2

How to use

1. Download the latest release for your platform from the releases.

2. Extract the executable to a folder on your local hard drive

3. Optional: Add the path to the executable to your environments PATH variable, so that you can execute the program from any folder

4. Copy the config.default.yaml to your local directory, rename it to config.yaml and adjust it with your required settings

5. Execute the command mqtli (Optionally specify the name of the config file mqtli --config-file config.default.yaml)

Configuration

The configuration can be passed to the tool via the following ways in descending priority:

1. command line arguments

2. environment variables

3. config file

If a config entry was not explicitly specified, the default values applies.

All but the topic configuration can be given as a command line argument or environment variable. The topic configuration can only be specified in the config file because it would be too complex to specify in another way.

Note

The program does not do anything really useful if no topics are specified or all specified topics are disabled. It is fine to just use the command line arguments or environment variables to connect to the broker. If you want to see subscribe to topics or publish messages, you need to configure them in the config file.

CLI arguments and environment variables

The following lists all possible command line arguments and environment variables (also available via mqtli --help):

Usage: mqtli.exe [OPTIONS]

Options:
--help                       Print help
--version                    Print version
-c, --config-file <CONFIG_FILE>  Path to the config file (default: config.yaml) [env: CONFIG_FILE_PATH=]

Broker:
-h, --host <HOST>                  The ip address or hostname of the broker (default: localhost) [env: BROKER_HOST=]
-p, --port <PORT>                  The port the broker is listening on (default: 1883) [env: BROKER_PORT=]
--protocol <PROTOCOL>          The protocol to use to communicate with the broker (default: tcp) [env: BROKER_PROTOCOL=] [possible values: tcp, websocket]
-i, --client-id <CLIENT_ID>        The client id for this mqtli instance (default: mqtli) [env: BROKER_CLIENT_ID=]
-v, --mqtt-version <MQTT_VERSION>  The MQTT version to use (default: v5) [env: BROKER_MQTT_VERSION=] [possible values: v311, v5]
--keep-alive <KEEP_ALIVE>      Keep alive time in seconds (default: 5 seconds) [env: BROKER_KEEP_ALIVE=]
-u, --username <USERNAME>          (optional) Username used to authenticate against the broker; if used then username must be given too (default: empty) [env: BROKER_USERNAME=]
-w, --password <PASSWORD>          (optional) Password used to authenticate against the broker; if used then password must be given too (default: empty) [env: BROKER_PASSWORD=]

TLS:
--use-tls <USE_TLS>
If specified, TLS is used to communicate with the broker (default: false) [env: BROKER_USE_TLS=] [possible values: true, false]
--ca-file <TLS_CA_FILE>
Path to a PEM encoded ca certificate to verify the broker's certificate (default: empty) [env: BROKER_TLS_CA_FILE=]
--client-cert <TLS_CLIENT_CERTIFICATE>
(optional) Path to a PEM encoded client certificate for authenticating against the broker; must be specified with client-key (default: empty) [env: BROKER_TLS_CLIENT_CERTIFICATE_FILE=]
--client-key <TLS_CLIENT_KEY>
(optional) Path to a PKCS#8 encoded, unencrypted client private key for authenticating against the broker; must be specified with client-cert (default: empty) [env: BROKER_TLS_CLIENT_KEY_FILE=]
--tls-version <TLS_VERSION>
TLS version to used (default: all) [env: BROKER_TLS_VERSION=] [possible values: all, v12, v13]

Last will:
--last-will-payload <PAYLOAD>  The UTF-8 encoded payload of the will message (default: empty) [env: BROKER_LAST_WILL_PAYLOAD=]
--last-will-topic <TOPIC>      The topic where the last will message will be published (default: empty) [env: BROKER_LAST_WILL_TOPIC=]
--last-will-qos <QOS>          Quality of Service (default: 0) (possible values: 0 = at most once; 1 = at least once; 2 = exactly once) [env: BROKER_LAST_WILL_QOS=]
--last-will-retain <RETAIN>    If true, last will message will be retained, else not (default: false) [env: BROKER_LAST_WILL_RETAIN=] [possible values: true, false]

Logging:
-l, --log-level <LOG_LEVEL>  Log level (default: info) (possible values: trace, debug, info, warn, error, off) [env: LOG_LEVEL=]

Config file

In addition to all configuration values from command line arguments, the topics can be configured via the config file.

See config.default.yaml for all possible configuration values including their defaults.

Topics and automatic conversion between payload formats

The general idea behind the topics configuration is that each topic on the mqtt broker is used for transporting messages of the same type and data format, but possibly different content. Even though the MQTT specification does not at all apply any restrictions how topics may be used, it is common practise to only use the same data formats for the payload of a specific topic. In case the structure or data format of the payload of a topic differs between two messages, it is recommended to use different topics for these messages.

For each topic, the following three main aspects can be configured:

1. The format of the payload of the messages on the topic

   The format is defined once for all message on the topic, assuming that the format of the payload does not change between messages. Depending on the format, several options may be passed, see Supported Payload formats and conversion.

   For example, all messages on the topic may be formatted as hex string or JSON value.

2. The display of received messages on subscribed topics

   If enabled, a subscription for the topic is registered on connect. Each subscription may have several independent outputs. Each output has a format type and a target.

   • Format type (default: Text): This may be one of the types defined in Supported Payload formats and conversion. It defines which format the received message will be displayed in. If the format type of the topic is different, an automatic conversion is attempted. If it fails, an error is displayed. See the referenced chapter to see which conversions are currently possible.

   • Target (default: Console): The target defines where the message is being printed out. Currently, the following targets are supported:

     • console: Prints the message to the stdin console.

     • file: Prints the message to a file.

     • topic: Send the payload to another topic

       Apart from the path to the output file, string for prepending or appending or the behavior for overwriting can be specified.

3. The format of messages published on the topics

   When messages are published to a topic, for example via a periodic trigger, the message may be specified in another format than the payload of the topic. If the payload format of the published message is not the same format as the payload format of the topic, the payload will automatically be converted to the payload format of the topic. If a conversion is not possible, it will fail and an error will be printed. See Supported Payload formats and conversion for possible conversions.

   For example, it might be easier to specify a binary payload as hex or base64 encoded string than as raw bytes. This way, the payload could be written directly into the config.yaml file instead of an external file (YAML files only accept UTF-8 content; a binary payload may contain invalid bytes).

One of the most important advantages of this separate definition of format types is that it is then possible to automatically convert between formats. For example:

• The payload format of the topic is protobuf

• The published messages are written as hex string for storing it directly in the config.yaml

• The received messages on subscribed topics are displayed as json and written to a file as raw (bytes)

Even though protobuf is not human-readable by itself (as it is encoded using bytes), this setup allows to read messages on the topic as human-readable json while storing received messages as original bytes in a file (for later use or whatsoever). The message to publish does not need to be stored as bytes but can be encoded to a hex string which will automatically be decoded to protobuf before being published.

Filters

Filters can optionally be applied to messages received on a subscribed topic or before publishing data to a topic. They allow additional processing of the message before sending it to the output.

The following examples show the filters used for the subscription. The filters can equally be applied to a publish entry similar to how they are specified for a subscription entry.

Filters try to convert the input data to the required payload type automatically. in case the input data cannot be converted, an error is thrown and further processing is stopped.

It is possible to manually convert to different payload formats with the appropriate filters. Usually this is not necessary.

Filter: Extract JSON via JSONPath

Extract elements or singular values from an JSON type via JSONPath.

• Name: extract_json

• Processable input types: JSON

• Output type: JSON

Table 1. Attributes of extract_json

Attribute	Description	Type	Default value
jsonpath	A valid JSONPath directive	string	""

Example

# Input JSON value:
# {
#   "name": "MQTli",
#   "description": "MQTT cli client"
# }
#
# Result is still a JSON value, but just a string:
# "MQTli"

topics:
  - topic: mqtli/json
    subscription:
      enabled: true
      outputs:
        - format:
            type: json
      filters:
        - type: extract_json
          jsonpath: $.name
    payload:
      type: json

Filter: To upper case

Convert all ascii characters to upper case.

• Name: to_upper

• Processable input types: Text

• Output type: Text

Example

# Input Text value: "MqTli"
#
# Result: "MQTLI"

topics:
  - topic: mqtli/text
    subscription:
      enabled: true
      outputs:
        - format:
            type: text
      filters:
        - type: to_upper
    payload:
      type: text

Filter: To text

Convert a message to text type. This filter can be used to transform the data in a chain of filters, see the example.

• Name: to_text

• Processable input types: Any

• Output type: Text

• No attributes

Example

# Input JSON value:
# {
#   "name": "MqTli",
#   "description": "MQTT cli client"
# }
#
# Result text value:
# "MQTLI"

topics:
  - topic: mqtli/json
    subscription:
      enabled: true
      outputs:
        - format:
            type: text
      filters:
        - type: extract_json
          jsonpath: $.name
        - type: to_text
        - type: to_upper
    payload:
      type: json

Filter: To json

Convert a message to json type. This filter can be used to transform the data in a chain of filters, see the example.

• Name: to_json

• Processable input types: Any

• Output type: JSON

• No attributes

Example

# Input YAML value:
# name: MqTli
# description: MQTT cli client
#
# Result text value:
# "MQTT cli client"

topics:
  - topic: mqtli/yaml
    subscription:
      enabled: true
      outputs:
        - format:
            type: text
      filters:
        - type: to_json
        - type: extract_json
          jsonpath: $.description
    payload:
      type: yaml

Example config: Protobuf as topic format, no TLS

This example assumes that all messages on topic mqtli/test are protobuf messages as defined in the file messages.proto with the name Proto.Message.

A properly formatted message is published on the topic every second. As the protobuf message itself is represented as binary, the input has been converted to hex format so it can be entered in the configuration. You could also use any other format, e.g. JSON, which would be more readable. Also, you can enter the data in a file and load it from there. This would allow you to enter binary data directly in the file without having to convert it before.

All messages are printed to the console formatted as YAML (the conversion from Protobuf is done automatically according to the definitions in messages.proto). Additionally, all messages are encoded to base64 and written to a file log.txt.

broker:
  client_id: "my_client_id"
  username: "yourusernamehere"
  password: "yourpasswordhere"

  use_tls: false

  last_will:
    topic: "mqtli/lwt"
    payload: "Good bye"

topics:
  - topic: mqtli/test
    subscription:
      enabled: true
      outputs:
        - format: # target is console; protobuf message will be shown as yaml
            type: yaml
        - format:
            type: base64
          target:
            type: file
            path: "log.txt"
            overwrite: false
            prepend: "MESSAGE: " # prepends the string "MESSAGE: " to the beginning of the base64 encoded message
            append: "\n" # appends a new line to the end of the message
    payload:
      type: protobuf
      definition: "messages.proto" # path to file containing message definition
      message: "Proto.Message" # package_name.message_name
    publish:
      enabled: true
      input:
        type: hex
        content: AB23F6E983 # this must be a valid protobuf message according to the payload format (encoded as hex)
      trigger:
        - type: periodic # default trigger: periodic with no count (indefinitely) and interval 1 second

Supported Payload formats and conversion

The following table lists all possible payloads and their conversion options.

Table 2. Possible conversions between payload formats

from → to	Raw	Text	Hex	Base64	JSON	YAML	Protobuf	Sparkplug
Raw	yes	yes	yes	yes	yes	yes	yes	yes
Text	yes	yes	yes	yes	yes	yes	no	no
Hex	yes	yes	yes	yes	yes	yes	yes	yes
Base64	yes	yes	yes	yes	yes	yes	yes	yes
JSON	yes	yes	yes	yes	yes	yes	yes	yes
YAML	yes	yes	yes	yes	yes	yes	yes	yes
Protobuf	yes	yes	yes	yes	yes	yes	yes	yes
Sparkplug	yes	yes	yes	yes	yes	yes	yes	yes

Many formats can be converted to each other, given that the data contains the required information for this conversion. (For example, a conversion from text to protobuf is not possible because text is not a structured format).

Errors may occur during conversion, mainly due to invalid data. In case a conversion failed (e.g. because the payload of a topic was declared as base64 but the payload of a message on that topic contained invalid base64 encoded data), an error message is shown and the processing of the message is stopped.

Raw (binary)

This payload format represents data as binary.

All formats can be converted to the raw format.

Text (UTF-8)

This payload type represents data encoded as a UTF-8 encoded text.

All formats can be converted to the text format. In case the data contains invalid UTF-8 characters, a placeholder character will be shown if the text is printed. The conversion will not fail due to invalid UTF-8 characters and the invalid characters will be contained in the result. This allows to preserve all data when converting the text into any other format.

Hex

This payload format represents data encoded as hex. Characters are represented as lower-case when printed, but may be any case when read.

Base64

This payload format represents data encoded as base64.

The used alphabet contains the following characters: A–Z, a–z, 0–9, +, /´. Padding is enabled and the character = is used for it.

JSON

This payload format represents data encoded as JSON.

If JSON data is converted from text, the text is assumed to be properly JSON formatted. If JSON data is converted from a binary format (raw, hex, base64), the decoded data is assumed to be properly JSON formatted UTF-8 text.

YAML

This payload format represents data encoded as YAML.

If YAML data is converted from text, the text is assumed to be properly YAML formatted. If YAML data is converted from a binary format (raw, hex, base64), the decoded data is assumed to be properly YAML formatted UTF-8 text.

Protobuf

This payload format represents data encoded as protobuf.

All formats, except text (because it does not contain any structural information), can be converted to protobuf. When converting from binary (or encoded formats like hex and base64), the data is assumed to be a correct protobuf message that corresponds to the given protobuf schema.

When converting a protobuf message to text, the protobuf internal text representation is used.

Eclipse Sparkplug

This payload format represents data encoded according to Eclipse Sparkplug Specification 3.0.0. All Sparkplug messages are encoded as protobuf. The protobuf schema used can be found in the file assets/protobuf/sparkplug_b.proto.

All formats, except text (because it does not contain any structural information), can be converted to Sparkplug. When converting from binary (or encoded formats like hex and base64), the data is assumed to be a correct protobuf message that corresponds to the given Sparkplug schema.

Future plans

• Single-topic clients for each subscribe and publish

  • publish one message (or the same message repeatedly) to a single topic

  • subscribe for one topic

  • this mode is only configurable via cli args

• Support MQTT5 attributes

  • user properties

  • content-type (to automatically detect the format of a topic)

  • other attributes

• Support other topics as triggers for publishing