MQTT protocol analysis for embedded systems and IoT applications

MQTT protocol analysis for embedded systems and IoT applications

Overview

MQ Telemetry Transport is a Client – Server Publish/ Subscribe messaging transport protocol and was initially named after the MQ series of IBM where the protocol was first developed for proprietary embedded systems in the late 90’s.

Nowadays, MQTT is considered as the protocol name, as opposed to be an acronym previously, and is the de facto standard by OASIS and later ISO.

Throughout the development of MQTT, ease of use was always the key concern and thus the protocol are said to be easy to implement on the client side and excel at transferring data over the wire since it is lightweight, simple, and a widely adapted open standard.

Ultimately, the goal for this type of protocol is to ensure the communication channel in constrained environments, e.g. Machine to Machine (M2M) and Internet of Things (IoT) applications where a minimal code footprint is desirable and a limited network bandwidth.

Prerequisite

  1. A MQTT Client (for Desktop), e.g. MQTT.fx, a cross-platform client (Windows, macOS, Linux) built under the Eclipse Paho Project. Another option is Mosquito MQTT.

  2. (Optional) Setup Mathwork free account for ThingSpeak & MATLAB Online

If you have already understand MQTT, well done ! Skip to the second part of the article for instruction on how to setup and observe MQTT in action or read more about how the packets transmitted over the TCP/IP protocol.

Note: From this point onwards, the term MQTT is referring to the MQTT Version 3.1.1 of the OASIS standard. Official PDF can be downloaded here.

Understand how MQTT works

The Pub/ Sub pattern of MQTT provides a decoupling mechanism between the Client and Server where the Broker acting as a medium between them (this can be extrapolated from the preview image).

  • Space Decoupling: Publisher and Subscriber do not need to know each other, i.e. no exchange of IP address and port.

  • Time Decoupling: Publisher and Subscriber do not need to run at the same time, e.g. sensor update at specific interval and only retrieved to end-user when needed.

  • Synchronization Decoupling: Operations on both components do not need to be interrupted during sending or receiving.

The Fixed Header

There are three different aspects of MQTT include basic concepts (Publish/ Subscribe, Client/ Broker), basic functionalities (Connect, Publish, Subscribe) of MQTT and basic features such as Quality of Service, Retained Messages, Persistent Session, Last Will and Testament, Keep Alive and more.

Specifically, these functionalities (CONNECT, PUBLISH, SUBSCRIBE, UNSUBSCRIBE and DISCONNECT, etc.) are defined in a MQTT Control Package Type and comprise the Fixed Header (bit 7 - bit 4), i.e. the protocol first header.

Fixed Header format

In general, the protocol has three different headers vary by the package type.

Structure of an MQTT Control Packet

Side note: Full list of Control Packet Types.

The Variable Header

A Variable Header includes a 2 byte Packet Identifier field and resides between the Fixed Header and the Payload. Although, not every Control Package requires the identifier field.

An example of Variable Header of the CONNECT packet with Protocol Length, Protocol Name, Protocol Level, Connect Flag, and Keep Alive fields

For the above example, every bits encoded in hexadecimal format, e.g. Keep Alive field of 60 seconds ().

The Payload

A Payload is the final part of the packet and also optional depends on the Control Package type. In the case of the PUBLISH packet this is optional and commonly named as the Application Message, whereas it’s required in the SUBSCRIBE package and contain Topic Filter, Quality of Service.

An example of Payload of the CONNECT packet with Client Identifier fields

Quality of Service

MQTT Protocol ensures high delivery guarantees with 3 levels of QoS serve to the constraint of embedded system communication in previous discussion.

  • QoS 0: Guarantees a best effort delivery but not result.

Image courtesy of MATLAB MQTT API

  • QoS 1: Guarantees that a message will be delivered at least once.

Image courtesy of MATLAB MQTT API

  • QoS 2: Guarantees that each message is received only once by the counterpart

Image courtesy of MATLAB MQTT API

Beside that, MQTT also provides other mechanism for quality control.

  1. Last will means in case of unexpected disconnection of a client all subscribed clients will get a message from a broker.

  2. Testament and Retained message means that a newly subscribed client will get an immediate status update.

Implement MQTT for practical uses

Now that you have an idea of MQTT, let’s examine it in action using MQTT.fx to establish the connection and Wireshark to capture the packet.

In MQTT.fx Setting tab, select “MQTT Broker” under Profile section and fill in ThingSpeak Server info as above

Note: If you have a MathWork ThingSpeak/MATLAB account, sign in under User Credentials sub-menu (Refer to Prerequisite). Otherwise, leave at default.

Under tab Subscribe, use the below syntax to Subscribe any Public channel

1

channels/<channel_ID>/subscribe/json

Once you receive the channel feed, it means that

  1. The connection was requested & Server Acknowledged Request

  2. Subscription requested & Server Acknowledged Subscription

  3. Message Published from Server to MQTT.fx client

This can be verified in Wireshark by select “Start Capturing Packet” icon in the Top bar and re-do every steps.

MQTT packet captured while using MQTT.fx to the Broker on ThingSpeak server

Side note: Use “MQTT” (protocol) or “1883” (port) as the filter for better analysis.

Under network layer section, it can seen that MQTT indeed working on top of the TCP/IP layer. Beside that, Wireshark provides deeper look into the protocol and exactly how the packet was formed.

Analysis of MQTT CONNECT packet headers and flags setup

This can breakdown into several headers as discussed, i.e. Fixed Header with Control Packet Type, Variable Header with Protocol Length/ Name, Connect Flags, Keep Alive fields and lastly the Payload with Client ID and Authentication.

Also, these information can be interpreted in raw hexadecimal format too.

Same MQTT packet in raw HEX code with Headers highlighted and Fields separated.

Tips: You can hover over the HEX code in Wireshark to display the MQTT field that matches in the control bar at the bottom.

By using any DEC-HEX decoder, these code will yield similar result as expected.

Analysis of the same CONNECT packet in Microsoft Excel. Grab it here

Note: In Excel, the built-in function CODE() convert character to ASCII and DEX2HEX() could seamlessly achieve the task and of course modify to analyze other MQTT packets.

ThingSpeak & MATLAB Application

The great thing about ThingSpeak & MATLAB combination is that it provides an easy-to-implement approach without any restriction of further improvement.

ThingSpeak panels provide basic data collection and visualization

Many tools can be found under Apps top menu on your ThingSpeak profile with two main categories: Analytics and Actions.

For instances, you could use ThingSpeak to capture the temperature from sensor via MQTT, then import to MATLAB using script for Data Cleansing or use a Widget ThingTweet to inform public on sudden temperature changes.

On the other hand, MATLAB also provides excellent MQTT Support Toolbox for both reading and writing data directly to the ThingSpeak database.

Connect to MATLAB Online and try it out !

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

% thingSpeakRead(<channelID>, <Name>, <Value>)
[data,time] = thingSpeakRead(12397)

% Or specific Fields with Output Format flag
thingSpeakRead(12397,'Fields',[1 4],'NumPoints',10,'OutputFormat','table')

% Now try to write to your private channel
% NOTE: Example Key Value

% Update multiple fields at once 
thingSpeakWrite(<channelID>, 'WriteKey', <write-key>,...
		[field1_val, field2_val, field3_val]) 

% Or automated process for batch authentication from multiple channels
thingSpeakAuthenticate(<channelID>, <Name>, <Value>)

Tips: Use help or doc prefix in MATLAB Console Window on instruction how to implement the commands and its parameters

An Example of Bi-Histogram from ThingSpeak data imported for a Transportation Tracking application.

Troubleshooting (Extra)

  • Authorization issues in MQTT.fx

Make sure that you type in User-ID and MQTT-API-key for fields <User Name> and <Password> of User Credentials Tab respectively.

Both can be found under Account My Profile on your ThingSpeak profile.

  • Decimal-Hexadecimal Conversion

In case you could not access the Excel file for reading the content of a MQTT packet, try to use directly in MATLAB Online with the Dec2Hex built-in function.

Disclaimer

This article offers knowledge and personal opinion based on the author’s point of view which perceived at the time of writing this article.

It is intended to serve “as-is” without any guarantee to work, proceed with caution. Your inputs and feedbacks are welcome as always.

comments powered by Disqus