Understanding MQTT PUBLISH: Message Structure, QoS Levels, and Response Codes

Welcome to the second article in the MQTT 5.0 Message Series. In the previous article, we introduced the CONNECT and CONNACK messages of MQTT 5.0. Now, we will discuss the MQTT PUBLISH message used to transmit application messages in MQTT and its response messages.

Both client-to-server message publishing and server-to-subscriber message forwarding require the use of the PUBLISH message. The topic determining the message flow, the actual content of the message, and the QoS level are all included in the PUBLISH message.

In the process of message transmission between the client and server, in addition to the PUBLISH message, the PUBACK, PUBREC, PUBREL, and PUBCOMP messages are also used to implement the QoS 1 and QoS 2 mechanisms in MQTT. In this article, we will explore the composition of these five messages.

Example Message: MQTT PUBLISH

We use the MQTTX CLI to publish three messages with different QoS levels to a public MQTT server and capture the MQTT messages exchanged between the client and server using the Wireshark tool. In a Linux environment, you can use the tcpdump command to capture the messages and then import them into Wireshark for analysis.

Here are the MQTTX CLI commands used in this example. To display the attribute fields of the PUBLISH message, the command also sets the Message Expiry Interval and Response Topic properties:

Language: mqttCopy

mqttx pub --hostname broker.emqx.io --mqtt-version 5 \  --topic request --qos 0 --message "This is a QoS 0 message" \  --message-expiry-interval 300 --response-topic response

Below is the MQTT PUBLISH message with QoS 0 captured by Wireshark from the MQTTX CLI:

Language: mqttCopy

30 31 00 07 72 65 71 75 65 73 74 10 02 00 00 01 2c 08 00 08 72 65 73 70 6f 6e 73 65 54 68 69 73 20 69 73 20 61 20 51 6f 53 20 30 20 6d 65 73 73 61 67 65

This string of hexadecimal bytes corresponds to the following message content:

01publishpacket.png

When we simply change the QoS option in the MQTTX CLI command to set the message QoS level to 1, we will see the server reply with a PUBACK message after receiving the PUBLISH message. Their message data are as follows:

Language: mqttCopy

Client  -- PUBLISH (32 33 00 .. ..)    ->  ServerClient  <- PUBACK  (40 04 64 4a 10 00) --  Server

At this point, the first byte of the PUBLISH message changes from 0x30 to 0x32, indicating that this is a QoS 1 message.

The structure of the PUBACK message is relatively simple, and the Reason Code is 0x10, indicating that the message was received, but there are no matching subscribers. Once someone subscribes to the request topic, the Reason Code in the PUBACK message will change to 0x00, meaning the message was received and there are matching subscribers.

>02pubackpacket.png

Continuing to use the MQTTX CLI to publish a QoS 2 message, we will see two message exchanges occur between the client and server. Wireshark will show us that these messages are PUBLISH, PUBREC, PUBREL, and PUBCOMP, and they all share the same packet identifier 0x11c2:

Language: mqttCopy

Client  -- PUBLISH (34 33 00 .. ..)    ->  ServerClient  <- PUBREC  (50 04 11 c2 10 00) --  ServerClient  -- PUBREL  (62 03 11 c2 00)    ->  ServerClient  <- PUBCOMP (70 04 11 c2 00 00) --  Server

How to accurately know from the hexadecimal data whether this is a PUBLISH message, its QoS level, and the Reason Code in its response message will be answered in the subsequent introduction of these messages.

MQTT PUBLISH Message Structure

“MQTT PUBLISH Fixed Header”

The fixed header of a PUBLISH message has the value of the upper 4 bits of the first byte fixed at 3 (0b0011), while the lower 4 bits are composed of the following three fields:

• DUP (Bit 3): The DUP flag is set to 1 when a client or server retransmits a PUBLISH message, indicating that this is a retransmission. The number and frequency of received DUP = 1 PUBLISH messages can reveal the current quality of the communication link.
• QoS (Bits 2 – 1): Specifies the QoS level of the message.
• Retain (Bit 0): Set to 1 to indicate that the current message is a retained message; set to 0 to indicate it is a regular message.

Following that is the Remaining Length field, which indicates the number of bytes remaining in the current message.

PUBLISH Message Structure

MQTT PUBLISH Variable Header

The variable header of a PUBLISH message contains the following fields in order:

• Topic Name: A UTF-8 encoded string indicating the information channel to which the message should be published.
• Packet Identifier: A 2-byte unsigned integer that uniquely identifies the message being transmitted; it appears in the PUBLISH message only when the QoS level is 1 or 2.
• Properties: The table below lists all available properties of the PUBLISH message. We will not elaborately introduce the purpose of each property here; you can click on the property names to view the corresponding blog:

Identifier	Property Name	Type
0x01	Payload Format Indicator	Single byte
0x02	Message Expiry Interval	4-byte integer
0x23	Topic Alias	2-byte integer
0x08	Response Topic	UTF-8 encoded string
0x09	Correlation Data	Binary data
0x26	User Property	UTF-8 string pair
0x0B	Subscription Identifier	Variable byte integer
0x03	Content Type	UTF-8 encoded string

Payload

The actual content of the application message we send is stored in the payload of the PUBLISH message, and it can carry an application message in any format, such as JSON, ProtoBuf, etc.

PUBACK Message Structure

“MQTT PUBLISH Fixed Header”

The upper 4 bits of the first byte of the fixed header are fixed at 4 (0b0100), indicating that it is a PUBACK message, and the lower 4 bits are reserved bits, all fixed at 0.

Following that is the Remaining Length field, which indicates the number of bytes remaining in the current message.

PUBACK Message Structure

MQTT PUBLISH Variable Header

The variable header of the PUBACK message includes the following fields in order:

• Packet Identifier: Unlike the PUBLISH message, the packet identifier in the PUBACK message must exist, and it is used to indicate to the counterpart which QoS 1 PUBLISH message this is responding to.
• Reason Code: A single byte unsigned integer used to indicate the publish result to the publisher of the PUBLISH message, for example, whether publication was refused due to lack of authorization. The table below lists all available Reason Codes of PUBACK messages:

Value	Reason Code Name	Description
0x00	Success	The message was accepted.
0x10	No matching subscribers	The message was accepted, but there are no matching subscribers.
0x80	Unspecified error	Indicates an unspecified error. This Reason Code is used when one party does not want to disclose the specific reason for the error or no appropriate Reason Code is available in the protocol specification to match the current situation.
0x83	Implementation specific error	The PUBLISH message is valid but not accepted by the current receiver’s implementation.
0x87	Not authorized	The PUBLISH message did not pass the server’s authorization check, possibly because the current client does not have the permission to publish messages to the corresponding topic.
0x90	Topic Name invalid	The format of the topic name is correct, but it is not accepted by the receiving party.
0x91	Packet identifier in use	The packet ID in the PUBLISH message is already in use, which usually indicates that the session state between the client and server does not match, or one party’s implementation is incorrect.
0x97	Quota exceeded	Indicates that the quota limit has been exceeded. The server may impose a sending quota on the publisher, such as forwarding a maximum of 1000 messages per day. When the publisher’s quota is exhausted, the server will use this Reason Code in the PUBACK and other acknowledgment messages to alert the other party.
0x99	Payload format invalid	The format of the payload does not match the format indicated by the Payload Format Indicator property.

• Properties: The table below lists all available properties of the PUBACK message.

Identifier	Property Name	Type
0x1F	Reason String	UTF-8 encoded string
0x26	User Property	UTF-8 string pair

Payload

The PUBACK message does not contain a payload.

PUBREC, PUBREL, PUBCOMP Message Structure

The structure of PUBREC, PUBREL, and PUBCOMP messages is basically consistent with that of PUBACK, with differences mainly in the value of the message type field in the fixed header, and the Reason Codes that can be used.

The value of the message type field is 5 (0b0101), indicating a PUBREC message; 6 (0b0110) for a PUBREL message; and 7 (0b0111) for a PUBCOMP message.

PUBREC, as the acknowledgment message for the PUBLISH message in the QoS 2 message flow, can use the same Reason Codes as PUBACK. The Reason Codes available for PUBREL and PUBCOMP messages are as follows:

Identifier	Reason Code Name	Description
0x00	Success	When returned by the sender of a QoS 2 message in the PUBREL message, it indicates that the message has been released, meaning it will no longer be retransmitted. When returned by the receiver of a QoS 2 message in the PUBREC message, it indicates that the Packet Identifier used by the message has been released, and the sender can now use this Packet Identifier to send a new message.
0x92	Packet Identifier not found	Indicates that an unknown packet identifier was received, usually meaning that the session state between the current server and client does not match.

Summary

The topic in a PUBLISH message determines the message flow, while the QoS determines the message reliability and also dictates which messages will be used during transmission—PUBACK messages for QoS 1 messages, PUBREC, PUBREL, and PUBCOMP messages for QoS 2 messages. When the QoS is greater than 0, a packet identifier is also needed to associate the PUBLISH message with its response message.

The payload of the PUBLISH message does not restrict the data type, allowing us to transmit application messages in any format. Additionally, properties can accommodate needs in various scenarios, such as reducing the size of each message with topic aliases and setting expiration times for time-sensitive messages with message expiry intervals.

The response message of the PUBLISH message not only indicates to the sender that the message has been received, but can also further indicate the publishing result through the Reason Code. Therefore, when the subscriber side cannot receive a message, we can troubleshoot the issue using the Reason Code in the response message received by the publisher.

This concludes the introduction to the MQTT PUBLISH and its response messages. In the next article, we will continue to explore the structure and composition of subscribe and unsubscribe messages.