Gunnebo Business Solutions, MQTT, Protocols

MQTT for Dummies

In this article, I will be discussing one of the most trending topics in IoT. I will take you through a beginner level tutorial on MQTT which is currently the most used protocol in IOT projects. 

MQTT Message Queuing Telemetry Transport-1_low

MQTT stands for Message Queueing Telemetry Transport Protocol. To put MQTT in a nutshell, it is  “A lightweight event and message-oriented protocol allowing devices to asynchronously communicate efficiently across constrained networks to remote systems”. I know that this doesn’t really help much. So let’s try to decode that definition and understand what MQTT is and how to use it.

What is MQTT?

Again, for people who have no idea about MQTT, it is a protocol for machine-to-machine communication. It uses a publisher-subscriber model for communication. If you are from a programming background, you probably would have some knowledge about the publisher-subscriber model. Anyway, we will discuss the publisher-subscriber model and how MQTT works later in the tutorial.

MQTT over HTTP for IoT

Before going onto discuss how MQTT works, let’s first try to understand how it came to existence. MQTT came to exist as a replacement for HTTP because HTTP could not properly answer the challenges in IOT and M2M projects. Unlike web applications, IOT projects have some peculiar challenges. One of the main concerns is that IOT requires the event-driven paradigm. Some of the features of this event-driven paradigm are:

  • Emitting information one-to-many 
  • Listening to events whenever they happen 
  • Distributing minimal packets of data in huge volumes 
  • Pushing information over unreliable networks  

Some other challenges you face in an M2M application 

  • Volume (cost) of data being transmitted 
  • Power consumption 
  • Responsiveness 
  • Reliable delivery over fragile connections 
  • Security and privacy 
  • Scalability

MQTT was successfully able to cope with these challenges due to their features. 

Why MQTT is good for M2M and IoT applications

MQTT has unique features you can hardly find in other protocols, like:

  • It’s easy to implement in software as it is a lightweight protocol.
  • MQTT is based on a messaging technique. This makes it faster in data transmission compared to its alternatives.
  • It uses minimized data packets which results in low network usage.
  • Low power usage. As a result, it saves the connected device’s battery.
  • Most importantly it works on real-time which makes it ideal for IoT applications.

We learnt earlier that MQTT works through a publisher-subscriber model. In a P2S system, the publisher sends its messages to a topic. Then, every subscriber of that topic will receive the message. In MQTT, Broker handles the topic and messaging process while MQTT clients behave as publishers and subscribers.

Components of MQTT

To learn about how MQTT works, we have to understand some concepts in MQTT. The fundamental components of MQQT protocol are explained below.

Broker

The broker is a server that handles the communication and data transmission between the clients. It is responsible for the distribution, management and storage of data sent and retrieved by the clients. The broker acts like a centralized hub that regulates the message exchange. 

In the case where a broker breaks down, the whole communication process breaks down as there is no way for the clients to communicate with each other directly. Therefore, the Broker Bridging mechanism was introduced to prevent such cases and build a fail-safe broker network. 

There is a number of broker applications available on the internet including the popular ones; Mosquitto and HiveMQ or you can also use cloud-based brokers from cloud providers such as IBM or Azure.

Clients (Publisher, Subscriber)

These are basically the end-users who retrieve the data distributed by the broker. Each client is assigned a unique ID to identify themselves and the session when connected to the broker. A client could either be a publisher who publishes messages under a specific topic or a subscriber who receives messages relevant to a topic, at one time.

Message

These are the chunks of data sent and received by the clients. Each message consists of a command and a payload section. The command part determines the type of message and there are 14 message types available in MQQT.

Topic

This is the namespace or literally the topic that describes what the message is about. Each message gets assigned to a topic and clients can publish, subscribe or do both to a topic. The clients can also unsubscribe from a topic if they want to. MQTT topics are just strings with a hierarchical structure. 

Assume that there is a topic called “home/kitchen”. We call home and kitchen as levels of the topic while home being topper level topic than the kitchen. Also, topics can use wild cards such as ‘+’ and ‘#’.

Publish

This is the process of clients (Publisher) sending data to the broker under a topic be distributed among the clients (Subscriber) who have requested data from the same topic.

Subscribe

This is the process of Clients (Subscribers) receiving data specific to a topic they have previously subscribed to, from the Clients (Publishers) through the broker.

QOS: Quality of Service

Each message is given an integer value from 0 to 2 to specify the delivery mode. This is known as Quality of Service. There are three different types of QOS.

  • 0 (Fire and forget) – the message is delivered only once, acknowledgement not given, high-speed delivery method.
  • 1 (Acknowledgement) – the message is delivered once or several times until an acknowledgement is received. 
  • 2 (Synchronized) – the message is delivered only once, guaranteed delivery, comparatively slower.

Practical use of MQTT

Its time to do some practical things here and get used to dealing with the MQTT protocol. As you learnt previously, there are many MQTT clients developed for each programming language.  I will use Paho python MQTT client as I am a fan of Python and it is probably the best MQTT client out there. 

mosquitto-text-side-28

First, you need a broker to create an application with MQTT. One of the most popular MQTT brokers is Mosquitto. You can install it with the following command. 

sudo apt-get install mosquitto

We set up it to work on our localhost. By default, Mosquitto listens to port 1883. Next, install the MQTT client with pip command.

sudo apt-get install mosquitto

We setup it to work on our localhost. By default, Mosquitto listens to port 1883. Next, install the MQTT client with pip command.

pip install paho-mqtt

This command will install python MQTT client library on your machine. The core of the client library is the client class which provides all of the functions to publish messages and subscribe to topics. 

There are several important methods in Paho MQTT client class which you should know:

  • connect()
  • disconnect()
  • subscribe()
  • unsubscribe()
  • publish()

Each of these methods is associated with a callback

Publishing a message

One of the main tasks you do with MQTT is publishing messages. A simple code that publishes a message usually has 4 steps.

  • Import the paho.mqtt.client class
  • Creating a client instance with Client() constructor
  • Connecting to the server with connect() method
  • Publish messages with publish() message.
import paho.mqtt.client as mqtt

clientName = mqtt.Client("uniqueClientId)
clientName.connect("localhost",1883,60)
clientName.publish("TopicLevel1/test", "Your Message Here");
clientName.disconnect();

Most of the code is self-explaining. First, you create an instance of an MQTT client. Then you connect with the broker running on Localhost. Then the client publishes its message on “TopicLevel1/test” topic. After that, it disconnects from the broker.

Subscribing to a topic

You know that MQTT is not a one to one messaging protocol as it connects many devices. The trick here is that message from any device is assigned to a topic. Any devices that are subscribed to that topic will receive the message. Similarly, you can publish messages to Topics.

You can subscribe to a topic with subscribe() method in Client class. Subscribing to a topic has the same steps as to publishing messages. I am not going to repeat it as you can easily identify these steps from the code.

import paho.mqtt.client as mqtt

def on_connect(subscriber, userdata, flags, rc):
  subscriber.subscribe("topic/test")

def on_message(client, userdata, msg):
  if msg.payload.decode() == "Disconnect!":
    subscriber.disconnect()
    
subscriber = mqtt.Client("subscribeeId")
subscriber.connect("localhost",1883,60)

subscriber.on_connect = on_connect
subscriber.on_message = on_message

subscriber.loop_forever()

In this application, the client works as a subscriber. It subscribes to the topic to the broker which in this case, runs on localhost. Whenever it receives a message, it calls for the on_message() method. If the received message is disconnected, it immediately disconnects from the broker. This is a very simple use of subscriber method. You can write more complicated logic using the same callback functions. 

So in this article, you get a concise yet comprehensive idea about MQTT. Its’ time to move on to the conclusion to recall yourself as to what is the gist of the article.

Conclusion

MQTT is a lightweight, flexible and a simple but very efficient protocol that has a definite advantage over others when it comes to IoT and M2M solutions; considering its low bandwidth and low power consumption, response time and multiple usages. In conclusion, it could be said that MQTT is the best protocol so far when it comes to IOT development.

If you want to know more about MQTT, you can check the links below or have any questions, please reach out to me: bjorn.nostdahl@gunnebo.com

MQTT and ActiveMQ on RPI

MQTT for PIC Microcontrollers

 

 

Microchip, Microcontroller, MQTT, PIC24, Protocols

MQTT for PIC Microcontrollers

The IoT (internet of things) world is bursting, in 2018 there were 23.14 billion connected devices, and it is projected to get to 30.73 billion by 2020 (from statista.com).

Embedded systems are at the center of this IoT drive, smart homes, smart cars, etc. all have embedded systems as their backbone.

Global network background-1

Microcontrollers are the drivers of embedded systems. They give devices the ability to collect data from the environment, send and receive these data and execute the needed instructions or carry out specified actions. Like turning on the heater when the temperature in the room goes below a specified level.

ARM and the PIC microcontrollers are the common microcontrollers used in embedded systems and IoT. When these devices send and receive information over a network (say the internet), they do so using transfer and transport protocols that control this transfer processes.

The hypertext transfer protocol (HTTP) is the most popular communication protocol used over the internet to send and receive data. In IoT communications this protocol is still used in most applications. A more efficient protocol is the messaging queue telemetry transport (MQTT) protocol that is optimized for low connectivity and low power requirement. The MQTT protocol finds immediate application in remote locations where batteries are used and need to be conserved.

The HTTP system transfers data via the request-response paradigm. This transfer protocol requires devices querying other devices directly for data. This leads to increase in bandwidth requirement and power consumption. Since devices have to respond to requests one after the other, multiple, asynchronous and simultaneous communication cannot be effected. This comes as a disadvantage for IoT applications where multiple devices communicate at the same time. HTTP does not allow for multiple simultaneous communication, being synchronous.

The MQTT protocol solves these.

What is the MQTT protocol?

I gave a detailed description of what MQTT is in a previous post. But for this post, I’ll reintroduce just the important points.

The MQTT is a lightweight broker-based publish/subscribe messaging protocol designed to be an open, simple, and easy to implement data transfer protocol, designed to optimize bandwidth and power consumption. It is a machine-to-machine (M2M) communication paradigm that allows devices to send and receive data faster and more reliably without being connected directly.

MQTT finds immediate need where the network is expensive, unreliable or of low bandwidth. As well as when the embedded devices are of limited processor or memory resources.

The MQTT (message queue telemetry transport) protocol works in direct contrast with the hypertext transfer protocol (HTTP) which is popularly used in sending data and communicating with devices over the internet.

MQTT provides for one-to-many communication and message distribution. It is unconcerned of the sender or the content of the message, and uses the TCP/IP to provide network connectivity. Has a small transport overhead (a message sent with this protocol can have a header of as small as 2 bytes), and with features that ensures lost connections or data can be accessed.

MQTT in microcontroller connectivity

Using the MQTT protocol in microcontrollers will improve the efficiency of data transfer, reduce the power and bandwidth requirements and introduce asynchronous communication among devices. All these come in handy with the limited memory capacity of microcontrollers, need for faster and more reliable data transfer among IoT devices and increase in IoT devices in circulation and mainstream adoption of the technology.

This protocol guarantees faster, more power efficient (than the HTTP), low latency and dependency communication among devices. This is because the MQTT protocol works on a publish-subscribe paradigm. With this model, there is no direct connection and communication between network devices, instead there is a middleman, called the broker.

To use the MQTT protocol for communication with your microcontroller, a broker is required to collect and dispatch data among devices. The broker (also known as the server) facilitates the publish-subscribe model, in a similar fashion as the client-server models. The clients (that is the connected devices) subscribe to virtual channels, known as topics. Other devices that want to send out information (known as a message) will publish the information on a specified topic to the broker. The broker then distributes the message to all the clients that subscribe to that topic topic.

Telecommunication concept with abstract network structure and server room background.png

The broker is the core part of the MQTT operation. The broker is the middleman in data transfer using this protocol. The broker/server stands at the center of M2M communication. It receives messages (on a particular topic) from devices connected using the protocol and aggregates them for transfer to other devices that subscribed to the topic.

This kind of communication provides for continuous availability and distribution of data among devices. The advantage that this kind of communication provides is lack of dependence on direct M2M connection (that besets the HTTP communication system). Devices practically work and run on their own independent of the presence or interruption of other devices. This type of connection provides real time data; this is because the broker constantly publishes the received messages to connected device. Messages that were not subscribed for are destroyed, and those that have subscribers are dispatched to the devices. With this, any interruption that occurs with the connection of one device does not affect the entire network, instead, all the messages sent while it was away are retained and push to its when it comes back on the network. The MQQT protocol is more data-centric that identity-centric.

The Programmable intelligent computer (PIC) is a Harvard architecture microcontroller that is regarded as the industry standard due to its robust features. It is a more sophisticated microcontroller than other microcontrollers like the Raspberry Pi microcontrollers, as it provides more functionalities and features than the other microcontrollers.

My previous article talked about the deployment of the MQTT protocol on the Raspberry Pi microcontroller, Gunnebo AB’s mikroPascal MQTT library puts the MQTT protocol on the PIC microcontroller.

SIU04

Our MQTT library for PIC Microcontroller brings faster and better connectivity for the PIC microcontroller. It enables PIC microcontrollers to communicate using the MQTT protocol. The MikroPascal library is built around MQTT protocol with QoS of 0, it is built on the existing TCP and IP stack based on the mikroPascal TCP/IP library, lib_enc600_v3_5, adding the MQTT layer on it.

The MQTT library is built as a wrapper around TCP/IP protocol with the purpose of providing features to publish and subscribe to text messages to specific topics, by the means of MQTT protocol.

The library carries out the following core functions:

  • Establishes TCP/IP sockets,
  • Formats MQTT packets and prepares them for transmission,
  • Extract contents from subscription messages arriving in MQTT packets,
  • Transmits MQTT packets over TCP/IP,
  • Provides test (ping) methods to test the health of connection,
  • Provides functions for subscribing to and publishing to topics as well as unsubscribing from topics.

The library reduces RAM memory requirements and provides better performance by supplying the library functions with input parameters that are pointers to arrays.

mp1

The basic work flow of the library on the PIC microcontroller is as follows. The microcontroller reserves the address for the message/information and provides pointers to this address. The MQTT library takes it from here and use the pointer to obtain or receive messages to the controller.

mp2

To communicate via the MQTT protocol on your PIC project, there are some prerequisites that your project must meet.

With the mikroPascal MQTT library, we implement this light weight protocol for the PIC microcontroller. The library can be downloaded here.

The library brings all the benefits of the MQTT protocol to PIC users enabling users to package and send data in their IoT project seamlessly, faster, with less memory requirement and wider connection with other devices.

The library can be downloaded here at the libstock repo, you can run a demo of the library to see how it works, and also check out our open source codes on github.

We welcome you to contribute to this library and please also fork it for other mictrocontrollers. If you have any questions, please reach out to me: bjorn.nostdahl@gunnebo.com

Gunnebo Business Solutions, MQTT, Protocols, Raspberry PI, Technical

MQTT and ActiveMQ on RPI

MQTT  is an ISO standard publish-subscribe-based “lightweight” messaging transport protocol for use on top of the TCP/IP protocol. It is designed for connections with remote locations where a “small code footprint” is required or high latency/ low-bandwidth networks. Andy Stanford-Clark and Arlen Nipper of Cirrus Link authored the first version of the protocol in 1999.

This week we have installed played with MQTT and ActiveMQ on a Raspberry PI. Read more about it and try it yourself below!

Preparation

To get rolling, we need a Raspberry PI3 and an SD card with Raspbian preinstalled. If you don’t have a preinstalled card, you can install Raspbian yourself by following this instruction.

Raspberry_Pi_3_Starter_Kit_
Raspberry PI 3B from element14 that we will be playing with today

Next steps after installing Raspbian:
1. Open a terminal and run these commands to update to latest Raspbian and packages:

(This will take a short while)
2. Next open a terminal and run command:
sudo raspi-config
3. Now you will have to make some smaller configs to Raspbian
4. First open Advanced Options and choose Expand Filesystem
5. Secondly open Interfacing Option and Enable SSH and VNC
6. Exit the raspi-config and it will ask you to reboot

Steps to install working Java for ActiveMQ:
1. Open a terminal using CTRL+T and then run

ActiveMQ Installation

ActiveMQ is an open-source multi-protocol message broker with a core written around JMS. It supports MQTT and maps MQTT semantics over JMS.

Here are few steps to get it installed on RPI:

1. Create folder named in a place of your choosing, preferably in home folder

2. Download compressed file from Apache website, either manually or commandline:

3. Uncompress the downloaded file

4. To run ActiveMQ

(To run as service) or

(To run in terminal)

5. To enable start from boot:
a.

b. And then this command to update rc.d

c. Now ActiveMQ should be started during boot

Node RED Installation

1. Run this command to update nodejs and Node-RED since it’s already installed in Raspbian

2. To run Node-RED, open a terminal and run

3. To enable Node-RED from boot:

4. Open browser and browse to localhost:1880 to configure flows and deploy them

Simple Flow

Here is a simple Node-RED flow illustrating subscribe/publish scenario.

mqtt

Flow above is represented by the following json:

[{“id”:”8788d23c.6f5b2″,”type”:”mqtt in”,”z”:”b9bd2972.30ee88″,”name”:”Subscribe to /test/topic”,”topic”:”/test/topic”,”qos”:”1″,”broker”:”212b12a2.a8c08e”,”x”:350,”y”:276,”wires”:[[“d2b166e9.2fda58”]]},{“id”:”d2b166e9.2fda58″,”type”:”debug”,”z”:”b9bd2972.30ee88″,”name”:”Show message”,”active”:true,”console”:”false”,”complete”:”true”,”x”:595,”y”:276,”wires”:[]},{“id”:”d4762c5e.b8f34″,”type”:”mqtt out”,”z”:”b9bd2972.30ee88″,”name”:”Publish to /test/topic”,”topic”:”/test/topic”,”qos”:”0″,”retain”:”true”,”broker”:”212b12a2.a8c08e”,”x”:576,”y”:182,”wires”:[]},{“id”:”d8358446.c5e948″,”type”:”inject”,”z”:”b9bd2972.30ee88″,”name”:”Message”,”topic”:””,”payload”:””,”payloadType”:”date”,”repeat”:””,”crontab”:””,”once”:false,”x”:332,”y”:182,”wires”:[[“d4762c5e.b8f34”]]},{“id”:”212b12a2.a8c08e”,”type”:”mqtt-broker”,”z”:””,”broker”:”localhost”,”port”:”1883″,”clientid”:””,”usetls”:false,”compatmode”:true,”keepalive”:”60″,”cleansession”:true,”willTopic”:””,”willQos”:”0″,”willPayload”:””,”birthTopic”:””,”birthQos”:”0″,”birthPayload”:””}]

Conclusion

You now have a small IoT gadget ready to start your own IoT adventure. Later we will look into how we can add more hardware and even interconnect multiple devices, such as sensors and relays.

Many thanx to Taner Yavuz and Maksym Gerasymchuk for the help with this post 🙂

You can also check out my other articles around MQTT:

MQTT for Dummies

MQTT for PIC Microcontrollers

 

 

 

DMX, Gunnebo Business Solutions, Protocols, Technical

Prolight node for Node-RED

In the previous article on Node-RED, I talked about Art-Net node for communicating with DMX devices. In this article, I will build on that, and look into a specific device implementation.

During planning of Euroshop 2017 one requirement from the product owner was to simulate sunrise and sunset with a Prolights PixieWash. Sounds quite easy, but when you start looking into the details we found that we needed to create mathematical calculations to build arc transitioning algorithm.

Continue reading “Prolight node for Node-RED”

Art-Net, DMX, Gunnebo Business Solutions, IBM International Business Machines

Art-Net and Node-RED join forces

During a visit to IBMs Client Center in Hursley, UK  I first familiarised myself with Node-RED, and I became quickly attracted to this practical platform so helpful in experimenting with new ideas and implementing them as fast working prototypes.

Later on, when preparing for the Euroshop 2017, where we really wanted to put up a show, I went for an excursion to Fellesverket Dragen where I came up with the idea about using DMX lights and moving heads connected to Node-RED and the Gunnebo Business Solution.

Continue reading “Art-Net and Node-RED join forces”