I have been on a journey to explore Azure IoT and push the thousands of events that flow through my local MQTT broker (Mosquitto) in to Azure IOT. After my last post in using the Azure IoT SDK for Python conjunction with Paho MQTT I thought my work here was complete, but have just recently been made aware that their is native support for various Arduino devices by Microsoft and Espressif.
Before you get too excited given the requirements of such libraries, this is not going to work on your Arduino Uno, Arduino Mega 2560 and so on. Support for Azure IoT Hub is reserved for the newer generation of boards from Espressif (ESP32, ESP8266) and the Realtek Ameba D. These boards can contain megabytes, not kilobytes of RAM, multi-core CPU’s and are able to load in a TCP/IP stack, MQTT and so on.
If there is a theme for my house it is bookended with reliability and with that, its time to put my rack-mounted Raspberry Pi away and adopt a microcontroller. A Raspberry Pi, as great as it is, is a SBC (Single Board Computer) that needs to be updated, watered and feed. It uses a file system, a flash memory subsystem. How does the bode for reliability and have you ever had a corrupt file system on a microcontroller?
So like any good opinionated architect I would urge you to stop, put away your Raspberry Pi’s and and take a different approach. A microcontroller.
Today I leverage around 30 outputs on an Arduino Mega 2560 with an Ethernet and PoE shield using MQTT (The pub/sub client library) but it’s time to modernise, and given my love of ESP devices with Tasmota I decided to purchase an ESP32 for this very task.
In this post I will illustrate how to build a bridge from Mosquitto MQTT in to Azure IoT Hub using an ESP32 device.
I covered in a prior post why I am going down this path of publishing telemetry in to Azure IoT Hub, along with the various ways I have illustrated how one can go about achieving this goal.. From direct connection to Azure IOT-Hub (via MQTT and SAS tokens) through to Azure IOT-Edge running locally with MQTT and finally the SDK’s.
I have been able to achieve my goals with varying levels of success, but have a few concerns on the approaches I have tried thus far.
- Direct-Connection to Azure-IOT Hub introduces latency to the cloud.
- Authentication, from SAS tokens to X509 certificates, its not anonymous and some of my tiny devices (Tasmota) dont bode well.
- Topic structure, it is defined (devices/{DeviceID}/messsages/events/) and not free form. It means reconfiguration, which isn’t hard, but a lot of friction.
- Reliability, all solutions thus far have relied on a OS which require patching, updating and are even whilst small an administrative burden.
My goals for building a solution
- No reconfiguration of any of my MQTT devices (Home Assistant, PLC, Arduino Mega 2560, ~75 Tasmota devices).
- Bridge my existing MQTT broker (Mosquitto) in to Azure IOT.
- Run on microcontroller, as I want to be reliable.
Pretty lofty goals, you may even say I am being lazy, but the reality is I want a low friction away to derive operational intelligence from the many thousands of events each day (read below, its over 10K per day!)
What we are going to build
To overcome, the limitations described above we are going to use an ESP32 microcontroller with C++ code and a few libraries. Lets quickly talk about both of these.
ESP32
Where do I start, what is not love about this chip? The ESP32 is a modern, powerful Arduino compliant microcontroller that power many devices from my irrigation controller (Opensprinkler) through to my kids learning robot (MBot) they are either using an ESP32 or an older derivative such as an ESP8266.
The ESP32 is a series of low-cost, low-power system on a chip microcontrollers with integrated Wi-Fi and dual-mode Bluetooth. The ESP32 series employs either a Tensilica Xtensa LX6 microprocessor in both dual-core and single-core variations, Xtensa LX7 dual-core microprocessor or a single-core RISC-V microprocessor and includes built-in antenna switches, RF balun, power amplifier, low-noise receive amplifier, filters, and power-management modules. ESP32 is created and developed by Espressif Systems, a Shanghai-based Chinese company, and is manufactured by TSMC using their 40 nm process. It is a successor to the ESP8266 microcontroller.
ESP32 – Wikipedia
Lets build.
Code Summary
See the steps below as I tease out this solution or my GitHub repo for the full Arduino sketch. To give you a better understanding on how this works I will break it down in to the logical steps below required to receive messages from Mosquitto over MQTT using ‘PubSubClient’ and to then re-publish them in to Azure IoT Hub using the ‘Esp32MQTTClient’.
Step 1 – Arduino IDE – Add ESP32 to the Board Manager
The Arduino IDE does not know about the ESP32 so the very first step we need to do leverage the the Arduino IDE’s ‘Board Manager’ capability to provide support for the ESP32.
In the Arduino IDE, open ‘Preferences’ and enter in one of the following URL’s
- Stable release : https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json
- Development release : https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json
Open ‘Boards Manager’ from ‘Tools’ > ‘Board’ menu and install esp32 platform. Select your specific ESP32 board from the menu post installation.
Restart the Arduino IDE.
Step 2 – We Need A Library – PubSubClient
Whilst we now have support for the ESP32, we need to add a library that will allow us to subscribe to and receive MQTT messages from our Mosquitto broker. For this very purpose we need a MQTT library. There are many but I have used ‘PubSubClient’ in the past on other projects without any issues.
To install, ‘Tools’ > ‘Manage Libraries’ > ‘PubSubClient’
Step 3 – Author Some Code (Libraries and Variables)
After validating your board is working (I would suggest uploading a Blink sketch) we can start coding. This example is based off the ‘Examples’ > ‘ESP32 Azure IoT Arduino’ > ‘Simple MQTT’ that I have modified heavily
We need to include some libraries, we will be using the Wi-Fi (for connectivity), PubSubClient (for Mosquitto MQTT) and the ESP32MQTTClient (for Azure IoT Hub) and we will define variables for the MQTT Topic and Payload to which we will use between functions.
#include <WiFi.h>
#include <PubSubClient.h>
#include "Esp32MQTTClient.h"
const char* ssid = "Wifi SSID";
const char* password = "Wifi Password
const char* mqttServer = "IP Address of MQTT Server";
const int mqttPort = 1883;
String MQTTTopic;
String MQTTPayload;
Regarding Azure IoT Hub you will need to define your connection string. This post does not cover creating an IoT Hub or creating a device and assumed you have created this prior. See Use the Azure portal to create an IoT Hub | Microsoft Docs for more information on creating an Azure IoT Hub, adding a device and obtaining a device connection string.
//Azure IOT Hub Setup
static const char* connectionString = "*******************=";
static bool hasIoTHub = false;
Step 4 – Author Some Code (Setup Function: Connect to Wi-Fi , Azure and Mosquitto MQTT)
Our ‘setup’ function will establish connection to our LAN via Wi-Fi and then connect in to Azure where as the ‘MQTTConnect’ function not only connects to our local MQTT broker, but it defines the MQTT topics to subscribe to. You can subscribe to multiple MQTT topics by having multiple subscribe lines. You can also use MQTT wildcard filters to match events using fewer subscriptions to which I am using in this scenario.
Plus sign (+): It is a single level wildcard that matches any name for a specific topic level. We can use this wildcard instead of specifying a name for any topic level in the topic filter.
Hash (#): It is a multi level wildcard that we can use only at the end of the topic filter, as the last level and matches any topic whose first levels are the same as the topic levels specified at the left-hand side of the # symbol.
Understanding wildcards | MQTT Essentials – A Lightweight IoT Protocol (packtpub.com)
The serial monitor is handy in debugging any issues either with Wi-Fi or connecting in to Azure IoT Hub.
client.on_message = on_message
void setup() {
//Set baud rate
Serial.begin(115200);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.println("ESP32 : Connecting to WiFi...");
}
Serial.println("ESP32 : WiFi connected");
Serial.println("ESP32 : IP address: ");
Serial.println(WiFi.localIP());
//Set MQTT details
client.setServer(mqttServer, mqttPort);
client.setCallback(callback);
//Connect to Azure IOT
if (!Esp32MQTTClient_Init((const uint8_t*)connectionString))
{
hasIoTHub = false;
Serial.println("Azure IoT Hub : Initializing IoT hub failed.");
return;
}
hasIoTHub = true;
}
void MQTTConnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("MQTT : Attempting MQTT connection...");
// Attempt to connect
if (client.connect("ESP32Client")) {
Serial.println("MQTT : Connected");
// Once connected, publish an announcement...
client.publish("stat/ESP32/IP_Address","10.0.0.203");
//Subscribe to topics, one topic per line.
client.subscribe("stat/+/POWER");
} else {
Serial.print("MQTT : Failed to connect to MQTT , rc=");
Serial.print(client.state());
Serial.println("MQTT : Trying again to connect to MQTT in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}
Step 4 – Author Some Code (MQTT Call Back & Publish To Azure)
After the setup functions we now need to create a function that will listen for incoming MQTT messages that match our subscription (callback), extract the topic and payload before massaging this data and sending to Azure via another function (AzureIoTHub).
void callback(char* topic, byte* payload, unsigned int length) {
MQTTTopic = String(topic);
MQTTPayload = "";
for (int i = 0; i < length; i++) {
// Serial.print((char)payload[i]); - Use for debugging
MQTTPayload = String(MQTTPayload + (char)payload[i]);
}
}
void AzureIoTHub() {
if (hasIoTHub)
{
String tempString;
tempString = "{" + MQTTTopic + ":" + MQTTPayload + "}";
if (Esp32MQTTClient_SendEvent(tempString.c_str()))
{
Serial.println("Azure IoT Hub : Sending data to Azure IoT Hub succeed");
}
else
{
Serial.println("Azure IoT Hub : Failure...");
}
MQTTPayload = "";
MQTTTopic = "";
}
}Step 5 – Author Some Code (Our Main Loop)
The main loop is leveraging all of these functions and its logic can be best sumarised in to a few points.
Connect to MQTT if there is no connection
If there is a MQTT Topic/Message which was decoded via our ‘callback’ function send this to Azure IoT Hub and re-connect if there is no connection.
void loop() {
//Connect to MQTT and reconnect if connection drops
if (!client.connected()) {
MQTTConnect();
}
//Respond to messages received
if (MQTTTopic != "") {
Serial.println("MQTT : Topic is [" + MQTTTopic +"]");
Serial.println("MQTT : Payload is [" + MQTTPayload + "]");
AzureIoTHub();
}
client.loop();
}
Pulling It All Together
Here is a complete copy of the above, plus a bit more. You could cut and paste the below or clone my GitHub repository.
#include <WiFi.h>
#include <PubSubClient.h>
#include "Esp32MQTTClient.h"
const char* ssid = "****";
const char* password = "****";
const char* mqttServer = "****";
const int mqttPort = 1883;
String MQTTTopic;
String MQTTPayload;
//Azure IOT Hub Setup
static const char* connectionString = "****";
static bool hasIoTHub = false;
WiFiClient espClient;
PubSubClient client(espClient);
void callback(char* topic, byte* payload, unsigned int length) {
MQTTTopic = String(topic);
MQTTPayload = "";
for (int i = 0; i < length; i++) {
// Serial.print((char)payload[i]); - Use for debugging
MQTTPayload = String(MQTTPayload + (char)payload[i]);
}
}
void MQTTConnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("MQTT : Attempting MQTT connection...");
// Attempt to connect
if (client.connect("ESP32Client")) {
Serial.println("MQTT : Connected");
// Once connected, publish an announcement...
client.publish("stat/ESP32/IP_Address","10.0.0.203");
//Subscribe to topics, one topic per line.
client.subscribe("stat/+/POWER");
} else {
Serial.print("MQTT : Failed to connect to MQTT , rc=");
Serial.print(client.state());
Serial.println("MQTT : Trying again to connect to MQTT in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}
void setup() {
//Set baud rate
Serial.begin(115200);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.println("ESP32 : Connecting to WiFi...");
}
Serial.println("ESP32 : WiFi connected");
Serial.println("ESP32 : IP address: ");
Serial.println(WiFi.localIP());
//Set MQTT details
client.setServer(mqttServer, mqttPort);
client.setCallback(callback);
//Connect to Azure IOT
if (!Esp32MQTTClient_Init((const uint8_t*)connectionString))
{
hasIoTHub = false;
Serial.println("Azure IoT Hub : Initializing IoT hub failed.");
return;
}
hasIoTHub = true;
}
void loop() {
//Connect to MQTT and reconnect if connection drops
if (!client.connected()) {
MQTTConnect();
}
//Respond to messages received
if (MQTTTopic != "") {
Serial.println("MQTT : Topic is [" + MQTTTopic +"]");
Serial.println("MQTT : Payload is [" + MQTTPayload + "]");
AzureIoTHub();
}
client.loop();
}
void AzureIoTHub() {
if (hasIoTHub)
{
String tempString;
tempString = "{" + MQTTTopic + ":" + MQTTPayload + "}";
if (Esp32MQTTClient_SendEvent(tempString.c_str()))
{
Serial.println("Azure IoT Hub : Sending data to Azure IoT Hub succeed");
}
else
{
Serial.println("Azure IoT Hub : Failure...");
}
MQTTPayload = "";
MQTTTopic = "";
}
}
Seeing This In Action
Lets drop to a video to see this in working end-to-end, to validate messages are flowing in to Azure IoT Hub I can use the Azure CLI (AZ-CLI) to monitor the output coupled with the Arduino Serial monitor.
az iot hub monitor-events --output table --device-id devicename --hub-name hubname --output jsonFor the purpose of this demo, I have left a handful of messages at QoS level 2 and set LWT (Last Will and Testament) to true.
After 24 hours of running, we can see I have published 10.52K of messages in to Azure IoT Hub and there are certain ebbs and flows that occur in my house.
Conclusion
There are many ways to skin this code cat. My requirements was to publish messages in to Azure and we have been able to achieve this via different ways (I am sure there is more). Automation is a journey, which path will you take?
We illustrated a transparent side-car approach that will listen to an existing broker, on topics you desire and push these in to Azure IoT, all without making any configuration changes (the most important thing for my implementation).
This method runs on a microcontroller, consumes less than 5w of power and just works. Are there any draw backs, sure there are, right now this is one way in direction (simplex) and allows me to push messages in to Azure IoT but not receive messages back.
Personally, I like this approach, it combines the elegance of a SDK as its my code and couples the reliability of a microcontroller. It’s my code, my choices on what I do but I do understand this is not for everyone.
We now have my messages, my events, in Azure and now its time to make some friends and learn how to derive operational intelligence from visualisations through to machine learning and beyond.
Think big and happy building
Shane