NOTE (6-7-22): Please download the flows.json file again if you did prior to 10AM June 7th. The original one was not correct and will possibly break your node-red install. Thanks to Rohan Karamandi for catching it for me.

NOTE (6-9-22): Thanks to the help of Rohan Karamandi, the flows have been updated to now create device entities in Home Assistant! You can now see each MT and all its associated telemetry, as well as a link directly to it in dashboard!

NOTE (7-12-22): I have updated the Home Assistant auto population dashboard to filter based on entity ID vs name, so that you can rename all of your sensors as you please! To download, navigate to the section at the bottom and download the Dashboard.yaml_ zip file!

Alex’s summary and musings

To preface all of the following blog, I am NOT a programmer.

Hey folks! I have been absent for quite a while trying to get s*** (stuff) done. One of my side projects that never ends has been home automation and learning how to tie APIs and telemetry together through fun things like Node-Red and Home Assistant. As you may be aware, Meraki released the MT sensor product line a year or two ago, and added MQTT support for low latency telemetry streaming. MQTT is incredibly fun to work with, mostly due to the simplicity of the protocol and how much potential it has to change how you work with devices, services, telemetry, etc from many different sources. That being said, one thing I’ve always wanted a reason to learn/figure out, has been Home Assistant’s auto-discovery functionality for MQTT devices.

TL;DR If you dont want to read my take on HA / Node-Red etc., please skip the next section or so to where I have outlined the required packages, addons etc, and node-red flows to get going. >> The goal of this project was to figure out how can I automate onboarding of MT Sensors, and the data sent through MQTT into Home Assistant without having to do much manual work after the initial configuration.

A Quick-ish Overview of the Technologies Used

Home Assistant

Home Assistant is an open source, community driven Home Automation, IoT, etc platform that allows you to bridge together completely disparate platforms, systems, protocols, and solutions into a single pane-of-glass that is able to be installed on something as small as a Raspberry Pi, to running on old PCs or even as a Virtual Machine.

I have run Home Assistant on Intel NUCs, Ultra-small form factor PCs, VMs, and more, as I have upgraded, got bored, or in general wanted to try something new over the last 7 or so years. I have used Home Assistant to integrate Z-Wave, Zigbee, Wifi devices, as well as Cloud APIs together to allow for control, visualization, and automation in my house, and even in a few business settings to provide a simple functional front-end to interact with.
To get started with HA (home assistant) check out this link here:
https://www.home-assistant.io/getting-started/

MQTT Auto Discovery: https://www.home-assistant.io/docs/mqtt/discovery/

MQTT Mosquitto Addon (how I use it): https://github.com/home-assistant/addons/tree/master/mosquitto

Node-Red

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click. I have used node-red to learn how to program things that I never imagined was even possible including:

Creating a busy light based on my Cisco Webex Teams status, Cell-Phone call activity, and Computer’s camera and microphone state to light up a panel for my family to know not to barge into my office without knocking (however, news flash, they don’t care).
Consolidate various sensors around my house from DIY ESP devices, to Smartthings, to Meraki MT and MV cameras, and turn them into entities for Home Assistant to graph things like Temperature, Humidity, Noise Levels, Car/Human detection, Air Quality, Door open/close, Wet/Dry status etc. as well as send me alerts through Telegram, Email, Discord, Slack, and more!
Endless other things like telegrams bots to yell at you when you try and talk to them, which was an oddly interesting exercise in learning and entertainment.

To get started with Node-Red:https://nodered.org/
- Home Assistant Node-Red as an addon (how I use it): https://community.home-assistant.io/t/home-assistant-community-add-on-node-red/55023
- Node-Red MQTT: https://cookbook.nodered.org/mqtt/

Meraki MT & MQTT

Instead of getting too wordy, I would highly recommend the following link to review how MT and MQTT work together:

https://documentation.meraki.com/MT/MT_General_Articles/MT_MQTT_Setup_Guide

If you can wade through the “marketing” for useful content, there is a summary of MT somewhere in here: https://meraki.cisco.com/products/sensors/

My personal take:

Meraki MT is incredibly interesting. It effectively is the careful combination of the Meraki Dashboard, Cameras, and Access Points, with purpose built BLE IoT sensors, that provides you an incredibly easy to implement, manage, and consume, IoT platform that keeps expanding every year. Since Day 1 of getting to test these little sensors, I have been enamored with the ease of deployment and data consumption models. Initially the telemetry was only available in the dashboard and API, but has expanded to using local MQTT streaming from the BLE Gateways (MV Cameras or MR Access points) directly to your MQTT broker. This additional functionality is what spawned this entire project and blog post.

The Project Starts Here

Technology Used:

Home Assistant in Home-Assistant OS install mode

HA Add-on for Mosquitto MQTT Broker

HA Add-on for Node-Red for subscribing, re-writing, and publishing topics and information

Meraki MR/MV as an MT Gateway and MQTT publisher

Meraki MT Sensors (of all sorts) for measuring all the things

Topology Diagram:

Home Assistant Setup

In my installation of Home Assistant, I am using the completely supervised Home Assistant OS installed on an ultra-small form factor PC. The reason for using Home Assistant OS is for the awesome add-on store that makes integrating things like MQTT Brokers and Node-Red dead simple.

If you have not added the Node-Red and the Mosquitto MQTT Broker go to:

**Select Add-ons, Backups & Supervisor**

**Install the Mosquitto MQTT Broker Add-On:**

Once Installed walk through the configuration documentation and then start the container/add-on. Home Assistant creates its own user account, however make sure you configure user accounts for Node-Red, and MT via the configuration page of the add-on like so:

I suggest configuring a simple permissive topic ACL while you are learning as outlined in the documentation. For simplicity sake I have my development environment’s Topic ACL users set to allow all like so (note: this is not necessarily recommended for your day to day / production use):

user homeassistant
topic readwrite #

user mt
topic readwrite #

user node-red
topic readwrite #

Next install the Node-Red Add-On package:

Once installed go through the configurations and start the container/add-on.

Now in Home Assistant we need to configure the MQTT integration, so go to:

Configure the MQTT settings. Note: they may be prepopulated with the broker name of the add-on, and also uses a homeassistant user account to connect to mosquitto.

Make sure that discovery is enabled.

This is key to making this integration successful. This allows Home Assistant to listen for specific topic formats in MQTT that tell Home Assistant to import the defined sensor/entity into the configuration automatically. For more information on how that works (and what I used to build the node-red flows) please check out: https://www.home-assistant.io/docs/mqtt/discovery/

HACS Install (Optional but useful)

There is a third party store of integrations and useful themes etc you can add to Home Assistant through the Home Assistant Community Store. To use some of the additional things I will call out at the end of the post, install HACS through this guide: https://hacs.xyz/docs/setup/download

Configuring Dashboard to publish to your MQTT broker

Meraki Dashboard has a few places you can configure MQTT servers to support publishing topics to a broker.

Here you can see the topics that are going to be sent to the broker

That sums up the process to enable MQTT messaging in dashboard for Meraki MT. One thing to note is the source of the MQTT updates will be any supported MR (access point) or MV (camera) that is in the same network in dashboard as the MT sensors. This means there wont be a guaranteed IP address as the source for every message, but to an MQTT broker will not affect the message publishing.

Configuring the Meraki Dashboard for APIs

Once we have enabled MQTT, we will need to grab your User’s API Key so that we can resolve the names of the sensors in the node-red flow. Please follow this document here and set up your Organization for APIs and generate your API key:
https://documentation.meraki.com/General_Administration/Other_Topics/Cisco_Meraki_Dashboard_API

What does MQTT look like on the network?

Since I am a network engineer at heart, lets take a look at a packet capture of the traffic over the network:

Capture at your broker to see all possible messages being sent.
Filter the traffic based on the TCP port. For example: tcp.port == 1883

After applying the filter we can see the messages going to and from the broker

Wireshark by default does not decode the MQTT payload, so to see the information being published:

Right click the message and go to Protocol Preferences > MQ Telemetry Transport Protocol preferences…

Now you can filter based on message or topic contents. For example: tcp.port == 1883 && mqtt.topic contains “meraki/v1/mt”

As you can see here the filter gets us just the messages about MT

Time for some Node-Red action!

Node Red in this case will be attaching to the MQTT broker and subscribing to the MT topics using a wildcard. This could be constrained further to a more specific topic, but for the purpose of the integration we want to capture all topics and data. Below I am providing the node-red flows that we will use to pull the information from dashboard on the sensor names, and then convert the topics to discoverable topics in Home Assistant. One thing we will need to do first is add a few npm packages that are part of the flow including:

node-red-contrib-string (Used to make parsing and cleanup of the topic easy)
node-red-contrib-secret (Used to protect the Meraki API Key)

To add these navigate to:

Manage Palette lets you add and remove packages

Once in the install tab search for the package names above and install. Once installed the nodes will be available on the side bar like so:

Importing the flows into Node-Red

Download the below ZIP file. It contains a text file with the flows in JSON format. DO NOT open in the native MacOS text editor. I suggest downloading Atom, Sublime, or Visual Studio Code as those editors will not replace back ticks and quotations with other characters like TextEdit does.

flows.6-9-22 Download

Now to import this flow named MT3.0 (you can rename it) Go to the sandwich menu in the top right > Import:

The next page will bring you to an import page where you can import the file or paste the contents of the file and click Import:

While unlikely, If you have any nodes named the same with the same config, there may be a window that pops up asking you if you still want to import. I suggest reviewing and selecting to import the flows:

In this case I re-used the HTTPS node I created

The next section we will be editing some of the nodes so that we can build the node mac/serial to name database, and attach to the MQTT instance in Home Assistant.

Editing the Global key store flow in Node-Red

The global key store flow is used to create a table in memory of the Meraki device MAC and serial to name mappings, so that our nodes in Home Assistant have their dashboard names instead of their MAC addresses. This is not at all required but is an inherent part of both the MQTT flows I will walk through later in the post, and this flow here. It does make it so you can name, rename, etc in dashboard, and have it reflect in Home Assistant.

First double click on the “Set the Meraki Org_ID” node and put in your Meraki Organization ID in the variable.

If you are unfamiliar with how to gather this please see: https://developer.cisco.com/meraki/api-v1/#!getting-started/find-your-organization-id

Next double click on the “apiKey” and input your Meraki API Key that has at least Org read privileges.

For instructions on the API please see:
https://documentation.meraki.com/General_Administration/Other_Topics/Cisco_Meraki_Dashboard_API

Optional Validation Node

Double click this node and in the variable, populate a MAC Address of one of your sensors (or any other device in your Meraki organization with a name).

This will populate a field in the debug node called “Debug the response” of “globalvartest” as you can see here:

If this does not populate, there is something wrong with the MAC Address inserted in the flow, if the whole call fails and gives a statusCode other than 200, more than likely the API Key, or Org ID is incorrect.

If you drop the payload portion of the debug message down, you will see the list of all devices in the response. If you open the function node titled “Create Global Key Store” and review it, the small function iterates through the payload array, takes the MAC Address, the serial, and then maps them to global variables with the name configured in dashboard.

Now for the Flow to onboard all the things!

First things first, lets edit the MQTT configurations to map to your Mosquitto instance:

Starting with the left MT Topics go in and edit the configuration:

Change the server IP address, then go to the security tab and input your node-red user/pass

This should update the rest of the MQTT nodes, if it does not, double click on each of them and make sure the HASS MQTT server configuration is correct. There are multiple nodes to try and ensure that we set retain on specific information needed in the Mosquitto db while on others not storing the readings as they do consume resources.

From here you SHOULD be good to go. However, please read through the nodes and make sure you understand them. I will more than likely iterate on this blog post as time goes on. For a break-down of the nodes please continue reading!

Explaining the flow in a little bit of detail:

Section 1:

The first node is just the MQTT subscription node. Where we import messages from MQTT into the flow based on the topic we matched:

the topic we match has a wildcard after MT to match all topics from all networks

The next node is used to configure a couple variables we re-use later in the flow for auto discovery.

The string node is used to grab the sensor_mac from the topic for further discrimination

The “Look up Name” node takes the sensor_mac we isolated, and looks up the name from the global database

The last node in this portion of the flow creates a sensor_mac_clean variable that is used for Home Assistant Discovery topic creation

Section 2 (the beefy part):

This section is where we sort the topics, then output them to specific nodes to create topics Home Assistant will be able to discover and add to your installation.

1. Sort Topics:

This node just looks at the topic and sorts based on the topic contents for each type of telemetry

2. Rate Limit:

We take each topic and rate limit them in case there is a flood per topic

3. Door node:

Here is where we set up the contents to create the topic. The msg.payload is incredibly important and formulates the discovery data for Home Assistant

{"name": "MT-" & msg.sensor_name & "-" & msg.sensor_type ,"unique_id" : msg.sensor_mac_clean & "-" & msg.sensor_type, "device_class": msg.sensor_type , "payload_on": true, "payload_off": false, "value_template": "{{value_json.open}}"}

If you check the contents here, we are informing home assistant that this is a binary sensor, and what the sensor values are for on and off, aka open and closed.

4. Binary_Sensor node:

This node builds the topic for home assistant discovery from the data built into the message from earlier in the flow

5. Humidity Node:

This node is a standard sensor node capturing humidity, and is much simpler in configuration as the information is less structured than binary sensors and is then piped into a sensor function node like the binary_node

6. Stop the LOOP:

Use this to drop the already reformatted messages

7. PM25 Node:

This node is a little different due to an issue I have found with auto discovery topics and containing periods.

Due to the default structure of the PM2.5 payload with MTs, Home Assistant tries to filter the payload as a nested JSON key and does not properly onboard the MT sensor data. To combat this, I built a flow to rewrite the payload data. As seen here, the PM25 node looks fairly close to the others, but as we proceed through the flow, I change the topic to PM2.5MassConcentration-reformat so that we can differentiate the source of the msg being either direct from the Meraki environment into MQTT or from node-red, then process it through the next section.

8. The PM2.5 fix

Here we shift the values around to reformat the data payload that comes through, to have the payload key of pm25 instead of PM2.5MassConcentration. This allows home assistant to write the values to the db correctly, instead of reading it as {“PM2” : {“5MassConcentration”: “$value”}}.

9. Retain MQTT output:

This is just an MQTT output node where we set the message to be retained in the db.

10. MSG debug nodes:

Debug nodes are how you learn, and figure out where you messed up, or what data you receive at different points in your flow. You will notice a fair number of debug nodes configured but disabled in this flow. If you enable one, do so by clicking the button to the right of the green nodes. You dont have to deploy the flow to enable it, however if you want the flow to start with it enabled if you reload node-red, deploy the configuration. Example:

OPTIONAL: HACS Add-Ons and Dashboard

HACS allows for developers to create themes, new lovelace cards, and even new integrations (like wyze) without having to have them merged directly into the HA code-base. There are 2 specific HACS add-ons I suggest installing so that you can import my HA Dashboard that auto populates MT as they are added to the Meraki network.

Here is what it looks like:

To install HACS: https://hacs.xyz/docs/setup/download

Once installed, navigate to the HACS tab in Home Assistant:

Next we head to FrontEnd:

Install the packages:

Auto-Entities:

Then install Mini-Graph-Card:

Once these are installed, you will need to reboot your Home Assistant instance to merge the new front end packages.

Import the Lovelace YAML for MT Auto Graphs

importing this dashboard is fairly simple. The process is:

Dashboard.yaml_Download

Download the zip file with the .yaml inside
Open and copy the yaml contents
Navigate to the top right > Edit Dashboard:

Click the 3 dot menu again and select Manage dashboards:

On the bottom right then select +Add Dashboard

Call the Dashboard whatever you would like:

Select the new Dashboard on the side bar:

Start with an empty Dashboard and “Take Control”

Top Right select Raw Configuration Editor

Select all and paste the contents of the file over anything in the window
Click Save and close the editor

BOOM! You’re Done!

Conclusion

This has been a lengthy blog post. I wanted to ensure I explained as much of this as I could and hope that if you have any feedback, find any errors, or want more info that you leave a message in the comment area below!

Useful Resources

htt ps://hasspodcast.io/ – Listen to Rohan Karamandi and Phil Hawthorne talk about new things and fun stuff with Home Assistant
https://www.hom e-assistant.io/docs/mqtt/ – Home Assistant MQTT overview and integration info
https://nodered.org/docs/tutorials/first-flow – Node-Red getting started
https://www.home-assistant.io/ – Home Assistant’s main web page with a plethora of information
https://www.youtube.com/c/NodeREDStudio – Node-Red’s youtube channel with great videos