Premise MQTT for Premise.

Back in December, in the Roll Call thread, I made a passing reference to MQTT and how I wished Premise supported it. After quite a bit of code-wrestling (and it's not over yet) I'm pleased to report I've taught Premise to speak fluent MQTT.


In a nutshell, if a Light's Brightness changes from 0.25 to 0.75 (25% to 75%) Premise will announce it via MQTT. Anything external to Premise that speaks MQTT is able to receive the message (like another Home Automation system, a UI client, a relay, etc). Premise also listens for MQTT messages and will act on them. For example, it can receive an MQTT message to set the Light's Powerstate to off.

In an ideal world, I would've implemented this new functionality as a native Premise driver. However, I was unwilling to invest the substantial time needed to write a driver in C++ and incorporate all the finer points of the MQTT protocol. It was too steep a learning curve (for me). I turned to open-source software to get the job done. In other words, this solution requires another resource to work. That resource is Node-Red. I have it running on a Raspberry Pi but there's a version available for Windows (which I have not tried).
 
Unfamiliar with Node-Red? It's called the 'Visual wiring tool for the Internet of Things'. Here's an entertaining tutorial:
 
https://www.youtube.com/watch?v=WxUTYzxIDns

I created a driver called SyseventBroker (implemented as a Module) that sends/receives data to Node-Red via TCP. All of Premise's state-changes (sysevents) are transmitted, in JSON format, to Node-Red. A 'flow' in Node-Red converts it into an MQTT message and 'publishes' it to an MQTT Broker (more open-source software in the form of Mosquitto running on the Raspberry Pi). The same flow receives MQTT messages and converts them into a format more palatable to Premise. The flow sends them to SyseventBroker (via TCP) which converts them into actions (turn lights on/off, adjust the thermostat, arm the security panel, etc).
 
It may look like there are many links in this chain but the transit time is virtually instantaneous.
 
[ Light turns ON --(sysevent)--> SyseventBroker ] ===(TCP)===> [ Node-Red --(flow)--> MQTT ] ===(TCP)===> [ Mosquitto MQTT Broker ] ===(TCP)===> [ MQTT Client ]
 

The first challenge I encountered is that Premise has no 'sysevent highway'. By that I mean there's no single place where you can monitor all sysevents (state-changes) taking place within Premise. If there is such a place, I'm unaware of it. My solution was to extend each and every Device class. Now when any Device's property changes state, it reports it directly to SyseventBroker. In turn, SyseventBroker passes the state-change on to Node-Red which publishes it as an MQTT message.

There's still work to be done. For starters there's still much more testing I want to do because I've already learned, the hard way, that its possible to create a nasty little feedback loop. How? Easy, if you're handling temperature values! Premise stores temperature in Kelvin, so if you send it out via MQTT in Fahrenheit (for convenience), it'll get echoed back to Premise in Fahrenheit. Except Premise will handle it as Kelvin! This causes another state-change that is sent out in Fahrenheit, echoed back in Fahrenheit, treated like Kelvin and ... the whole mess goes round and round like this until you stop it. I've mitigated the endless looping but not the problem of receiving temperature in Fahrenheit or Celsius (it *must* be in Kelvin). It's glitches like this that may demand design changes so more testing is needed.

I also want to add a 'heartbeat' function to SyseventBroker so that it can monitor the health of its TCP connection to Node-Red.

@123
 
Curiosity question.
 
Here have configured Node Red, OWFS on a RPi 2 now running with Stretch in the attic and just made a new RPi  / temperature hub for the basement (covers main floor of the house).
 
This is sort of a mini 1-wire hub using a Maxim 9097 for just multiple temperature and humidity reads.
 
I have configured OWFS to output in Fahrenheit.  I am also using MQTT and that is working great on the mothership.
 
Only issue that I have is that my temperatures to MQTT are all in Celsius.  Watching my Red Node flow debug numbers are always in Celsius.
 
Just curious what I am doing wrong and why my temps stay at Celsius even though my OWFS configuration is Farenheit.
 
Reading a bit it seems that I have to create a mini script in my Node Red flows for conversion.
 
Curious if you know why the Node Red OWFS flow reads the temperatures in Celsius?
 
Guessing I just have to convert the number but thinking this might slow down the reads a bit...
 


 

I don't mean to sound snarky but I think the simple answer is the entire world uses the Metric system ... except two (three?) outliers. So, speaking as a Canuck, "Celsius" is the norm.
 
But seriously, I've never used OWFS (not with Node-Red or anything else) so I'm unfamiliar with the specifics of how it converts (or doesn't convert) data in Node-red/MQTT messages. It seems like if you specified Fahrenheit then it should respect that choice when transmitting its data. Unless, of course, that choice is limited to being a "display format" (i.e. only converts for presentation) and is ignored for transmission (i.e. for  export). Of course, that's kind of weird for 1-wire devices since everything they produce is expected to be transmitted for use elsewhere.
 
Premise stores property values in Kelvin, meters, grams, etc and uses "LocaleSettings" (configured by the user) for converting them for display purposes. If you specify "Fahrenheit" then temperatures are automatically converted from Kelvin to Fahrenheit for display purposes. It means you have to be careful when setting or getting temperatures.
 
The native unit is Kelvin, so Temperature=72 means 72 K and not 72 F or 72 C. The correct way to do it (in Premise) is Temperature.Fahrenheit=72.
 
Similarly, if you retrieve (or export) Temperature, you'll get the value in Kelvin, regardless of whatever LocaleSettings say because that's used exclusively for presentation purposes. To export Temperature in Fahrenheit, you have to export Temperature.Fahrenheit. Perhaps OWFS works in a similar fashion ... or it's just an old-fashioned bug!
 
Worst case, add a Function node in your flow to convert Celsius to Fahrenheit (the calculation and overhead are trivial).
var tempF = tempC * 1.8 + 32;
 
In Node-Red it'll look something like this:
msg.payload = msg.payload * 1.8 + 32;

Understood and thank you 123.  Will create a conversion node.
 
Still in learning mode relating to Node Red use.  Used xAP / xPL for multiple 1-Wire sensors and did do the conversion at the display side of things.
 
Lately before playing with MQTT and Node Red been outputting a text file using Digitemp and doing a line by line conversion to variables which works but is sort of the long math way.
 
Digitemp outputs concurrently in Celsius and Fahrenheit.

I have been following your blog posts concerning your explorations with Node-red and MQTT. It's powerful technology and I've only scratched its surface. Node-red is fantastic "glue" for binding together disparate sources of data (as per their slogan "a visual wiring tool for the Internet of Things"). Meanwhile, MQTT is shaping up to becoming "The One", namely the long-desired "one universal protocol" for Home Automation (insert requisite XKCD reference here).
 
MQTT will open new doors for Premise. It'll be able to communicate with other devices and HA systems that speak MQTT. For example, I can leverage Node-red's extensive connectivity or even connect to openHAB or Home Assistant and use their device-drivers. Both offer custom nodes for Node-red so I can, theoretically, connect Premise to both using a flow in Node-red. Instead of migrating away from Premise, as was my original intent, I can maintain my decade-old investment (in time and effort and driver development) and simply extend it via Node-red and MQTT. 
 
 
 
PS
 
I don't know if you have the need to send/receive via TCP in Node-red but I'll take this opportunity to share something I discovered that might save you some time and grief. My need was to create a flow that receives via a TCP port and sends data back through the same port. Seems easy enough but, surprisingly, sending stuff back through the TCP port doesn't work "out of the box" and the documentation doesn't explain how to make it work. I found the solution in this Google Groups post: https://groups.google.com/forum/#!topic/node-red/uI-DA5XU8CY
 
Upon receiving the first inbound TCP transmission, you save the "session ID" in a global variable. Later, when you need to send something back through the TCP port, you retrieve the global variable and set the outbound message's "session ID". Now it'll happily use the same TCP port. Otherwise, it tries to use a new TCP port and gets nowhere. 
 
The "gotcha" is that at least one inbound message has to arrive first in order to set the "session ID" for use by outbound messages. Basically, when the external source connects to Node-red, it ought to transmit something (anything) to get that "session ID" recorded.

Thank you 123.
 
Just typed a paragraph here and Firefox took a dump.
 
Here was doing a Python script which read 1-Wire stuff and sent it out via MQTT and mostly just tinkering with MQTT.
 
The RPi in the attic doing node red has an internal ZWave GPIO card, 1-wire 9097, running Domoticz and an SDR (doing satellite maps).  No issues right now running all of this stuff.  Just updated this RPi from Jessie to Stretch in vivo remotely.
 
Homeseer MCSSprinklers author Michael McSharry wrote a Homeseer MQTT plugin in the last quarter of 2017 that works in Linux.
 
I have yet to venture in the the automation pieces and only currently tinkering with the 1-wire Red Node stuff.
 
Michael McSharry is playing with
 
"ESP8266/Sonoff units that sell for under $5 and these include the power supply, Wifi, Relay, multiple discretes and a well supported software environment for hacking. Emulation for Wemo and Hue is included so Echo/Alexa voice control is built in"
 
Here are some snapshots of the configuration.  I am doing this baby steps fashion.  I have two RPi's configured right now using Node Red going really slow with it.  Two Homeseer servers are configured running in Ubuntu 16.04 / 64 bit.  One is doing multiple Oracle Virtual Boxes for Windows only stuff like Microsoft SAPI (collect voice fonts here).
 

I should've realized that any attempt to send property values may require converting them to another unit of measurement and receiving them requires validation prior to conversion!  In other words, it took a bit longer to complete this phase of development.
 
 
SyseventBroker handles the following property types (validation and conversion wherever required).

• Acceleration
• Angle
• Area
• Boolean
• Currency
• Current
• Date
• DateTime
• Density
• Energy
• Force
• Frequency
• Length
• Luminance
• Mass
• MultiValue
• Number
• Percent
• Power
• Pressure
• Real
• Resistance
• Temperature
• Text
• Time
• Timespan
• Velocity
• Voltage
• Volume

FWIW, there are a few property types SyseventBroker does not handle and simply rejects them (like Color, HTMLColor, ObjectReference, Picture, StaticText, etc). Otherwise, all the principal property types (listed above) are covered.
 
Mapping between Premise and MQTT is super easy. If you have this in Premise:

• Home.SecondFloor.MasterBedroom.Closet.Light

then its two properties, PowerState and Brightness, are referenced like this in MQTT:

• home/secondfloor/masterbedroom/closet/light/powerstate
• home/secondfloor/masterbedroom/closet/light/brightness

To control the light, set the MQTT message's payload to the same values used by Premise

• powerstate, set the payload to 0 or -1 (off, on).
• brightness, set the payload to a value between 0 and 1.

It's that simple.
 
I've successfully tested SyseventBroker with MQTT-Spy and with Home Remote (using its MQTT client). Communication is virtually instantaneous. There's one more thing I want to add before sharing it.

Wow!  Thanks so much for all your hard work on this.  I can't wait to use it.

Thank-you 123
 
Got the conversion going for one sensor...after a revisit...it does help to walk away for a bit then a revisit...
 
Here is a picture of my flow and the verbiage...
 

 

 
Here is the flow (still a bit kludgy to me).
 
 

FWIW, if you like brevity, one line of code can do the trick.
 
msg.payload = Math.round((msg.payload * 1.8 + 32) * 10) / 10; // Convert C to F with 1 decimal place
 
Although this one-liner won't improve performance noticeably, there's no real need to create variables (tempc, tempf) if they're not referenced many times. You can avoid performing a division (9/5) if the result is already known and entered as a constant (1.8).
 
BTW, are you using the change node to set the received message's retain property to true (or false)? If so, is there a reason you're not using the mqtt node to perform that step? It would eliminate the need for the change node.

Thank  you 123.
 
Changed function to two lines:
 

1. msg.payload = Math.round((msg.payload * 1.8 + 32) * 10) / 10;
2. return msg;

 
BTW, are you using the change node to set the received message's retain property to true (or false)? If so, is there a reason you're not using the mqtt node to perform that step? It would eliminate the need for the change node.
 
Yes retain property is set to true.  This was a copy and paste from someone using OWFS for 1-wire.
 
I am still learning.   Looking to maybe shorten said 1-wire flow to maybe 3 nodes using nro read.

Curious 123 are you using Node Red for your MQTT stuff or are you just writing specific python modules for instances of MQTT?
 
Still in learning mode here and mostly trying to keep it to using Red Node on the RPis and now testing python MQTT on the micro routers which operate using OpenWRT.

Node-Red serves to connect Premise to MQTT. Effectively, I'm using 'off-the-shelf' software to get Premise talkin' MQTT.
 
Instead of creating a native MQTT driver for Premise (a non-trivial undertaking), I'm using Node-Red's ability to speak MQTT (and TCP). The driver I've written (SyseventBroker) reports all state-changes (via TCP) to a flow in Node-Red and it, in turn, publishes them as MQTT topics. The flow is also subscribed to all Premise-related MQTT topics and reports them to SyseventBroker in Premise. SyseventBroker converts units for all inbound and outbound data and also validates inbound data. The result is seamless integration between Premise and MQTT (without having to write a native MQTT driver).
 
The flow is shown in the attached file.

1. The flow receives messages from Premise via TCP.
2. It stores the TCP session ID for later use.
3. It converts the incoming message into a JSON object.
4. It reformats the JSON object into an MQTT message.
5. It publishes the MQTT message.

 
If the incoming JSON object is a heartbeat, the flow echoes it back to Premise via the TCP port (heartbeat is used to periodically monitor the connection's health).

1. The flow is also subscribed to all Premise topics (premise/#).
2. Received topics are converted into a format understood by SyseventBroker and sent back via the same TCP port (using the session ID stored earlier).

 
Topics look like this:

• premise/home/first/family/mantlelight/brightness/0.45
• premise/home/second/master/tablelamp/powerstate/0
• premise/home/first/hallway/thermostat/temperature/21.5
• premise/scene/evening/play/-1

 
 
 
 
 
 

Thank you 123.

I've completed MQTT for Premise (a.k.a. SyseventBroker). It's been running reliably in my home for over a month. There's very little configuration to do beyond installing it. In a nutshell, all sysevents in Premise (i.e. all state-changes) are automatically reported as MQTT topics (except for MediaZone and Buttons). It also listens for incoming MQTT topics and converts them into Premise sysevents. 
 
Topics are just the device's path and look something like this:
premise/home/livingroom/accentlights/powerstate
premise/home/first/hallway/mainthermostat/currentsetpoint
 
The first application of MQTT was for a new UI for my Android phone and my wife's iPhone and iPad. I used Home Remote (which talks MQTT, among other things) and we now have native apps with instantaneous control of Premise devices and their status. 
 
The second use was for two simple graphs of indoor/outdoor temperature and luminance. It was ridiculously easy using Node-red Dashboard and MQTT (see the attached images).
 
Future MQTT applications will probably involve inexpensive devices based on the ESP8266 (i.e. WiFi-controlled sensors and switches). Just about all home-brew devices talk MQTT nowadays and it can also be used to bridge Home Automations systems. For example, you can get Premise to talk to openHab or Home Assistant or the SmartThings Hub because all have the ability to communicate via MQTT.
 
 
Rather than spend time documenting SyseventBroker and posting it here (let's be honest, there are precious few Premise users left), if you're interested just let me know and I'll share it with you and help you get it running. Be advised it needs an MQTT Broker (a pre-requisite for any use of MQTT) and Node-Red. I have both running on a Raspberry Pi but there are other options.