Premise MQTT for Premise.

[pete_c]

Very nice 123.   Curious about the current Premise user base here on Cocoontech? 
 
I have seen you writing about Premise now here for over 10 years.  I am now over 60 here and tinkering keeps me going these days.
 
I was starting to tinker with graphing using Node Red here with Homeseer and the author of the Mosquitto plugin added two created based on payload variables - rate and accumulation (payload numbers) and charting based on either payload or rate of payload or accumulation rate which moved the charting over to the mothership.
 
Charting of payloads was already there and initially wanted to create a rate of payload function in Node Red.
 
Only graphing right now is relating to the old Hobby Boards lightning sensor rate which is working fine.
 

 
Ordered a couple of SonOff basic WiFi switches which I will be modifying to utilize with Mosquitto in the next few days with concurrent SSL testing.  This will be modifying the SonOff hardware with new firmware and doing a hands on to the SonOff board.  These used to be around $5 each and now noticed that they are around $7 each.  Still a reasonable price for a remote controlled automation device.

 

[123]

Pete,
 
I figure there are only a handful of active Premise users left ... and I believe they're all here. The software just works, can still be extended to incorporate new things, so I continue to use it.
 
 
If you're going to flash the Sonoff devices with new firmware, have a look at this blog post where the author compares three popular firmware replacements: Tasmota, ESPEasy, and ESPurna.
https://lobradov.github.io/FOSS-Firmware-comparison-overview/
 
It's the first of four articles and it highlights the principal differences between the competing choices. Well worth the read and useful for choosing the one that's bet for your needs.
 
I'm also following this project: https://github.com/Koenkk/zigbee2mqtt  It's open-source firmware for an inexpensive ZigBee adapter and converts the ZigBee protocol to MQTT. It supports several common ZigBee devices from Philips, Ikea and Xiaomi directly (i.e. without requiring the manufacturer's hub or cloud). Xiaomi makes inexpensive ZigBee sensors, typically under sold for under $15 (see GearBest)
 
https://github.com/Koenkk/zigbee2mqtt/wiki/Supported-devices
 
Most of the discussion is happening on the Home Assistant forum (but the project is not tied to HASS): https://community.home-assistant.io/t/zigbee2mqtt-getting-rid-of-your-proprietary-zigbee-bridges-xiaomi-hue-tradfri/52108

 

[pete_c]

Thank you 123.
 
Last week purchased a couple of the SonOff basic WiFi modules and flashed one OTA style last night with a Tasmota firmware.
 
Still in tinkering mode learning baby steps style.
 
Historically and today still mostly in analog wired mode for my automation. 

 

[pete_c]

I am way behind compared to what you are doing 123 today with Premise. 
 
I am still going very slow here ...as mentioned baby steps...
 
Did my first hardware modifications today to a Sonoff Basic WiFi device turning it in to a low voltage sensor and switch from a high voltage lamp switch and changing the firmware to Tasmota.  It involved cutting the high voltage to relay traces on the Sonoff board, a bit of soldering and installation of some header pins (for JTAG and GPIO access). 
 
This will be the first device that I control using Mosquitto.
 
It is a bit similar to my tinkering with micro travel routers and RPis, using new firmware, exposed GPIO ports and a bit of bit banging over the last few years. 

 

[123]

UPDATE
 
 
I've enhanced SyseventBroker so that it can export Premise Home devices into OpenHAB format.
 
 
What's the advantage? I can now use OpenHAB's user-interface(s) to control Premise objects. OpenHAB offers "BasicUI", which is rendered in a browser and in OpenHAB's iOS/Android apps, and "HABPanel" which is also rendered in a browser and targeted for touchscreen devices.
 
 
If you have a dimmer in the kitchen called Home>FirstFloor>Kitchen>CeilingLight, it'll get converted into an OpenHAB Dimmer Item. It'll look something like this in the Items file (take a deep breath):
 
Dimmer Home_FirstFloor_Kitchen_CeilingLight "Ceiling Light" (Dimmers, Kitchen) ["Lighting"] {mqtt=">[brokerremise/home/firstfloor/kitchen/ceilinglight/brightness:command:*:default],<[brokerremise/home/firstfloor/kitchen/ceilinglight/brightness:state:default]"}
 

• Dimmer is the item's type.
• Home_FirstFloor_Kitchen_CeilingLight is its name for programming purposes.
• Ceiling Light is its display name in a user-interface.
• light is its icon for a user-interface.
• Dimmers, Kitchen are groups this item belongs to.
• Lighting identifies this item for use with Alexa, Google Assistant, and HomeKit.
• mqtt is the binding (aka device driver) that allows the item to talk to the actual kitchen light. 

 
Anyway, SyseventBroker takes care of performing the conversion. It also creates location-based Groups that reflect all the buildings, floors, and rooms in your Premise Home as wells as a few function-based Groups like Sensors, Lights, Security and their sub-groups like SecuritySensors, TemperatureSensors, Dimmers, NonDimmers, etc.
 
This diagram shows how objects in a Premise Home are converted to OpenHAB Items.
 

 
 
So far, I've created one UI using BasicUI and another, much simpler one, using HABPanel. A BasicUI is defined using a text file called a Sitemap. For HABpanel, you define it within the browser; just drag widgets around and define their purpose (super easy). 
 
Both UI's load quickly and responsiveness is excellent so don't let this multistep communications path fool you into believing there'll be unacceptable delays:
Premise on a PC <===>  MQTT Broker on a Raspberry Pi <===> OpenHAB on the same Raspberry Pi.
 
 
Rather than show you my own meagre efforts, I'll direct you to the impressive work done by others.

• Here are examples of BasicUI. 
• Here is a video demonstrating a HABpanel UI. The first half-minute shows the UI in action (albeit somewhat out of focus). Jump ahead to 1:40 to see how simple it is to add widgets and design the UI.
• Here are examples of HABPanel.

My next step is to leverage MyOpenHAB.org. It's a free cloud service that allows you to securely access OpenHAB remotely. The phone app detects it is not on your local network so it logs into MyOpenHAB.org (using an account you create) and accesses your OpenHAB machine at home. I believe you can optionally constrain which devices are remotely accessible.

 

[Shinyshoes]

123, Did you post your SysEventBroker module anywhere? I wouldn't mind taking a stab at it.  OpenHAB has some drivers I'd like to take advantage of.....

 

[123]

Shinyshoes, on 12 Nov 2018 - 19:16, said:
Shinyshoes, on 12 Nov 2018 - 19:16, said:
Shinyshoes, on 12 Nov 2018 - 19:16, said:
123, Did you post your SysEventBroker module anywhere? I wouldn't mind taking a stab at it. OpenHAB has some drivers I'd like to take advantage of.....

So far only @etc6849 has been actively using SyseventBroker. Over the summer we worked together, offline, to refine SyseventBroker. I didn't get much feedback in this forum so I skipped posting updates about the latest greatest version of SyseventBroker.

But since you asked, I've attached the latest version of SyseventBroker and its important companion, namely the Premise-MQTT flow for Node-Red. Both are needed to make Premise fluent in MQTT.

FWIW, I have an RPi running OpenHAB with a Mosquitto MQTT Broker. To install the broker, I used openhabian-config and select the option to install the MQTT broker.

NOTE:
If you have a dimmer like this: Home.MasterBedroom.Bathroom.Sconces
Being a dimmer, it will have two MQTT state topics:

• premise/home/masterbedroom/bathroom/sconces/powerstate
• premise/home/masterbedroom/bathroom/sconces/brightness

It will also have two MQTT command topics:

• premise/command/home/masterbedroom/bathroom/sconces/powerstate
• premise/command/home/masterbedroom/bathroom/sconces/brightness

Basically, you control the light using its command topic and receive its status via the state topic. SyseventBroker has an Export function that can export most of your Premise home objects into an openHAB Items file. This save you the hassle of manually transcribing all of your Premise objects into openHAB format.

Here's some documentation to help gain a better understanding of what SyseventBroker does, what's needed to make the magic happen, and how to install everything.

INTRODUCTION
SyseventBroker is a Module that provides Premise with the ability to communicate via MQTT.
SyseventBroker sends Premise's sysevents (state-changes) to an MQTT Broker (state topic) and receives MQTT messages as sysevents (command topic). It uses Node-Red as a middle-man to handle actual MQTT communications. The system works like this:

OUTBOUND INFORMATION
1. SyseventBroker listens for sysevents occurring on supported devices (currently all standard Home devices, except MediaZone and Keypads).
2. The sysevent is converted into a JSON string.
3. The JSON string is sent, via TCP, to Node-Red.
4. A Node-Red flow converts JSON into MQTT format and publishes it to an MQTT Broker (state topic).
Information can travel in the opposite direction as well.

INBOUND INFORMATION
1. The Node-Red flow is subscribed to all Premise-related MQTT topics (i.e. it listens for MQTT messages intended for Premise).
2. It converts a received MQTT message (command topic), from the MQTT Broker, into a simpler format and sends it, via TCP, to SyseventBroker.
3. SyseventBroker validates the received information and uses it to control Home devices (dim a light, adjust thermostat setpoint, arm security system, etc).

SyseventBroker transforms a device&amp;amp;rsquo;s path and property into an MQTT topic. Here are examples of how Premise properties appear as MQTT topics. Notice all characters are lower-case and there's no leading slash.

• premise/home/secondfloor/masterbedroom/tablelamp/brightness
• premise/home/garage/ceilinglight/powerstate
• premise/home/house/hallway/mainthermostat/currentsetpoint
• premise/home/entrance/securitysystem/stayarm
• premise/home/yard/temperaturesensor/temperature

CONVERT AND VALIDATE
SyseventBroker can convert all outbound messages according to one's preferred Units of Measurement. For example, Premise can convert all outbound temperature values into Celsius (for use in MQTT) and convert inbound values back to Fahrenheit. It also validates all incoming values. If it receives an invalid value, it rejects it, and then sends the current, correct value back to MQTT. For example, to turn a light on/off, the valid values are 1 and 0, respectively. If SyseventBroker receives any other value, it ignores it and sends the light's current value.

MONITOR CONNECTION
SyseventBroker offers a Heartbeat function to monitor its connection with Node-Red. It sends a heartbeat message to Node-Red which replies with an acknowledgement. If there is no reply by the third heartbeat, SyseventBroker enables its Communication Failure property and logs a Premise Event.

SEND STATUS
SyseventBroker supports Status Requests for a single device or all devices. Here are the MQTT topics for the two types of status request (single and all):

• home/secondfloor/masterbedroom/tablelamp/devicestatus
• home/devicestatus

Upon initial configuration, it is recommended to send the status of all devices to the MQTT Broker. The message&amp;amp;rsquo;s payload (value) is configured to be retained by the Broker. Now the Broker knows the status of every Premise device. When an MQTT client connects to the Broker and subscribes to Premise-related topics, it will immediately receive the status of all Premise devices.

INSTALLATION
Installation involves importing the Premise-MQTT flow into Node-Red and configuring it to connect to your MQTT Broker. In Premise, a Lantronix UDS10 is created to provide TCP communication with Node-Red. Lastly, the SyseventBroker Module is imported into Premise and configured.

PREREQUISITES
1. MQTT Broker
MQTT communications require an MQTT Broker. Eclipse's Mosquitto is a free, open-source version available for Linux, Windows, and OSX (www.mosquitto.org).
2. Node-Red
It is described as the "Visual wiring tool for the Internet of Things" and serves as the middle-man between Premise and MQTT. Node-Red is available for Linux, Windows, and OSX (www.nodered.org).

HOW TO IMPORT THE PREMISE-MQTT FLOW INTO NODE-RED.
1. Open the Premise-MQTT file in a text editor. Select and copy all content.
2. Switch to the Node-Red editor.
3. If needed, click the "+" tab to create a new flow (empty canvas; no nodes).
4. Click (Hamburger) Menu > Import > Clipboard.
5. Paste into popup window and click Import.
6. Click on the canvas to anchor the flow.

HOW TO CONFIGURE THE FLOW'S MQTT CONNECTION.
1. Double-click the MQTT node (upper right in flow).
2. Click pencil icon.
3. On Connection tab:
a. Enter Name to identify the node (such as the host name of your MQTT Broker).
b. Enter Server. This is the MQTT Broker's host name or IP address.
c. Enter Client ID. Choose a unique name to identify the MQTT session.
4. On Security tab:
a. Enter Username (if your MQTT Broker requires authentication otherwise leave fields blank).
b. Enter Password.
5. Click Update, then click Done.
6. Click Deploy to commit all the changes.

The MQTT node should indicate "connected". If not, double-check all information you entered.
If the MQTT node connected, the flow is now operational and awaiting a TCP connection on port 5100 (default). Most of the flow's Debug nodes are disabled. They can be enabled to examine how messages are processed.

HOW TO CONFIGURE WINDOWS.
Use Windows Firewall to confirm Premise Server has access to Private Network. SyseventBroker will use TCP port 5100.

HOW TO CONFIGURE PREMISE
1. Ensure the Lantronix driver is installed.
2. Create a New > UDS10.
3. Set its Name to SyseventBroker (for convenience; it can be any name you wish).
4. Set IPAddress to the address of the machine hosting the MQTT Broker.
5. Set IPPort to 5100.
6. Import SyseventBroker.xdo.
7. Create Devices > CustomDevices  > New > SyseventBroker.

HOW TO CONFIGURE SYSEVENTBROKER

• Double-click Port, select Devices > Lantronix > SyseventBroker.
• Confirm PortStatus indicates "Port(Opened)".
• Scroll to UnitsOfMeasurement and indicate your preferred units for communicating with MQTT.
• Click SendStatus to transmit the current status of all devices to MQTT.
• Optionally enable Heartbeat.
• The default Heartbeat interval is 30 seconds. If you wish to change it, navigate to Modules > SyseventBroker > Broker > Timers > HeartbeatTimer.

SCRIPT MACROS
The Module contains two Script Macros:
1. ListSupportedObjects
Generates a list (in Debugview) of all Home devices containing a DeviceStatus method. These are devices that are (ostensibly) supported by SyseventBroker. All Lighting, Appliance, HVAC, Security, Safety, and Scene objects are supported.
2. ListUnsupportedObjects
Generates a list (in Debugview) of all Home devices that do not contain a DeviceStatus method. These are devices that are (ostensibly) not supported by SyseventBroker. MediaZone and Keypad objects are (currently) not supported nor are any custom devices.



ADDENDUM
There's one more thing you'll need for openHAB and that's Transforms. Premise sends 1 and 0 for True and False but openHAB wants to see ON and OFF (for a Swtich). So in openHAB's Paper UI, go to Add-Ons > Transformations and install MAP-Transformation (it makes the service available to openHAB). Then you create a file in the transforms folder containing the conversion from 1 to On and 0 to OFF. You'll need another transformation file to convert in the other direction from OFF to 0 and ON to 1. To spare you this hassle, I've attached all the transformation files I've created.

 

[chucklyons]

Forget my other post. I'll take a look at what you have here....

 

[Shinyshoes]

wow! Thanks for the extremely comprehensive overview.  I'll be attempting it this weekend. Thanks so much.

 

[chucklyons]

I finally downloaded this - not sure if I missed the files or what, but after rereading this a dozen times, I think I can spend some time trying to make it work in my environment. 
I did look at Home Assistant and other than not supporting some of my stuff (like others have said), it seems like a lot of 'do this, then that, and don't forget to' sort of thing...a little too much for me, I'm afraid...

 

[chucklyons]

I havent been able to find the syseventbroker module? Is it named something different?
 
Does it use the Premise built-in pub/sub functionality?

 

[123]

I havent been able to find the syseventbroker module?

 
Scroll up ~5 posts. It's in a zip file attached to the bottom of my post dated November 13.
 

Does it use the Premise built-in pub/sub functionality?

 
No. The built-in pub/sub system is based on Premise's web server. I didn't think it would be wise to test its limits. If you have 50 objects with an average of 2-3 properties each, that's 100-150 subscriptions to establish.

SyseventBroker extends all classes (except one or two) so that when one of their properties undergoes a state-change, it transmits the change, via TCP, to Node-Red. In turn, it converts it into an MQTT topic and message and publishes it to the MQTT Broker.
 
For example, assume we have a Light object called Home.Living.Spots. If you turn it on, PowerState property changes from 0 to 1. This state-change is automatically detected and transmitted to Node-Red which converts and publishes it to the MQTT Broker.  
 
The light's MQTT topic will look like this: premise/home/living/spots/powerstate
It's message (a.k.a. payload) will be: 1
 
That same light has another MQTT topic that is used for controlling it: premise/command/home/living/spots/powerstate
 
If I publish a 0 to that topic, Node-Red will receive it, convert it into a format that can be digested by SyseventBroker, and transmits it via TCP to Premise. Upon receipt, SyseventBroker validates it and then acts on it  (turns off Home.Living.Spots).
 
Response time is sub-second (even though Node-Red and the MQTT Broker aren't hosted on the same machine running Premise). I've been using it since last summer and it works well. Occasionally Premise momentarily disconnects from the other machine but I have SyseventBroker configured to re-establish the connection.

 

[chucklyons]

Now I recall -- It's a lot of configuration work i.e. things that go bump in the night. Plus a new UI, plus... I thought the SYS pub/sub handled around 30 connections and did a self reset if it lost communications? I would think for a lot of homes, that would be adequate. I've been running around 100+ objects, plus multiple mediazones, thermostats, lights, sensors, and at least one instance of the other objects (like a fridge and aquarium!) I was able to generate a SYS subscription, but it would be quite a while for me to make it hardy.   I would expect it would take me a long time to convert from the UI I have to something that would work with Node and MQTT. (not that anybody will ever use it anyway, except the folks in my homes).   I've been able to run with polling working well. I haven't seen any issues with timing, response time or memory usage (browser or CPU)   Your sub-second timing seems very cool, but for my typical usage - lighting, media, sensors, and climate, I'm not sure that resolution is required?   But I'll give eventbroker a try after I get my UI done...
p.s. 1) how did I forget that I downloaded it?? 2) why did search not pick it up? (and save me from myself?)

 

[123]

It's a solution that makes Premise speak MQTT. It allows Premise to communicate with an expanding universe of MQTT-aware devices (including other home automation systems). 
 
Yes, there's configuration involved as well as learning new things. I run Node-Red and Mosquitto MQTT on a Raspberry Pi. Home Assistant runs on an old netbook. I plan to consolidate them onto a micro PC.
 
As for 'bump in the night', as mentioned, I've been using it for many months (over a year if you count the development version) and it's solid.
 
If the subscription limit is 30 then that would be inadequate for many homes. Many objects have just one property but a dimmer has two properties. A thermostat has eleven properties. A security system has thirteen properties. It wouldn't take too many objects to have the sum of their properties exceed 30.
 
I use Home Assistant to serve as a frontend for Premise. It's full-fledged home automation software with a modern UI (that adapts itself to different screen sizes). I use some of its other features to enhance Premise. For example, it provides Homekit integration so I can use Apple's Home app (or issue a command via Siri) to control all devices managed by Premise. Developing such a feature specifically for Premise would require a lot of time and effort. I chose to invest that time and effort in learning Home Assistant and benefiting from everything it offers.
 
Anyway, there it is. If someone is looking for a new UI that's native to Premise, SyseventBroker isn't it. It's a gateway to MQTT and a world of new functionality.

 

[chucklyons]

123 - You're probably right.
 
HASS looks good, but the amount of effort to get to a starting point seems like a lot. Get a Pi. Or install a Docker instance. Or a VM.  
 
Seems like a lot of good stuff - drivers galore. I like some of the UI features, etc.
 
Probably my last question on this topic - with the assortment of drivers and other functionality, why keep Premise on the backend? (I know why I would, but curious as to your why). Fortunately, Jim released the RA2 driver for Premise - I left the RA at the other house and have installed RA2 at both places. Fortunately, I didn't install any Nest stuff (c/should have seen that coming. They yanked YouTube from the Echo stuff. So much for interoperability)
 
I'll continue with my dive into RTI (who sent me an email on the Nest obsolescence , as well as Lutron), at least until C4 gets into Channel Distribution.