Anyone interested in discussing building DIY Home Automation software and hardware?

deandob

Active Member
Hi,
 
I'm new to CocoonTech having introduced myself a couple of weeks back and sharing some of the work I have done to develop what I consider a unique home automation system in the 'welcome' section. Didn't get a lot of interest there so trying in this forum instead. 
 
I'm specifically looking to see if there are others like me here who like doing things from scratch rather than buying a package like homeseer or charmedquark (not there is anything wrong with doing that), and want to discuss experiences with software and hardware development of home automation systems. I'm a qualified Electronics Engineer who has built a career in the IT industry developing and designing large IT systems and I have used my experience with electronics, networking and software development to design and build my current system (evolved over several versions with 12+ years of improvements & learning).
 
My vision for a home automation solution is that it must be standards based using modern technologies, based on a local hub (cloud isn't fast enough or reliable enough although cloud services can augment local features), front end that is visually pleasing, extensible by the end user and dynamic, available on any device (including voice and video channels). An integration layer that is simple, scalable, easy to write and add new device drivers. An automation engine that is event driven, where historical data is just as easy to use as current data and being truly smart by layering sophisticated logic according to current or historic state (not just 'if this then that').
 
If you are interested in how I approach solving the home automation use cases, I've documented a couple of the concepts and some screenshots in the welcome forum here. I am considering making the software open source or even commercializing it - although I have a number of unique ideas / implementation, this field is getting very crowded! 
 
I can share how I managed to achieve the vision documented above but I'd also love to hear from others who are building their hardware and software DIY and what they have done and learning gained. Or maybe this type of discussion isn't a focus for CocoonTech but I haven't located another forum more relevant to discuss home automation software and hardware development.
 
I didn't capture the mobile interface in the welcome forum post, here it is (I have blanked out the house details for security reasons).
 
The mobile interface has the same drag and drop design surface that is super easy to customise as the desktop version, and uses the same HTML5 widgets so is usable for all mobile platforms. It is accessed via a cloud service over the internet that acts as a proxy for the websockets (the house hub creates the connection to the cloud service, so no problems with security and no configuration needed on home router firewall).
 

Attachments

  • HA mobile1.jpg
    HA mobile1.jpg
    91.5 KB · Views: 112
Hello deanbob,
 
I've done a fair amount of home automation software development.  My language of choice is C since my vocation is network administrator/ systems programmer.  I'm also proficient in.Net (C#) and C++. I am the author of FIDOH (Fetches Instant Data and Operates Homes) which is now defunct. FIDOH was essentially a GUI front end to operate various HAI (Leviton) panels.
 
Nowadays, I mostly write software to address  custom specific needs of my Leviton S&A (HAI) hardware platform. Unfortunately, I have minimal experience in the hardware development area.  My current project is developing the equivalent of the Leviton Omni Touch 7 (OT7) touch screen using a HP Windows tablet.  It's a work in progress.  The GUI front end needs a lot of work etc.  My work is on a much smaller scale than yours but I would enjoy sharing my experiences with you.
 
 
 
 
I've been using a hybrid approach to automation. I decided on an architecture where I write an interface to a device that runs as a service and broadcasts data using MQTT. Within this interface, there is an object that talks to the hardware and provides the data to a wrapper that handles the broadcasting of MQTT messages. This approach of splitting he interface and the protocol saved me a lot of work when I converted from xAP to MQTT. Then I create a driver for the home automation software (right now I'm using Elve) that reads the MQTT messages and provides the data to the software. Keeps everything separate and I can just create a small driver for whatever software I'm using and I don't have to redo all the device interfacing.
 
I have always been interested in coding for home automation, but I was not really interested in writing a complete, comprehensive home automation package, so I experiment with different products. Elve comes the closest to what I've always wanted in HA software, but it's non-supported and not upgraded. The touch screen functionality is great and not very difficult to do, and the price is right (free), so I can't really complain about the lack of support.
 
For what it's worth...
 
Matt
 
Guys, thanks for the responses, good to hear what others are doing.
 
Initially I was going to go the package approach but didn't like the idea of investing my time to learn and get locked into proprietary solutions (eg. CQ scripting model) where you are dependent on the evolution of the product compared to modern technology platforms like .NET, Node.JS and HTML5 where a lot of the platform services needed for a HA system are built in and you have so much more flexibility. I take a layered services approach where the you start with low level services (including those inherited from the technology platform like websockets) as the basis to build richer automation specific services (eg. event management service) so the features/functionality can evolve over time.
 
I use Node.JS for the device integration layer which has been a godsend - the ease of writing in JavaScript & its async nature along with the almost infinite number of NPM packages (think libraries for those who know C) available goes a long way to making it easier to write device drivers. I agree driver support is critical but it really isn't that hard if you concentrate on the logic of interfacing with the device in the driver and use the automation framework to do everything else (similar to how mdonovan describes his device layer). Most devices use a simple protocol to control them (eg. REST commands) and most of the time you are dealing with sensors (ingesting their data) rather than actuators (controlling the device) so you end up with most driver code reading their input and reformatting to the message format the automation engine uses - quite simple. For a new device driver I start with a template that exposes to the driver an API to interface with the automation engine and most drivers are a couple of extra lines of Node.JS code especially if you have a NPM package available to help (like the serial module). The automation framework provides a service to read and manage a settings file for the driver so you can describe the driver semantics in the settings file (editable as text in the front end) which makes it easier to write flexible and extensible drivers.
 
Here is an example of a driver which reads statistics from my solar power inverter over serial (all the 'fw.' functions are the API calls). For those who know JavaScript the code below isn't very complicated.


"use strict";

// Read power generated by solar panel inverter
var com = require("serialport");
var serialSolar;
var CR = "\r";
var oldData = -99;

// startup function
function startup() {
var startStatus = "OK"
serialSolar = new com.SerialPort(fw.settings.comport, {
baudrate: +fw.settings.baudrate,
databits: +fw.settings.databits,
stopbits: +fw.settings.stopbits,
parity: fw.settings.parity,
buffersize: 255
}, function (err) {
if (err) fw.log(err + ". Cannot open solar serial port, no solar generation functionality available.")
});

serialSolar.on("open",function() {
fw.log("Serial port open on " + fw.settings.comport);
});

serialSolar.on("data", function(data) {
serialRecv(data);
});

serialSolar.on("error", function(err) {
fw.log("Serial port general error " + err);
fw.restart(99);
});

setInterval(pollInv, +fw.settings.pollinterval * 1000, "POUT");
//setInterval(pollInv, +fw.settings.pollinterval * 1000, "VIN");

return startStatus;
}

function pollInv(cmd) {
serialSolar.write(new Buffer(cmd + fw.settings.cmdchar + CR), function (err) {
if (err) {
fw.log("Serial write error: " + err);
fw.restart(99);
}
})
}

function serialRecv(data) {
if (data.length > 0) {
var generated = parseInt(data.toString().split(CR)[0]);
if (generated < fw.settings.changetol) generated = 0; // ignore any spurious watts generated at night
if (Math.abs(generated - oldData) >= fw.settings.changetol) {
fw.toHost("Power Out", "W", generated);
oldData = generated;
}
}
}

// Shutdown the plugin
exports.shutPlugin = function shutPlugin(param) {
//Insert any orderly shutdown code needed here
return "OK";
}


 

Here is the settings file for the driver:
 


[PluginCfg]
Desc = Read power generated by solar panel inverter
Enabled = true

[General]
ComPort = COM3
BaudRate = 9600
StopBits = 1
DataBits = 8
Parity = none
pollInterval = 6
CmdChar = ?

; Dont broadcast any change greater than
ChangeTol = 10

; Define the channel characteristics
[channel0]
Name = Power Out
Desc = Current power generation
Type = integer
IO = output
Min = 0
Max = 5000
Units = watts
[channel0.attrib0]
Name = Power In
Type = Command
Value = PIN


[edited]
Wow - I just read the link from neverDie with a post from Dean Roddey about the 1 million lines for the CQ system and I'm blown away, and while I'm sure its a quality system, it reinforces my point that using modern technology platforms like HTML5 takes away most of the pain & effort. Dean mentions the UI and designer was most of the effort, I checked my front end HTML code and it is 4,300 lines of code which includes all the HTML and JavaScript, code to manage desktop interface and mobile interface, all the settings and configuration as well as the graphical designer. Including the back end .NET code to manage the interface, around 5,000 lines total. So still not a small system (and wasn't easy to develop) but a drop in the bucket compared to the effort Dean put into his CQ system for similar feature sets - although I'm sure Dean's system is more refined and better tested than mine. I guess different approaches to the same problem, I do have to live with some of the quirks of HTML5 (eg. compensate for browser compatibility) that Dean doesn't have to bother with, although evaluating effort versus benefit of the different approaches is a clear win for technologies like HTML5 and Node.JS versus coding even low level functionality all in C++ from first principles.
 
"I agree driver support is critical but it really isn't that hard if you concentrate on the logic of interfacing with the device in the driver and use the automation framework to do everything else"
 
Famous last words. It really IS that hard, which you'll discover soon enough. If you never support anything but trivial devices, whose interfaces lend themselves to whatever scheme you've chosen, it wouldn't be so bad. But the real world doesn't work like that. If you don't support the HAI, Z-Wave, UPB, Insteon, Elk, etc... you're doomed. And you'll find that supporting those very complex systems is very complicated, because they are very complicated. In the case of the HAI, it's binary and encrypted, for instance. You have to deal with asynchronous responses from many devices, and of course you have to deal with the fact that a lot of them are very quirky as well.
 
As I've said before, automation is one of those problem domains where it's incredibly easy to draw some boxes on a white board and connect them and think, hey, that's pretty straightforward. But there are innumerable details and dangers, and you'll never understand them all until you have already created a bunch of code, any users you manage to get will have done so as well, so that you have to rebuild the house multiple times over time while people are living it. That's when you'll realize why a limited scripting/customization system makes sense.
 
Hi Dean,
 
Granted I don't have your level of experience with home automation but I do have experience with architecting & developing complex commercial systems so I'm generically aware of the difficulties. I have a simple Z-Wave driver already but on the other hand found the C-Bus driver quite difficult to get right - debugging was painful (I used the C-BUS SDK and used the lowest level interface - sending/receiving strings). I agree that support for the popular controllers are critical if considering a commercial home automation product and requires a lot of work due to the vast number of devices that folks would want supported. So having the driver model simple but still powerful, and encouraging an open source approach where the community will help is important. I also agree that there are layers of complexity hidden below the surface of what seems like a simple use cases for home automation and I have learnt (and still learning) over the years - I've rebuild my system 3 times now to get rid of issues caused by initial design decisions (and also to take advantage of modern technology/toolsets).
 
There are multiple paths to solve the problem of home automation, each with their own set of advantages and compromises (eg. limited scripting approach versus a more open/standards based approach each have pros/cons). There is no right or wrong, just different ways of approaching the problem. 
 
Some examples to highlight the pros/cons of the different approaches, when I decided to add MQTT as the publish subscribe protocol for my sensor platform, it literally took a couple of hours to add MQTT support to the driver framework as there were NPM modules that had all the MQTT functionality available (both high and low level) as open source and in use by others so reliable. In the microchip sensor framework I've developed (in C - see my blog on hackaday.io) this took many days of work just to implement basic MQTT functionality and was quite difficult to debug, which really highlighted to me the advantages of modern development platforms like Node.JS versus old school C/C++.
 
On the other side of the coin, when implementing mobile functionality into the UI framework, which was about 10 hours work, I had to spend another 2 hours working out why icons weren't showing up on the mobile browser. As I was using Microsoft Azure as the mobile gateway, it worked out that certain mime types served from Express (Node's web server) were being intercepted by Azure and HTTP headers were being updated. If I had full control of the platform (like Dean's system) this wouldn't have been a problem.
 
You are kind of missing part of the big picture though. The issue really isn't how hard is to do, it's how robust will be in use, and how reasonable will it be to diagnose issues in the field when they arise. No one (other than the creator) really cares how hard it is to do, they only care about how it works for them. Stitching together various high level tools makes it easier to do, but less robust, more difficult to diagnose problems, and of course it greatly increases the likelihood that it will work for some folks and and not for others because of differences in the deployed versions of those bits on their particular systems. It's hard enough to keep it solid and consistent and debuggable when you have full control.
 
Ultimately the goal of automation systems is stability in the face of dealing with the real world and very complex, highly asynchronous environments. Anything that gets you closer to that goal isn't time wasted, even if it means that the time spent means fewer features overall. 
 
Anyway, I'm not trying to talk you out of it, just pointing out there is no free lunch. If you get something, you can be pretty sure you are also giving up something, and probably in relatively equal measure to what you gained.
 
Hi Dean, there is always a effort versus benefit equation to solve with decisions about design and implementation, and what you give up versus gain isn't always balanced.
 
Also home automation isn't in the same league as say, core banking systems (which I do have experience with) in terms of complexity and reliability requirements, where technology like .NET and Java are used extensively and recognized as fit for purpose. My home is large and has a lot of automation (my C-BUS driver alone handles over 70 lights and motors, never misses a beat), much more than most, and typically has to handle a transaction a second, the complexity and asynchronousity isn't mission impossible to manage or debug in typical HA scenarios.
 
I do agree with you about the interdependencies with versions of platform tools as a potential downside, however you can control that to a certain extent. Supporting a customer with a driver misbehaving in the field that tested fine on the test harness can be a nightmare but not so much easier with a monolithic system compared to one using platform tools like .NET or Node.JS - almost every time the problem isn't the tools but it is another variable to investigate. Conversely you would have to deal with interdependencies deep in your million lines of code that you possibly can't visualize or test for - using platform tools I have some confidence that the stack I'm using has been tested in many other scenarios by many other people and likely working to spec, and this has been my experience that problems are in my code not the stack.
 
So I don't subscribe that your approach is more reliable (but don't misunderstand me - I'm not saying your way is bad, you have developed a world class automation system that has been improved and refined over the years, very admirable).
 
[edit]
I spent some time this evening on the CharmedQuark site to see how it has been architected, and I am impressed, it has been designed from the ground up as a true distributed system and looks to be very scalable and flexible. The graphical interface designer is also very functional and I can see how it would have difficult and time consuming to write in C++. Nice work!
 
I can see the allure of wanting to leverage powerful tools that you're already familiar with, or that one  could justify learning because they would have broader applicability.  For instance, in HomeSeer the HSTouch tool has long been a lightning rod for just this issue.  It's supposed to allow people to design touch GUI's for phones and tablets, and though I'm not sure what it's current status is, for a long time it didn't work very well, leading some people to seek out DIY ways of doing it or, possibly in a few cases, jump ship entirely.  
 
So, rather than do a full DIY, there is the possibility of a hybrid approach: you leverage the drivers of a more mature and/or well maintained home automation software (after all, why re-invent the wheel?), but if it has a sufficiently exposed API, maybe you can go off and do whatever you need to do using tools that you most want to use.  I'm not sure which HA software is most amenable to that, but I'd be curious to know.
 
Yes, that would be a good approach as long as the application design is done well enough to separate the front end from the back end and exposed by API. In my solution the front end user interface is driven by HTML5 (the base page providing basic UI services like network, and pages for the dashboard/designer, tabs and settings hosted in an Iframe) with all the automation logic accessed on the server via websockets and uses a message based protocol (current version JSON based but next version will use MQTT). The problem is (as Dean is alluding to above) that to have a rich UI still requires a lot of work, even with toolsets like HTML5 and libraries like Bootstrap and jQuery. Using a widgets based system goes a long way to modularizing the UI where the logic and behavior of the UI element is contained in the widget (eg. a light widget has a dimming function that changes the CSS transparency of the widget image on the screen) but this requires more work to setup and manage the infrastructure for the widgets and also to provide the end user with a design surface to manipulate widgets to design screens.
 
On a different topic, there are a new breed of automation solutions coming on the market with the 'internet of things' hype but most of them are not as functional as the solutions 'pre-IoT' and have an emphasis on UI (especially mobile). I notice that many of these new HA solutions are cloud based, which I think is a fundamental flaw because of the latency to communicate with the cloud causes small delays (especially noticeable when sending commands, like switch on a light) and more importantly if the cloud or the network is down the automation is down. IMO a local hub that handles both the device interfacing as well as the automation logic is better design, still use the cloud but to augment the hub (eg. use WhatsApp for mobile notifications). What do others think about these new systems (like Smartthings or Ninja Sphere)?
 
IMO a local hub that handles both the device interfacing as well as the automation logic is better design, still use the cloud but to augment the hub (eg. use WhatsApp for mobile notifications).
 
What do others think about these new systems (like Smartthings or Ninja Sphere)?
 
Personally its a "win win" for the transport providers (wireless mobile and wired ISP) and big data providers and the companies that make those little doo what devices that sit in your home. 
 
The big data boxes are still on the ground (planet Earth) and not in the heavens or on orbiting earth data platforms yet.
 
It's cheap and easy relating to automation for the masses.  
 
There is that world of trust that all good comes from the heavens (cloud) or a wireless mobile automation widget that comes from an innate attraction. 
 
Cloud based services are a different way of monetizing - instead of a fee for the initial purchase, money is made through regular service subscriptions. For the closed source home automation solutions a subscription model probably makes sense due to the ongoing need to develop drivers for integrating new equipment (open source is somewhat easier - the community helps with the development) else it will be tough to make money as HA is still fairly niche, and likely to stay niche until it is easier for the masses and more functional (fragmented standards being a major drawback today).
 
Back
Top