Frustrated with home automation software

BTW, your post pretty much demonstrates the point I was making above. You can do a thousand things, but if you don't do that one thing that any given customer wants, they will ignore the thousand things and not commit. Given that every customer may have a different one thing, you effectively can never really have enough device support.
 
[Ops, got bitten by the someone else posted while I was replying bug...]
 
NeverDie said:
I've always wondered how the so-called Internet of Things would ever come to grips with this issue.  I'm curious how the driver aspect of home automation compares in magnitude to the more visible aspects like the conceptual design and user interface, as well as basics like event scheduling.
 
Is writing hundreds of drivers, say, 50% of a typical home automation software development?  Or is it more like 80%?  
 
It sort of depends on the size/capabilities of the system overall I guess. It's certainly not anywhere near as large as 50% for us. CQC is coming up on a million lines of code. The device drivers may make up, I dunno, 50,000 lines altogether, something like that. It can be a little misleading though, because the bulk of our drivers are in CML or PDL, both of which are very much designed to minimize the amount of work required, i.e. they have lots of built in functionality designed to make it easier to write a driver.
 
But, what device drivers might lack in code bulk, they can certainly make up for in tediousness. Many devices are not well documented, or incorrectly documented, or have bugs, or have many versions over time that are slightly different, or require the driver writer to be practically an expert in the device itself (and how many complex devise can you really be highly knowledgeable about given that you will never actually use them all yourself probably.)
 
Still, in the grand scheme of things they pale compared to the rest of CQC. You have to remember that CQC isn't written in a high level language, using lots of third party code. We do almost everything ourselves, so that we can control the quality. So it's been a huge effort. We have everything from our own custom build tools, to our own standard object libraries for core stuff (strings, memory buffers, sockets, serial ports, synchronization services, threads, streams, collections, processes, file system, etc...), our own implementations of many technologies (e.g. PNG, ZLib, HTTP, Websockets, Web Server, regular expressions, text transcoding, validating XML parser, JSON, image manipulation, CD ripping, various cryptographic standards, metadata extraction, etc...), we have our own ORB technology and IDL language/compiler which is a huge benefit, our own object oriented language with virtual machine and IDE, we have our own virtual kernel that encapsulates all access to system services so our entire system is built in terms of our own object interfaces, and we have our own entire custom windowing system (we only use the Windows frame window, everything else is ours.)  And that's all before CQC itself. CQC is built on that stuff, and is something around half'ish the size of the of the general purpose code base.
 
Compared to all of that, drivers are tedious and time consuming but not so much of a challenge other than in their bugs and inconsistencies, and having sufficient access to the devices. Far and away the heaviest aspect of CQC itself is the interface viewer and designer system. It's brutally complex, as you could imagine. It's a full bore object oriented, graphical application design system with a very nice graphical designer, with full integration into the product. So it's killed more brain cells than anything other single part of the system.
 
* I hadn't run the line counter for a while, so I just did it. The C++ code comes out to 962,867 lines of code. If the CML code for drivers was counted, I guess we'd be almost there already, but the line counter doesn't understand CML files, so it can't count them. But, with the CML code, likely it'll hit a million for the upcoming 4.7 release.
 
swamiller said:
Dean and Chris, I really appreciate the point of view of the software provider.  Both of you have very robust applications. CQC is closer to filling my needs because it contains drivers for my Omnipro controller and most of the other devices I use. It looks as though CastleOS has a nice product as well but seems more focused on pulling together several random home automation/entertainment devices (mostly wireless) into one interface to control them all.  The usage of the Kinect is very intriguing!  But no interface with my Omnipro or support of zigbee that I could see from the website.
 
The unfortunate part for both, and the primary frustration that started this thread to begin with, is neither can fill common gaps that most of us have with our systems...and none of them are the same gap, which from a software perspective appears to be the big challenge.  The other is evolution and keeping us with whats new and not abandoning the legacy users or worse, quitting the game all together leaving the users to fend for themselves.
 
I have tried Elve and I was able to make a pretty impressive interface using the provided templates with relative ease.  Elve unfortunately has an unknown future at this point, however,  because the primary developer stepped away from the project.  I have also tried the v4.5 trial of CQC and I was able to eventually make it work but I wasn't happy with the initial results (which was no doubt a user head-space issue). With my first attempt using CQC I had issues controlling my UPB devices via the Onmi and I ended up letting the trial expire before attempting to figure out how to get under the hood and resolve them.  Because I lean more toward using a central, non-computer controller (Omnipro II in my case) with software as an enhancement, CastOS doesn't seem like it would be suitable for my environment.  I also tend to lean toward wired devices when at all possible with exception of my zigbee door locks and thermostats which don't appear to be supported by CastleOS.
 
The missing links for me seems simple.  As Chris mentioned, entertainment systems are a large part of many home automation systems.  For me, it's not the distribution of audio that is a frustration, it's controlling my media and video library the way I want to control it.  I use Plex and Plex server for this task because it's easy, it's simple, and it controls everything I need it to control and not a bit more.  Most home automation software seems to lean toward XBMC or other semi complete media management applications.  I have dabbled in XBMC in the past but it doesn't "just work" like Plex does.  I can change, but I need a complete solution to push me over the edge.
 
Today I have resorted away from bringing in any new home automation software for the time being to let some of the war between vendors play out a little more. I currently use Haiku for my "always functional" home automation control software. I use iRule for a pseudo all in one remote because it will talk to my entertainment devices as well as the Omni, although performance is a bit to be desired. I am also playing with Roomie remote because it does more for me on the entertainment side by presenting my FIOS tv guide (and all metadata) on my iPad (vs. looking at the tv screen to select a show to watch) and does the same for my Plex Home Theater content.  All the movies are on the tablet device to be browsed and selected vs. looking on the tv screen and scrolling. On the down side, Roomie does not interface with my Omni at all and I'm not happy with the look and feel of the remote buttons. So the search continues...
 
Dean, CQC is very close to being a solution for me but I hesitate to invest in an application that won't tie the knot between my home and entertainment. I'm not married to the media manager I am using but I haven't seen another one that does what I want it to do. I also have had a hard time wrapping my head around how to manipulate and customize the software which might be part of my problem. I know it's a learning curve issue but that is an issue.

Chris, great product!  I just can't use it with MY house :(
 
@lupinglade - I'm not sure if you are monitoring this thread, but I am in daily suspense waiting for the next generation of Haiku (called Space) to come out and I hope it can fill some of the gaps for me.  Haiku has been a solid solution for me although it too lacks integrations between my house and my entertainment needs.
 
Hey Swamiller! You're right about our device support being focused a bit more on the newer devices - that's simply because that's where the new automation customers are coming from. We're working top down to complete our compatibility with legacy systems though, so hopefully by the end of 2015 we'll be able to meet all your needs!

One thought in the meantime - if you're looking for Plex control you can download our Kinect Service (voice control) app separately and run it just to control Plex by voice, with no need to license the software or pay us anything. We have a very active forum user who's writing custom Plex commands as well, for instance he can query his Plex server to get a list of new shows and movies. 
 
NeverDie said:
I've always wondered how the so-called Internet of Things would ever come to grips with this issue.  I'm curious how the driver aspect of home automation compares in magnitude to the more visible aspects like the conceptual design and user interface, as well as basics like event scheduling.   Is writing hundreds of drivers, say, 50% of a typical home automation software development?  Or is it more like 80%?  
 
For us right now it's about 50%, but I expect it to grow to 80% as we complete the rest of our GUI features that are in development. 
 
Dean Roddey said:
It sort of depends on the size/capabilities of the system overall I guess. It's certainly not anywhere near as large as 50% for us. CQC is coming up on a million lines of code. The device drivers may make up, I dunno, 50,000 lines altogether, something like that. It can be a little misleading though, because the bulk of our drivers are in CML or PDL, both of which are very much designed to minimize the amount of work required, i.e. they have lots of built in functionality designed to make it easier to write a driver.
 
But, what device drivers might lack in code bulk, they can certainly make up for in tediousness. Many devices are not well documented, or incorrectly documented, or have bugs, or have many versions over time that are slightly different, or require the driver writer to be practically an expert in the device itself (and how many complex devise can you really be highly knowledgeable about given that you will never actually use them all yourself probably.)
 
Still, in the grand scheme of things they pale compared to the rest of CQC. You have to remember that CQC isn't written in a high level language, using lots of third party code. We do almost everything ourselves, so that we can control the quality. So it's been a huge effort. We have everything from our own custom build tools, to our own standard object libraries for core stuff (strings, memory buffers, sockets, serial ports, synchronization services, threads, streams, collections, processes, file system, etc...), our own implementations of many technologies (e.g. PNG, ZLib, HTTP, Websockets, Web Server, regular expressions, text transcoding, validating XML parser, JSON, image manipulation, CD ripping, various cryptographic standards, metadata extraction, etc...), we have our own ORB technology and IDL language/compiler which is a huge benefit, our own object oriented language with virtual machine and IDE, we have our own virtual kernel that encapsulates all access to system services so our entire system is built in terms of our own object interfaces, and we have our own entire custom windowing system (we only use the Windows frame window, everything else is ours.)  And that's all before CQC itself. CQC is built on that stuff, and is something around half'ish the size of the of the general purpose code base.
 
Compared to all of that, drivers are tedious and time consuming but not so much of a challenge other than in their bugs and inconsistencies, and having sufficient access to the devices. Far and away the heaviest aspect of CQC itself is the interface viewer and designer system. It's brutally complex, as you could imagine. It's a full bore object oriented, graphical application design system with a very nice graphical designer, with full integration into the product. So it's killed more brain cells than anything other single part of the system.
 
* I hadn't run the line counter for a while, so I just did it. The C++ code comes out to 962,867 lines of code. If the CML code for drivers was counted, I guess we'd be almost there already, but the line counter doesn't understand CML files, so it can't count them. But, with the CML code, likely it'll hit a million for the upcoming 4.7 release.
 
I give you props for rolling your own in every way and pulling it off...that isn't easy to do! We're based on .NET so we pretty much spend 50% of our time building GUIs, and 50% working on APIs right now. All that other stuff is built in to the foundation. 
 
Great discussions and this has been a pleasure.
 
Well, I think we have generally shown the premise that Home Automation is not fulfilling the needs of the customers in general and there is frustration from both the Software Vendors and the End Customers.
 
When these type of conditions of 100s or maybe 1000s of interfaces are needed, then an Abstraction Layer is needed.
 
Software Vendors are frustrated that they need to develop proprietary interfaces for all the different devices/protocols/Hardware/etc.
There needs to be a layer that abstracts away from the proprietary interfaces needed to control all these devices.
The HARDWARE vendors need to support a software protocol that is commonly understood and that does not vary from one system to another. 
IMHO, SNMP was defined for the exact things we are talking about and fulfills all the needs I have been able to come up with.
Then the Vendor that build the interfaces to these devices should be happier as they only need to use SNMP to communicate.
 
Why not use SNMP?
 
End Customers are frustrated with the limits to the breadth of integration. So End Customers end up using more than one of the Software Vendor Products and are frustrated on how proprietary and difficult it is to integrate these Software Vendor Products.
 
The Software Vendors should consider providing a WEB Services API into their Products.
WEB Services API would abstract each system so that it could be interacted with in a reasonably open manner.
The API would allow the Software Vendors to provide the "protection" that some of them seem to think they need for support.
 
What are we waiting for?
 
-jim
 
I'm not sure that SNMP would be the appropriate all seeing, all dancing protocol, nor would it make THAT much difference to the automation systems even if such a thing was used. Any such protocol would just be the means of transporting the data. It won't make interpreting and dealing with the data any easier. It won't make errors and different versions of devices and inconsistencies go away, etc...
 
Really making it easier would involve a much higher level effort, something like what I was discussing above, e.g. standard device classes that provide standard interfaces for specific types of devices. And of course you then get into all the issues of composite devices that don't fall cleanly into any single such interface, and on and on.
 
That would simplify things, though it would also of course reduce the functionality exposed to core stuff that is likely to be commonly available. And you get into a situation where you are willing to just write off some devices (because they cannot meet that standard) or you end up with a standard that doesn't simplify anything because everything is optional so you can't depend on anything.
 
It's a very complex issue and that's why it's not been solved. We have gone through this very process over the last year to define our standard device interfaces, and all of the above issues come up. Some devices will just not be supportable under our standard because they just aren't 'good enough', so those devices get left out of the auto-magical functionality that these standards are based on.
 
Z-Wave is a good example of a standard that effectively becomes non-standard because they didn't enforce minimal standards, so all kinds of modules exist that do things this way or that or can't provide this functionality or that. UPnP is similar. It defines standards but lots of it is optional. You can query what is implemented but that doesn't really help. It makes it very complex to support because you have to dynamically adjust all user customization to deal with the hardware you have available.
 
We have said that this is the standard, and if hardware doesn't meet it, or cannot at least fake it, then it can't be supported. That way we can make reasonable assumptions about the hardware, at the risk of alienating customers whose hardware isn't sufficient, such as Z-Wave modules that don't support associations for example. We don't support them in our V2 enabled Z-Wave driver, because it's just too low end and impractical. But there are a lot of such modules out there. You lose one way or the other.
 
SNMP is a IP network protocol. In other words, you need an IP network. Home automation needs a mesh network that is outside the scope of SNMP. There are some working on it, check out Thread Group. 

http://www.threadgroup.org/
 
this is not an endorsement of thread, I'm neutral on it, just pointing out groups are pursuing new "unifying" protocols. I think we're still at least 5 years away from any sort of cohesion, however. 
 
Dean Roddey said:
I'm not sure that SNMP would be the appropriate all seeing, all dancing protocol, nor would it make THAT much difference to the automation systems even if such a thing was used. Any such protocol would just be the means of transporting the data. It won't make interpreting and dealing with the data any easier. It won't make errors and different versions of devices and inconsistencies go away, etc...
 
Really making it easier would involve a much higher level effort, something like what I was discussing above, e.g. standard device classes that provide standard interfaces for specific types of devices. And of course you then get into all the issues of composite devices that don't fall cleanly into any single such interface, and on and on.
 
That would simplify things, though it would also of course reduce the functionality exposed to core stuff that is likely to be commonly available. And you get into a situation where you are willing to just write off some devices (because they cannot meet that standard) or you end up with a standard that doesn't simplify anything because everything is optional so you can't depend on anything.
 
It's a very complex issue and that's why it's not been solved. We have gone through this very process over the last year to define our standard device interfaces, and all of the above issues come up. Some devices will just not be supportable under our standard because they just aren't 'good enough', so those devices get left out of the auto-magical functionality that these standards are based on.
 
Z-Wave is a good example of a standard that effectively becomes non-standard because they didn't enforce minimal standards, so all kinds of modules exist that do things this way or that or can't provide this functionality or that. UPnP is similar. It defines standards but lots of it is optional. You can query what is implemented but that doesn't really help. It makes it very complex to support because you have to dynamically adjust all user customization to deal with the hardware you have available.
 
We have said that this is the standard, and if hardware doesn't meet it, or cannot at least fake it, then it can't be supported. That way we can make reasonable assumptions about the hardware, at the risk of alienating customers whose hardware isn't sufficient, such as Z-Wave modules that don't support associations for example. We don't support them in our V2 enabled Z-Wave driver, because it's just too low end and impractical. But there are a lot of such modules out there. You lose one way or the other.
But, these same "complex" issues have been solved in other areas of technology. SNMP is widely used across all kinds of devices around the world. It can certainly does have "standard" device classes as are laid out for most common network equipment.
 
And the market would decide about the low end devices that do not support the "standard". Got any IPX devices laying around?
 
ChrisCicc said:
SNMP is a IP network protocol. In other words, you need an IP network. Home automation needs a mesh network that is outside the scope of SNMP. There are some working on it, check out Thread Group. 

http://www.threadgroup.org/
 
this is not an endorsement of thread, I'm neutral on it, just pointing out groups are pursuing new "unifying" protocols. I think we're still at least 5 years away from any sort of cohesion, however. 
Yes, I agree, SNMP is defined as a IP protocol. I am not convinced that is a limitation. (But I am not convinced it is not either)  RaspberryPI can run SNMP.
 
Why do you think that Home Automation needs to be a mesh network, is beyond me. But, Shortest Path Bridging will allow Ethernet to be connected in a mesh.
 
I will check out the http://www.threadgroup.org/.
 
What about the http://www.openhab.org/ or (sort of a spin off ) https://eclipse.org/smarthome/. These are both "vendor" agnostic projects and at least Eclipse has a well thought out architecture.
 
We need to get to the point, like we are with Networking in general, where how things are connected and which OS the things are running is a non-issue.
 
Again, really appreciate all the insights and discussions.
 
-jim
 
ChrisCicc said:
SNMP is a IP network protocol. In other words, you need an IP network. Home automation needs a mesh network that is outside the scope of SNMP. There are some working on it, check out Thread Group. 

http://www.threadgroup.org/
 
this is not an endorsement of thread, I'm neutral on it, just pointing out groups are pursuing new "unifying" protocols. I think we're still at least 5 years away from any sort of cohesion, however. 
 
Sorry, I can not deal with any more of the "standards" where the minimum entry to participate is $2,500.00. These are vendor consortiums and no way to develop a standard.
-jim
 
-jim said:
Sorry, I can not deal with any more of the "standards" where the minimum entry to participate is $2,500.00. These are vendor consortiums and no way to develop a standard.
-jim
 
Not sure you understand how these things tend to develop, that's pretty common and pretty inexpensive. You need to pay to get a seat at the table... 

Shortest path bridging is being misapplied here... it's for finding routes through a network, not building the network. That's why you need mesh... 
 
The O.P. is absolutely correct.  There should be standards.  A lot of this is reinventing already standardized protocols and features used to day to communicate across the world (and solar system).   I am convinced the reason they don't is a combination of incompetence and the fact that if they did use standards they would have a harder time patenting it.
 
SNMP would work fine as a foundation, the problems with it are that it only provides a way to send low level data.  A higher level of standard would be needed to define what a device is and the limits etc.  I.e. You could send the dim value of a light no problem, but you still need to define what are the limits, is it a percentage/0-255/lumens so that all systems understand the data.  And that seems to be where all these standardization attempts seem to break down.   SNMP is very complex for what it does, it was overengineered.  I have some first hand experience since I wrote the CQC SNMP driver from scratch (no libraries) using the RFC's for reference.
 
BTW, Mesh/Broadcast is actually a limitation that, again, networking moved away from 20? years ago.  It limits the size and scope of the solution and is a waste of resources.  There are much better ways to implement a communication protocol.
 
ChrisCicc said:
There are standards - the problem is too many of them, and the "solution" is more standards. 
 
Obligatory xkcd ref:
 
standards.png

http://xkcd.com/927/
 
;)
 
Craig
 
Back in the 80s, the music industry decided to embrace one communication standard for electronic keyboards.  MIDI.  After that, any brand of keyboard could talk to any other.  PCs and Macs could get an interface to talk to all of them. 
 
It caused an absolute EXPLOSION of sales in the industry.  Everyone sold a ton of product.  The consumer bought it because it made it easy and logical.
 
If the automation and consumer electronics industry would embrace that idea, we would see an explosion there too.  But unfortunately, in today's world, everyone thinks the answer is to control the protocol.
 
Back
Top