ELK M1 Logging Engine / Web Dashboard

[inline]

Hi Everyone -

I wanted to gauge some interest in software to log data from the ELK M1 and produce metrics data and charts based on that data. This would not be software to control the M1 such as is already present via Premise, Homeseer, Girder, PowerHome, etc, just a program to log and report on the data for metrics reporting or historical reference.

Things like:

• 1. How many times a zone was violated
  2. How long a zone was violated
  3. Graphs of Temperatures from Thermostats or sensors
  4. Report on other system statistics

I was starting down the path of using the Perl M1 script that's available to collect data from the M1 and was then going to put the data into a sql database and use that to populate a graph, via rrdtool, maybe using Cacti. As I was starting to work on this and think of ideas, I did some research to see what others have already done instead of re-inventing the wheel.

Some have already greatly accomplished what I'm trying to do such as ThreadHead who has done an awesome job logging the data into a mysql database and then using Open Flash to produce nice charts, etc. However as he's mentioned his setup is specific to his environment and would require work to get it customized to your own setup. It's also a bit involved and might be too much for someone not well versed with linux, ruby, open flash, etc.

Ideally it would be great if an app could be developed, (maybe by some of the folks here! ) that would be simpler for more people to use to collect and report on their ELK M1. I was thinking along the lines of what Brultech has done with their ECM-1240...there is an engine piece that logs data to a sqlite database and then a web dashboard to display the data.

I wanted to get input from others on best approaches, if you've already done something similar and if so what, and what the interest is on a collaborative effort to put something like this together as I think a lot of people could benefit from it. Maybe, if instead of everyone producing custom ways to accomplish this, a project was started, a more easy to setup and use logging and reporting tool could be developed.

If a simple engine/dashboard tool would not be feasible for whatever the reason, maybe a vmware appliance based on what ThreadHead has done would reduce complexity while still giving more people access to a reporting tool for their M1.

Thoughts?

 

[murphy]

Hi Everyone -

I wanted to gauge some interest in software to log data from the ELK M1 and produce metrics data and charts based on that data. This would not be software to control the M1 such as is already present via Premise, Homeseer, Girder, PowerHome, etc, just a program to log and report on the data for metrics reporting or historical reference.

Things like:

• 1. How many times a zone was violated
  2. How long a zone was violated
  3. Graphs of Temperatures from Thermostats or sensors
  4. Report on other system statistics

.................
I was starting down the path of using the Perl M1 script that's available to collect data from the M1 and was then going to put the data into a sql database and use that to populate a graph, via rrdtool, maybe using Cacti. As I was starting to work on this and think of ideas, I did some research to see what others have already done instead of re-inventing the wheel.
.................
Thoughts?

I'm very interested in the same option /use of the basicly great ELK-M1.

The easiest way would be: an option in the rules to write something into the log-file.
But until now you can:
- display any text and value at the Keypad,
- send any voice-infos to any phone
- send textmessages to any cellphone
- send emails as necessary,
- but you can not use the internal logfile for notes on condition by rules.

This would be a very easy extension of the firmware ELK-RP (which seems written under MS visual Basic ? or visual C++ ?)

Maybe any next version of the ELK-RP handling programm can do it.
There was no new version during more than a year!

 

[KentDub]

I have thought about adding similar functionality to my system when I get it up and running (I have a long list of custom software development to do =) )

I'm a .NET & Windows guy, so my implimentation will start simply with a custom system event log and performance counters. The process would be coded as a Windows service to run constantly in the background. This approach lets me keep everything very standardized and will integrate with some existing enterprise-class applications I run. The performance counters can be viewed with the built-in system utility, and configured to log to text files (and better!) a sql server. From there, you can use off the shelf products or write something custom to view the data in fancy graphs. I won't be doing any reporting - I'm more interested in the logs/counters for it's historical value, diagnostics, and debugging. However, even at this point you can still use the built-in viewers to get to the data. When I get around to this (It might be a month or two) I would be interested in working with other people, even if nothing more than testing.

Event Viewer: Start->run->'Eventvwr.msc' (no quotes)->ok
Performance Monitor: Start->run->'Perfmon.msc' (no quotes)->ok

 

[WayneW]

This would be a very easy extension of the firmware ELK-RP (which seems written under MS visual Basic ? or visual C++ ?)

I am not sure that I agree or understand... ElkRP is only connected to the panel when making changes. So how would that play here?

Basically, it needs to be something that receives all the Elk status updates sent via the XEP and log/analyzes them. I basically do something close to that with HomeSeer 1.7 and Dan's excellent M1 script.

I don't think the Elk M1 internal log file is the place for this detailed logging since it would be way too small. But your request to allow rules to add entries to the Elk log certainly has merit for its own reasons.

 

[Threadhead]

As I am the current maintainer (not author) of the ElkM1--control perl script, maybe I can help put into perspective what I am using it for and how it can be expanded.

1) My 'home' server (actually a sandbox for my client projects) utilizes the ELKM1--control script to receive and log all zone,output, etc. messages to a MySQL database. The script runs as a daemon on the server forever recording it's message stream. But the message stream is just a series of events as I add some logic to transform it into greater statistical usefulness. For example, it doesn't just record that a zone opened, or closed, it records the open time and appends the closed time when it closes.


The daemon also updates another table with the status of every zone, output, etc.



Thus it creates an excellent foundation for event execution and stat generation without interfering with the collection process.

2) While I don't do any specific event processing, another daemon (service) could easily poll the status table for changes and execute from there.

3) I do have a statistics application, written in Ruby on Rails that generates some pretty little graphs and metrics.

Everything I have done is open source. But I have not spent much time on documentation. Anyone with moderate programming skills should be able to get it up and running. The only tricky part would be the RoR app, if you haven't set one up before.


What would you like me to elaborate on?

I could open my stat app up for a limited time if ya'll would like to check it out. I don't like punching holes in my firewall for very long, so it would be very limited.

 

[Threadhead]

I respectfully draw your attention to my blog post: Monitor Your Elk-M1, Log to MySQL, and Daemonize it! which has all the code for the monitor daemon, sql for table creation, and more.

 

[Threadhead]

And lest I forget, I'm just finishing the same 'package' for HAI thermostats: daemon, logging, stats, etc.

Let me know if there is any interest in this as well.

 

[iostream212]

I am very interested in doing this. I am mainly interested in storing zone states and duration until state change in sql server. Just wondering if there are any more 'turn key' solutions before I dive into this. I am not a PERL guy and would really love to save myself the hassle if something else exists. Thanks!

 

[lazyone]

Sure wish this was a reality.  A nice, pre-made VM with a database and web interface to log, graph and analyze data from the Elk M1.  I tried to implement Threadhead's solution, but I am starting from scratch with perl and never really got it going.
 
lz

 

[sda]

I see the point in logging the data (for debugging/testing) and I do so myself with my own perl scripts logging to a daily text file.
 
I don't see the point in putting this into a database, graphing, or even knowing that "the back door was opened 3 times today".
Enlighten me.
 
I get a daily summary of "interesting events" emailed to me.
 
Zone 81 is my exterior driveway motion sensor, non-alarm.   Most usually a deer in the driveway.
 
?RR is read/set clock responses.   Not decoded so the raw packet data is displayed.
 

2014/05/22 00:00:10  area   1; Armed Away, Armed, No Alarm
2014/05/22 11:57:00  ??RR    ; 045711522051411000
2014/05/22 11:57:01  ??RR    ; 005711522051411000
2014/05/22 13:30:06  zone  81; Violated, Open (9)
2014/05/22 13:30:11  zone  81; Normal, Short
2014/05/22 16:39:26  zone  81; Violated, Open (9)
2014/05/22 16:39:37  zone  81; Normal, Short
2014/05/22 23:57:00  ??RR    ; 025723522051411000
2014/05/22 23:57:00  ??RR    ; 005723522051411000

 

[Threadhead]

 don't see the point in putting this into a database, graphing, or even knowing that "the back door was opened 3 times today".
Enlighten me.

 
Why did the chicken cross the road. Because he can.
 
Nobody said analyzing all the data to know the average time the front door is open is useful. Except for those who are up all night wondering if.
 
Just having a little fun  :wacko:

 

[Threadhead]

Holy cow. I had no idea anyone was using this or even trying to use it. It has been YEARS since I even looked at the code, let alone used it in my application.
 
But it's cool to think is has staying power.

 

[lazyone]

I have thermostats connected to my Elk. My main goal was temperature and status logging and graphing of HVAC equipment. Most building management software packages include this type of data.  Understanding HVAC runtimes versus temperatures, open doors/windows and other factors can help you optimize your system for energy savings.
 
Also, frequency and duration of zone violations is interesting to me.