Brultech ECM-1240 Software Development

[signal15]

You guys may want to take a look at the Brultech thread over on the CQC forums (Beta Drive Developments sub-forum). Brultech has a version of their server software that will stream the info over a TCP/IP connection. It will support up to six ECM's and may even have Google PowerMeter support in it. It's currently in beta.

I'm running this now, but I have 4 issues with it:
- Doesn't run as a service, I have to log in and start it whenever the box reboots
- I have to maintain a windows box to run it
- It's definitely beta
- No support for different costs like if I have main panel and an off-peak panel just for electric heat. My main panel is 6.5 cents/kwh, and a heating panel is 2 cents/kwh. The software assumes the same cost for everything.

This perl program can run on one of my linux boxen. RRD is awesome because it will aggregate old data automatically. And with RRD, you can apply different cost calcs to different channels, or integrate temperature data, HVAC setpoints/runtime, etc into your graphs to correlate data and figure out how outside temp or setpoints affect your heating costs. There are also tools to monitor the RRD data and send alerts based on different criteria, and a ton of tools to do other sorts of data analysis. RRD has been around for a long long time, and has become the standard for storing and dealing with this type of data.

I used to use it to figure out my bills from my ISP's when I worked for a large news organization to make sure those ISP's weren't screwing me when I got my monthly bill.

 

[signal15]

Do you have the command you used to create your RRD Database?

I have the tool running fine, but I'm just printing the data to the screen right now.

 

[mgandalf]

I didn't look over the code yet. Couple of questions:
- How do you handle multiple units?

I handle this by unit number - I'm not sure if that's the correct way. That could be easily changed to use the address field through. Running in server mode, the main process listens for new connections and will fork off a new process for each one. Keep in mind, I only have one unit, so this has never been tested.

- How does the perl script handle counters that wrap? The python one that is out there didn't handle these well, and you'd end up with enormous spikes in your database that didn't exist.

Can you explain further wrapping counters? Do you mean counters that get large enough to eventually go back to 0? I suspect this may happen over time. I have a method which sends commands to the ECM-1240 and one of them is to clear the counters when the script starts. Unfortunately, I couldn't get this working but I've not spent a lot of time on it either. The command seems to be "RQSRKW".

I wanted to get this all dumped into RRD, then then use something like this to generate my graphs/charts:

<url removed so I can post>

There used to be a demo of Jart up somewhere, but I can no longer find it. There was a way to gather cost information and stuff also.

I use cacti for my graph generation and it fits my needs pretty well.

One more thing that I didn't mention. The script running in client relay mode will also listen for commands from Brultech software and emulate them. Currently it will emulate TOGOFF (stop sending packets), TOGXTD (start sending again) and SPK (send one packet). I'm sure there are more which could be emulated or potentially passed through.

- Mark.

 

[mgandalf]

Do you have the command you used to create your RRD Database?

I have the tool running fine, but I'm just printing the data to the screen right now.

I used Cacti to create mine. I simply setup all the data templates/sources/inputs for each rrd. I set the input to "Script/Command" and used something like "cat /var/spool/cacti/brultech_aux1watts.out" and added the parameter/value pair to each file by hand like "Aux1_Watts:0" for the above file. I disabled the poller in cron and ran it by hand one time only to create the rrds.

- Mark.

 

[Ira]

I'm running this now, but I have 4 issues with it:
- Doesn't run as a service, I have to log in and start it whenever the box reboots
- I have to maintain a windows box to run it
- It's definitely beta
- No support for different costs like if I have main panel and an off-peak panel just for electric heat. My main panel is 6.5 cents/kwh, and a heating panel is 2 cents/kwh. The software assumes the same cost for everything.

Your first issue is the only one that bothers me. What makes it worse is that if after opening the server window, I click close instead of minimize, it will close it out completely without confirmation. If you don't pay attention, you don't notice that it closed, and you end up losing the data while it is closed. Hopefully this will be addressed soon.

I've been using the latest code available (but not the beta) for one ECM connected via wireless for a week or so. It has been stable for me. Three more ECM's on order should be in next week so I will have to move to the beta code soon.

On your fourth issue..if you are picking up the streamed data, can't you manipulate it in the process so that it "corrects" the cost?

Wish I had your energy cost. Mine is 11.7 cents and that's pretty cheap around here.

 

[signal15]

On your fourth issue..if you are picking up the streamed data, can't you manipulate it in the process so that it "corrects" the cost?

Wish I had your energy cost. Mine is 11.7 cents and that's pretty cheap around here.

You mean like multiply the data by .30 or whatever? Then my cost would be accurate, but the power usage would not be.

I have no idea why it's so cheap here. They even have another program where you can get another meter that only works from 10pm to 6am, and it's 1 cent/kwh. The idea is that it's for heat storage systems (like hot water, rocks, etc), and you charge them up at night and heat during the day with them. I ran across a 60kva UPS from a data center for free a few years ago, I should have grabbed it. I could have charged it up at night, and run my house off of it during the day. Traditional wet cells probably wouldn't last long being cycled like that, but good AGM batteries with a good desulphation device on them probably would. I wanted to try it more for an experiment than anything else. I have my electrical guy looking for one for me. He does a lot of commercial work and runs across them occasionally, and he's excited to try it out.

 

[Ira]

On your fourth issue..if you are picking up the streamed data, can't you manipulate it in the process so that it "corrects" the cost?

Wish I had your energy cost. Mine is 11.7 cents and that's pretty cheap around here.

You mean like multiply the data by .30 or whatever? Then my cost would be accurate, but the power usage would not be.

I don't know what data the ECM server is streaming in the beta version, but if you are getting the number of KWH on a per channel basis, you could multiply that by the cost factor for the specific channel and save that in a different field. You would end up with two fields for each channel...one for the actual KWH and one for the cost. Alternatively, only save the KWH and do the cost calculation whenever you need to display the cost or report it.

Since I don't have a clue what data is being sent or how the API works, it's hard for me to visualize what can be done with it. My main point is that you may want to consider leaving the interaction with the ECM devices to Brultech code as long as he is providing a stable server that is providing the data you need via a useable API. If nothing else, it insulates you from changes at the device level.

 

[mgandalf]

Do you have the command you used to create your RRD Database?

I have the tool running fine, but I'm just printing the data to the screen right now.

I used Cacti to create mine. I simply setup all the data templates/sources/inputs for each rrd. I set the input to "Script/Command" and used something like "cat /var/spool/cacti/brultech_aux1watts.out" and added the parameter/value pair to each file by hand like "Aux1_Watts:0" for the above file. I disabled the poller in cron and ran it by hand one time only to create the rrds.

- Mark.

I've uploaded an archive containing empty rrd files in that same directory which will give you a start. Keep in mind their interval is 1 second.

 

[signal15]

Do you have the command you used to create your RRD Database?

I have the tool running fine, but I'm just printing the data to the screen right now.

I used Cacti to create mine. I simply setup all the data templates/sources/inputs for each rrd. I set the input to "Script/Command" and used something like "cat /var/spool/cacti/brultech_aux1watts.out" and added the parameter/value pair to each file by hand like "Aux1_Watts:0" for the above file. I disabled the poller in cron and ran it by hand one time only to create the rrds.

- Mark.

I've uploaded an archive containing empty rrd files in that same directory which will give you a start. Keep in mind their interval is 1 second.

Awesome! Thanks!

On a side note, I had my interval set to 15 seconds because Brultech's software throws a warning if it's not set this way. However, since I started the server and replicated data to it from the Brultech software, I've noticed that it's sending data every second instead. Does your server software set this somehow??

On a side note, can one of the admins enable mgandalf's PM capability? He's clearly not a spammer.

 

[mgandalf]

Awesome! Thanks!

On a side note, I had my interval set to 15 seconds because Brultech's software throws a warning if it's not set this way. However, since I started the server and replicated data to it from the Brultech software, I've noticed that it's sending data every second instead. Does your server software set this somehow??

On a side note, can one of the admins enable mgandalf's PM capability? He's clearly not a spammer.

No, the only command I send is to clear counters. Of course, I could be completely mistaken on what that command does, but I don't think so.

BTW, I just uploaded v0.84. I'm finding it challenging keeping up with a 1 second interval and making rrdtool happy at the same time. I believe I have it working where no increment is 0 and no increment is skipped. I'm sure I have a bug somewhere in my socket listener that's causing the issue but for now I've worked around it. I would be easier of the Brultech device had an ntp daemon to sync up clocks. :rockon:

- Mark.

EDIT: When you say you're seeing a 1 second increment - you may not. Your ECM-1240 may be sending every 15 seconds, but my script running in client mode will still poll the IPC every second. So, if you see 15 1-second intervals of identical readings, you'll know your device is still set at an increment of 15.

I'll add the ability of auto-detecting the increment soon. First I need to sniff the communications between Brultech software and the ECM-1240 (unless someone is willing to share an SDK?).

 

[signal15]

Awesome! Thanks!

On a side note, I had my interval set to 15 seconds because Brultech's software throws a warning if it's not set this way. However, since I started the server and replicated data to it from the Brultech software, I've noticed that it's sending data every second instead. Does your server software set this somehow??

On a side note, can one of the admins enable mgandalf's PM capability? He's clearly not a spammer.

No, the only command I send is to clear counters. Of course, I could be completely mistaken on what that command does, but I don't think so.

BTW, I just uploaded v0.84. I'm finding it challenging keeping up with a 1 second interval and making rrdtool happy at the same time. I believe I have it working where no increment is 0 and no increment is skipped. I'm sure I have a bug somewhere in my socket listener that's causing the issue but for now I've worked around it. I would be easier of the Brultech device had an ntp daemon to sync up clocks.

- Mark.

EDIT: When you say you're seeing a 1 second increment - you may not. Your ECM-1240 may be sending every 15 seconds, but my script running in client mode will still poll the IPC every second. So, if you see 15 1-second intervals of identical readings, you'll know your device is still set at an increment of 15.

I'll add the ability of auto-detecting the increment soon. First I need to sniff the communications between Brultech software and the ECM-1240 (unless someone is willing to share an SDK?).

The 1240 is definitely sending a packet every 1 second. I'm relaying the packets to the box running the perl script through the box running the ECM1240 monitoring software. That software has a packet counter that used to increment once every 15 seconds. Now it's incrementing once every 1 second.

Those counters DO roll over. They rolled over on several locations when I was dumping the data into a Mysql database and uploading to Wattzon with ecmread.py. I use a lot of power, especially this time of year.

For values with counters, I assume you are just inserting the delta from the previous reading into the RRD database. If it runs long enough to where it rolls over a counter back to zero, what is your software going to do? Even if you're resetting the values to zero when the server starts, it's eventually going to happen.

The watt-second counters for CH1 and CH2 are 5 bytes long. The watt-second counters for the AUX channels are 4 bytes. There is also a seconds counter somewhere, but I'm not sure when that rolls over. It *DOES* roll over though. The ecmread.py software accounts for the seconds rollover, but does not account for the rollover of other counters. At least, this is how it used to be. I stopped using it for this reason because I would regularly get rolled over AUX channels that would cause me to get 30 million+ Watt spikes in my graphs and I got sick of going in and removing the spikes. In any case, if you account for the rollover, there is no reason to reset the counters. It's possible to have multiple software packages reading data from the thing anyway, so if you reset the counters, you could screw up the readings and calculations that in the other software.

 

[damage]

i reset the counters at midnight. i use this command string to reset the counters 0xFC 0x52 0x51 0x53 0x52 0x4B 0x57

 

[personalt]

You mention watts/second. Are they really watts per second or are the KWH? In another thread someone posted what their unit was echoing. It appears that the values that echo for KHW allow for at least 4 places to left of decimal and 3 to the right. If that is true it supports at least 10,000 KWH on a single channel before it rolls.. Am I totally off base here?

How big does it take before it rolls?

CH1Pb=53&CH2Pb=2&Paux1b=0Paux2b=6&Paux3b=0Paux4b=0&Paux5b=0KWhCH1b=66.025&KWhCH2b=301.318KWhAux1b=4. 91&KWhAux2b=1.128KWhAux3b=1.723&KWhAux4b=1194.006K WhAux5b=4.806&Timeb=11:10:44

The 1240 is definitely sending a packet every 1 second. I'm relaying the packets to the box running the perl script through the box running the ECM1240 monitoring software. That software has a packet counter that used to increment once every 15 seconds. Now it's incrementing once every 1 second.

Those counters DO roll over. They rolled over on several locations when I was dumping the data into a Mysql database and uploading to Wattzon with ecmread.py. I use a lot of power, especially this time of year.

For values with counters, I assume you are just inserting the delta from the previous reading into the RRD database. If it runs long enough to where it rolls over a counter back to zero, what is your software going to do? Even if you're resetting the values to zero when the server starts, it's eventually going to happen.

The watt-second counters for CH1 and CH2 are 5 bytes long. The watt-second counters for the AUX channels are 4 bytes. There is also a seconds counter somewhere, but I'm not sure when that rolls over. It *DOES* roll over though. The ecmread.py software accounts for the seconds rollover, but does not account for the rollover of other counters. At least, this is how it used to be. I stopped using it for this reason because I would regularly get rolled over AUX channels that would cause me to get 30 million+ Watt spikes in my graphs and I got sick of going in and removing the spikes. In any case, if you account for the rollover, there is no reason to reset the counters. It's possible to have multiple software packages reading data from the thing anyway, so if you reset the counters, you could screw up the readings and calculations that in the other software.

 

[damage]

it's watt/sec. i've sniffed the serial port & the values correspond to what the brultech software was displaying as watts/sec

 

[tenholde]

it's watt/sec. i've sniffed the serial port & the values correspond to what the brultech software was displaying as watts/sec

It is watt-secs (not watt/sec).

Counters max values (and hence rollover point) are (to the best of my knowledge):

5 byte counters: 1,099,511,627,775
4 byte counters: 4,294,967,295
3 byte counters: 16,777,215

The transmitted seconds field is 3 bytes. Max value is about 194 days.

The Ch1,2 and Polar1,2 fields are 5 bytes.
The Auxn fields are 4 bytes.

Hope this is helpful,

tenholde