Powerlinc 1132 time accuracy resolution

[Xymox]

I recently dealt with a time accuracy issue with the Powerlinc 1132CU. It was not obvious initially what was going on and required a few weeks of monitoring and testing along with installing new parts in the 1132CU and testing the solution. I was successful and I have isolated a serious hardware design problem with the currently shipping Powerlinc 1132CU. Below is my report to Smarthome.

I thought it wise to get this up on forums so Google could index it and help other DIY people in the future.

Once you understand what is going on, its very clear the board layout on the 1132CU was very poor and will lead to time accuracy issues in every system depending on the amount of power line garbarge present.

Below is my email to Smarthome:

____________________________________________________________


Having read the Q&A on timekeeping issues on the PLC at Smarthome and using the current software, I still had a issue where my brand new 1132 PLC would suddenly revert to the last previously set time and then continue from there leaving hours of time offset. This seemed to occur almost randomly. This would appear to a customer as a large time offset that occurs almost randomly and could also appear as large long term time accuracy drift with moderate amounts of power line noise. This problem may also come and go depending on the content of the power line noise.

I design electronics for a living so I decided to void the warranty and investigate the problem myself. Putting test equipment on the unit and letting it run produced interesting results. First, the Li battery was fine. However there were power line spikes from my Air Cond compressor at about 100Khz that were being radiated by the power transformer in the unit and being picked up by the high impedance crystal oscillator circuit acting as a antenna because of a lack of shielding and the proximity of the xtal to the transformer - 1/8". While this caused small time offsets This was not enough to explain why the PLC was reverting to a previously set time and then continuing. However according to Maxim, the people who make the chip, intermittant 100Khz on the 32Khz xtal circuit would cause unpredictable results. This was producing a small and random time accuracy issue in addition.

It appears that when a change of power state ( plugged in / un plugged on battery ) the 1132 PLC will read the DS1307 and if the DS1307 ends up in a stuck state then it reports the last time sent to the 1307. Rather then using the DS1307 + xtal to keep time however the PLC only seems to read the RTC under certain conditions and at least once a day. I confirmed this by removing the Xtal stopping all 32Kkz operations and stopping the RTC.

Stopping the RTC reliably reproduced the issue. I can set the time, the time will increment in the PLC even with the RTC is stopped. All operations work in the PLC. Timer events occur on schedule and it keeps very accurate time during this period even though the RTC is stopped and has the wrong time internally. However upon power line removal it will revert to the last time set in the RTC. So it appears that the PLC uses the power line as its primary time freq source ( wise ) and only refers to the RTC on power up and once a day and not as a primary time source ? Thats my best guess.

After reading the App notes for the Maxim DS1307 I decided to add the Maxim DS32Khz temp controlled crystal oscillator as Maxim suggests. The low output impedance, sharp waveform edges and 1.5 V of swing make the circuit far less susceptible to induced power line garbage in this case. It also makes the RTC -far- more accurate. I also opted to NOT use the 1M resistor and cap to reduce current consumption from the battery because this would increase impedance and lead to induced RFI/EMI susceptability. I added a bigger Li battery to cover the additional current draw.

Maxim talks about the most common problems with its RTC's not keeping accurate time comes from poor shielding where RFI induces issues by being picked up by the Xtal circuit. I can confirm this occurs with power line noise in a 1132A. By removing the xtal and hooking a 30Mohm scope to X0 and X1 I captured large noise components from the power line being induced into the chip and causing the RTC to increment even though no Xtal was present. This is prob due to the long, and not recommended by Maxim, lengths of PC board trace to the Xtal. The Xtal was moved to the location by the transformer prob to thermally couple it to the transformer for time accuracy reasons. Which was smart. However the noise picked up by the long trace and even by the Xtal itself makes this design less appealing and in fact created the problem. However the 1/2" distance from the Transformer to the DS1307 itself might be a issue without shielding.

The actual time in the DS1307 is only read once a day or when the 1132 makes a transistion in power - unplugged/plugged in. At that point the PLC will read the DS1307 time and update the clock in the 1132 PLC. Just FYI.

My problem has completely vanished after adding the DS32khz chip. My PLC now keeps hyper accurate time and never has issues. I feel the complete absence of the symptoms confirms my analysis of the problem.

I think a change to the PLC should be done. Adding a DS32khz TXCO to the circuit would obviously fix the prob by lowering the suscibility of the Xtal inputs, however these are expensive in production. Switching over to the DS3231 would also be very good and replace the current DS1307 - however this requires board rework, however the control protocol looks the same so it will just drop into the circuit without the need for code changes on the CPU side. However I have not fully investigated the software interface issues.

The noise issue is covered in the DS1307 app note under the subsection "Crystal Layout Guidelines" in the document
http://www.maxim-ic.com/appnotes.cfm/appnote_number/58

Shielding or board layout will help in fixing the problem if no part or board changes are planned.

 

[upstatemike]

Would this apply to the Insteon PowerLinc 2414 as well?

 

[Xymox]

If your a DIY, open the controller, flip the board over and look at the chips. Your looking for a DS1307, its a 8 pin close to the big chip which is the microcontroller. You could also locate the xtal which was on the front side of the board under the plastic glue used to secure the transformer. This is much harder to find as the Xtal was very tiny.

Do NOT pull the glue off unless your very carefull as this will, almost for sure, pull the Xtal from the board as mine was completly covered in the glue. As I wanted to remove the Xtal the glue made it easy for me

You would have to explore the boards in other makes and variations of this controller.

There are 2 chip cap locations connected to X0 and X1 that were unpopulated on my board so there may be variations on how this was done on various boards.


The mod was easy. Its 4 wires and you pull the Xtal. Took me 30 minutes. BUT im a Ninja level modder. For other Ninja level modders: You need +5, Vbat, Gnd and the connection to the X0 input on the DS1307. All these points are avaliable on the front side of the board with a bit of probing. I just tried to take some pics with my cam phone but they were horrible.

I mounted the DS32Khz on the top side and wired to thu-holes for the +5, Vbat, Gnd and X0. I was sloppy and used wirewrap wire for the connections just to see if it worked. The wire was about 1/2" long too. Surprizingly this has been very stable, I assumed I would need better grounding, shielding and a mini-coax to the X0 pin. I will still do these things, but it is working VERY well without these refinements. This shows the much lower impeadance output of the DS32Khz has fixed the issue even with long wires !

The DANGER with opening this module is 2-fold. FIRST its powered by the battery FULL TIME. The circuit IS ALWAYS LIVE. BE CAREFUL. Second there is a cap that always has 50V DC on it until discharged. Im not sure what this is, as I have not looked, but be carefull and discharge this. Before working on this board I discharged the cap and desoldered and removed the Li battery. Then discharged all the caps again to be sure. Be carefull working on this module - ITS ALWAYS ON EVEN WHEN UNPLUGGED. You cannot meter or scope the X0 and X1 pins normally. The 1M or 10M input of the meter or scope will effect the signal. I made a 50Mohm probe to do this and used a sensitive scope.

This is a fairly advanced mod. Sorry I don't have a how-to for it. Maybe someone can do a step by step how-to on this and post it somewhere.

 

[Xymox]

I should also add that this mod produced very accurate time keeping in the RTC and in the PLC.

However as the PLC only checks and updates its internal time once a day or so from the RTC it can drift a micro amount during the day based on how accurate your power line freq is. Usally the power line is very very accurate however and much less then 1 sec a day drift. The RTC will correct any offset at some point during the day or if the power cycles.

After a month of use setting the RTC time a month ago I cannot measure any drift at all. It is clearly better then 1 second a month so far.

This increased accuracy comes from the temp inside the module being well controlled because the transformer gets warm and stays very accurate temp wise as the current drawn is constant thru it. Turns out that continous measuring of the temp of the case of the DS32Khz over weeks showed no more then 1.2 degrees F drift. The DS32Khz is extremly stable anyway so by placing it in a temp controlled enviroment it becomes even more stable.


I also need to point out that the 1132CU PLC time accuracy is ultimately controlled by the RTC accuracy as it updates the power line derived clock in the PLC once a day. So the RTC does dictate the accuracy of the PLC long term and the power line freq determins the short term accuracy below 1 day.

The mod makes the PLC hyper accurate and should not drift by more then a minute a year in any situation assuming the module is indoors and in a normal thermal enviroment and of course that power line RFI/EMI does not add any extra clocks to the 32Khz X0 signal.

 

[Xymox]

Sorry for HUGE posts..

Setting the time on a 1132CU to a exact time requires some know how..

First SET YOUR COMPUTER CORRECTLY. Use a time service, I like Tardis. Becasue of the above mods you wanna be accurate to at least a second.

Because of the way the PLC has 2 clocks you gotta be tricky..

I use the Smarthome Manager software, it does what I want so its good enough. Make sure to have the current version.

The "Sync Time To PC" option is also Ctrl-Y. Open your PC clock and wait till the time JUST passes 00. If you hit CTRL-Y before 00 you will be off by exactly 1 minute. Press CTRL-Y as close to 00 as possible.

This sets the DS1307 RTC time to 00 seconds. IT DOES NOT SET THE PLC TIME in seconds however.

To get the PLC to update to the RTC time in seconds you MUST unplug and replug the PLC. This will update the PLC seconds to the RTC seconds. After power cycling your PLC will be accurate to what you set in seconds.

Restart the Smarthome Manager software and use CTRL-G to check the PLC time. Do this over and over until the time changes minutes. It should do so at 00.

This will set and confirm the PLC is set very close to 00.

To check the time in the RTC remember to power cycle the PLC before checking the time so you get the actual time in the RTC rather then the PLC.

You can also watch the log using the smarthome manager "advanced controller" and seeing if your timed events log with the correct time. The log time seems to be the computer time. so this is a good way to check the time that the controler does things against the computer which should be within milliseconds of atomic standards if you use Tardis.

Between these posts I carefully set my 1132CU to the exact time and I will now check as the months go by to see how accurate it stays.

I think there might still be a small variation in the PLC clock which cannot be explained away by power line variation. Its corrected by the RTC updates so the PLC still stays accurate - however I have come this far and I have no idea how the PLC keeps time without the RTC. So, I may investigate further depending if the variation is very high.

 

[Xymox]

Also remember the time in the bottom right of the Smarthome Manager software is useless. Its wrong seconds after you do a CTRL-G ( get time from PLC ). It does not match anything - not the PLC, RTC or the computer.

So, power cycle the 1132CU. Do a CTRL-G , Enter over and over and watch for the minutes to change in the bottom right corner display. thats 00. This is the only way to know exactly what time the RTC/PLC is at.

Set a "watchdog" timer in your Smarthome Manager code to occur at least once a day turning on or off a phantom module at a exact time. This will show in your logs and will be time stamped by the computer. Over time this will show your drift as long as you keep your computer accurate. I have one set up for once a hour, just in case there is drift during the day.

In the last month, with the above mod, the timestamps are right to the second. Every one of them after 30 days. So my accuracy is within 1 second a month.

Befrore the mods it was all over the place, randomly.

Because the RTC and the DS32Khz are not shielded they are still susceptable to EMI/RFI. So a better solution might be to get the RTC and DS32Khz shielded. There are many sounces of RFI/EMI. Many from inside the module and some from outside. For example a 2-way radio or any RF close to the module would cause massive issues and completly toast the timekeeping until reset.

In a unmodded 1132AU almost any strong source of RFI/EMI close to the module should cause issues with time keeping. Look at your install and locate the 1132CU away from RFI/EMI sources. For example 2-way RF modules, switching power supplies for laptops, almost anything that uses a CPU,,,,...

 

[az1324]

What do you mean by this:

"if the DS1307 ends up in a stuck state then it reports the last time sent to the 1307."

What is a stuck state and what causes it?

 

[az1324]

Would this apply to the Insteon PowerLinc 2414 as well?

The powerlinc 2414u I have uses the ST M41T81 Real Time Clock IC and the crystal is right next to it so no long leads happening there.

 

[Xymox]

I have no idea what happens in the 1307 to cause a "stuck state" and that term is wrong I suppose. Something happens, the RTC reverts to the last time sent to it and then starts ticking again. Hmm....

Well, the clock in the PLC is prob ticking and it just read a old value from the RTC.

I suppose the RTC could be stopped.

Its hard to investigate the real state of the RTC without reading the I2C bus and talking to it and seeing what it says is going on. Watching the bus over time...

I don't have a I2C emulator so I can't talk to the bus or read the bus. I really should get a I2C reader, I have needed it many times now.

Even with short leads to the Xtal, noise can still be induced. While I have not looked at that chip almost all RTC's clock with a 32Khz xtal. The DS32Khz will drop into any circuit where you need a hyper accurate 32Khz clock. Personally its hard for me to imagine NOT putting one of these chips in if you are good at DIY. Why not do it ? It also will elminate any noise induction as well.

Same procedure, pull the Xtal, find +5, Vbat and X1 and gnd. Your RTC then becomes hyper accurate.

 

[BLH]

Probably not completely related, but my 1132CU will loose mybe 30 to 40 minutes over night. If the USB cable is left connected and the computer is off. Disconnected so far it is much better than the newer 2414U 2414S and 2814U Insteon PLCs time wise.

 

[SmartLabsMike]

We are have received reports like Xymox's on problems with the original X10 PowerLinc Controller (C-1132CU) clock chip could be corrupted. To address this possibility, we developed a test in our incoming QC department to stress the controller and observe the results. Basically, the tests involve setting the time, stressing the unit with some heat and letting the unit run for three or four days. At the conclusion of the test, the time is read back and the operator looks for a corruption.

For the PowerLinc V2 series, as one already pointed out, it uses a different brand of real time clock. AFAIK, we have not seen similar issues in the newer PowerLinc V2 series. For those that are interested, it is not possible to swap RTC between units. The new RTC uses a different format and byte structure to set and read the clock.

Thanks for your continuing support.

 

[Xymox]

Nice honest answer !

Now if I had only known there was a V2 when I bought my unit a few months back

I have found something else weird today.

I was basing my accuracy test on watching my normal timed events. After doing the posts I set up a once a hour monitor by turning on a phantom device at the top of every hour.

Interesting....

It varies +- 2 seconds randomly it seems during the day. Its not just one turn-on thats off but you can see it vary over hours. First it goes slow a bit, then back to correct then fast... Its a very small and insignificant amount but I do not understand what is going on. Its clearly not using the power line 60hz which is far more accurate. I know from removing the Xtal that it cannot be referenced to the RTC as its 32Khz has stopped.

WHERE does it get its time reference between RTC updates ?

Does a power line vary in small amounts short term like this ? That I can monitor and I can't say I have ever looked at the power line freq deviation before.


SmartLabs Mike, It sure was not initially obvious to me to think about RFI/EMI. Its sure not what you think of right away. However with the E-core Xfmr right against the Xtal, RTC chip and battery the E-core will radiate power line RFI/EMI in the 32Khz range VERY VERY well into the Xtal, traces, battery and who knows what else.. Its a transmitter and very good antenna with the E-core prob having actual antenna gain at 10Khz-200Khz ! I think to be on the really safe side im gonna drop in a Toroid. Obviously this cannot be done in production due to the expense. A metal can around the RTC, no matter who makes it, I think makes sense. A penny invested in mfg costs will save thousands in call center support and replacement costs.

Testing wise, I would create a RFI/EMI source you can drop on the 120 line. This isn't that hard, the PLC does this with the X-10 carrier signal, same concept but do it at 20Vac with various noise content. Measure what gets picked up on the chip leads and traces. Of course you could just hook a oscillator to the input side of the Xfrmr and sweep looking at the traces for any of the signal that has radiated.

I don't do RFI/EMI testing. I know there must be procedures for this and test equipment made to interfere with things.

Everybody: While this issue might seem simple in hindsight, powerline noise being radiated by a E-Core transformer into a Xtal circuit and causing time issues, randomly, is not something you think about doing the design. The placement of the Xtal close to the transformer was, im sure, to couple it to the heat. This was a good idea. You don't normally test for powerline interference to the RTC !?!.. I stumbled across it almost by accident.

As to the question of time inaccuracy with the USB connected....
Before I did the mod, I think I saw this as well. I can tell you for sure its gone now tho. Of course adding a 6ft antenna ( the USB cable ) will pickup RFI and maybe could cause issues. Maybe with the laptop off, but still plugged in, the switching power supply might induce some noise that for complex reasons is not present when the laptop is on and this somehow ends up inside the module and causes odd clocking of the RTC ?

I have isolated my 1132CU. Its all by itself. Nothing around it.

I am seriously wide band coupled to the main line power. While this is awesome for inducing the PLC carrier into my power system it works both ways and also couples any power line noise right back into the module. I installed a nice quality outlet directly below my breaker panel. I carefully soldered 10Ga wire to the outlets thinking 200Khz.. One outlet is Phase 1 and the other is Phase 2. I placed a phase coupler ( exceptional caps and inductors ) between phases in the outlet box. I assumed a 200Khz signal when doing this work. I used 9" of 10Ga wire from the outlet box to the breaker in the panel. I chose a breaker which seemed to pass 200Khz better too. I plug in my 1132CU and my RF module to these outlets. By injecting the PLC carrier directly into the breaker panel and using a phase coupler connected directly to the back of the outlet that connects to the 1132CU I have seriously improved the PLC carrier signal strength at every point in the house but prob exposes me to more induced noise from the E-Core in the 1132CU.

So, anyone know exactly how the PLC keeps time without the RTC or 32Khz running ?

 

[az1324]

The microprocessor has its own clock source and a RTC implemented in software. The hardware RTC is meant to help with accuracy and keep time when the micro loses power. I think.

 

[Xymox]

The RTC chip generates a 32Khz clock that feeds the microcontroller. I saw that back when but ignored it as I was after the RTC..

This feed died when I pulled the Xtal and stopped the RTC.

I was sure the microcontroller would not function. It did, perfectly.

VERY weird. Prob something simple I just have not explored it at all.

I have noticed now that the RTC update seems to occur at midnight exactly. The time offset during the day does not vary much, however the update can be wrong and set the PLC time +-9 seconds which is then maintained all day until the next update at midnight where another offset of some seconds will occur. Most of the time so far it is very close, +-2 seconds.

So I think that somehow the PLC time is clocked from the power line. Somehow...

Anyway I may not explore this subject any more hardware wise as my time appears VERY accurate long term and I only see +-9 second offsets max on any day. Most times its less then 2 seconds off. Im quite sure my time will maintain this trend long term and I expect no more then a minute a year offset. Maybe quite a bit better.

I -might- explore giving the DS32Khz a hyper accurate power source as power devations will cause some change. Who knows maybe I might just do a battey only power source from Li for it and regulate that. That would be the best long term, but require changing a battery every 3 or 4 years. I could do that...

 

[az1324]

Really? Well if that 32kHz is going into the pic its not for the main clock source. Wouldn't want to run it that slow. Some pics have internal oscillators but in the 2414 I think I saw a ceramic resonator.

Its possible that the micro maintains a clock by counting zero crossings of the powerline since it has to monitor them anyway. I don't know enough about it to say whether it would be a reliable or preferred method.