UPB problems...puzzling

[cmarcus]

Fellow Cocooners,
 
I'm having a bit of a UPB headache and I've exhausted all the debug techniques I know to do. I have arrived at a conclusion, but the probability that my conclusion is right seems very low. Maybe you all can come up with better theories.
 
Here's the setup:
I have two 200A panels both with SAI phase couplers in the panels. I have five SAI switches; three US2-40s and two US11-40s. Most control their own load except one of the US2-40s. I have an Omnipro II panel with an HAI UPB PIM. Also, a SAI USB UMC that I use at my computer for programming the switches and experimenting with different setups.
 
I got everything setup how I wanted about 4 months ago. The switches all worked how I wanted and the Omnipro is programmed to turn off and on my exterior lights at sunrise and sunset. This worked beautifully barring a couple of mornings where the lights weren't turned off.
 
Symptom 1 - About a week ago, the exterior lights were not turning off or on at the programmed time. I tried manually turning them off and on using PCAccess and that didn't work. I thought it was the PIM, but I was able to turn on/off my interior UPB switches (this was a brief test one morning). The next morning, same problem so I opened UpStart and tried turning the exterior lights off from there. It worked, but turning them on took three tries. Turning them off worked each time. Thanksgiving rolls around so I table this until today. 
 
Symptom 2 - I double check control via PCAccess and I can't control any switches with it. It has to be the PIM, I'm thinking. So I go back to UpStart (which uses the SAI USB PIM) and try controlling switches there. I have marginal success. Mostly, nothing works. I open the Signal & Noise Meter in Upstart and I see I have medium noise and excellent signal. Hmmm? I then decide to create a brand new network file by using the Network Discovery feature in Upstart. This fails. I see my five switches show up as new, but they come and go. When I click the box to have each device added to the network design, every device fails with red in the upper part of the box and black in the lower part. So there's a "Problem" and "Comm fail". I then open my existing network file and attempt to control switches from there. No luck. I then decide to grab my HAI PIM and put it on my computer to see if it will work. It fails miserably, during network discovery it doesn't see any of my devices. I then reset both PIMs to factory defaults and try this all over again. Same results.
 
Oh, and I should mention here that the switches all work fine even the ones that only transmit links. The load controlling switches are still working so the communication among switches seems fine. I loaded UpStart on my laptop and moved the PIM to another room (different circuit) and still no luck.
 
Symptom 3 - During my testing, I have the Log Viewer opened so I can see the UPB messages that are being generated by UpStart and by the switches. I decide to put the two PIMs in different rooms and see if they can receive messages from the other. In my office, I have the HAI PIM connected to a PC that has my network file on it so I open it and start turning on switches. I do NOT see those message on the laptop running the SAI PIM. I then used the laptop to perform a network discovery which saw the switches, but ultimately failed the same as symptom 2. During the discovery process, not a single message was received by the HAI PIM connected to my office PC.
 
I forgot to mention that during this testing, I also put three different switches in setup mode (at different times) and UpStart was never able to detect that.
 
I don't see how this is possible, but I think that both PIMs are bad. I think they are unable to transmit even though they seem to receive fine. I just ran a Network Comm test using Upstart and it failed on every switch due to no response. The probability of two PIMs having the transmitters fail with a week or two of each other seems very, very low. That's a pretty specific failure on two devices of a different age and manufacturer. As far as I know, I haven't had any lighting strikes or major surges. The HAI pim is behind a surge protected outlet (and this was working). The SAI PIM is only plugged in when I need so it has been unplugged for weeks.
 
I'm at a loss. Is there some other test I can run or should I just buy a new PIM and see how that works out?
 
Thanks,
cmarcus

 

[Desert_AIP]

Have you changed any lighting loads? To CFL or LED? Even in fixtures not controlled by UPB.
Have you added any other equipment?
Do you have an electric water heater?
What kind of HVAC system do you have (I am assuming it is now running since it cold, but was not running a few months ago when everything worked).
Do you share a transformer with neighbors? They may have added equipment.

What do your noise levels look like?
Can you perform a network verify?
Does this happen at a particular time of day?

You may have to start turning off breakers sequentially until your comms are good to isolate a troubling piece of equipment.

 

[cmarcus]

Desert_AIP - Thanks for responding and asking good questions.
 
- I have a lot of LED lights, but they were installed prior to the UPB switches. No new bulbs or fixtures in the past 6 months. No issues thus far.
- I have added two things; an Elk 12v power supply with battery backup. I'll unplug that tomorrow. The second device I installed recently which is very suspect is a Brultech GreenEye Monitor. I have current transformers on almost all my branch circuits. These "shouldn't" cause a problem since they are passive devices that measure the magnetic field being generated from AC and convert those into very low level signals that are sampled by the GEM. The GEM itself has two transformers - a typical ac-to-5vdc and a step down transformer which steps 120vac down to 12vac as a reference voltage. Again, not something that should interfere, but definitely worth disconnecting when I get a chance.
- I do have an electric water heater, but it was not on when I ran the test.
- I have a geothermal HVAC but it also was not on when I ran the test.
- There's one neighbor who I suspect has automation and we are on the same transformer.
 
- Noise levels are fine which I find weird because my answers above even lead me to believe I have a noise issue. I just checked it again and the avg noise at the PIM is 2. I ran a network verify and it fails at the third step, "Test noise level at device". Upstart was unable to establish communications with this device.
 
I will definitely start killing breakers one by one tomorrow, but I still don't understand why the switches themselves have no issues. Two of them that are linked are even on different branch circuits and they work fine. Interestingly, all my switches and the PIM in my office are all on branch circuits in one of my panels and it's the panel with very few "noisy" loads. I know I can still get noise from the other panel circuits, but I would expect them to to be attenuated quite a bit since they are on a separate 200A main breaker.
 
Thanks for making me think about these things. I got into a bit of a bubble trying to figure this one out. More testing tomorrow...I need a night cap!

 

[Desert_AIP]

I had a problem similar to yours.
Everything worked fine for months, then links started getting missed.
I had. A single coupler in my panel.
The noise levels were negligible throughout my install (60+ units).
I couldn't get a good verify of all units. Plugging the PIM into different outlets have me different verify results.

It wasn't a noise issue but a signal attenuation issue.

Then I noticed the missed links were occurring sometimes and not others and then I,narrowed it down to right after we showered and the water heater turned on. Coupling both sides of the electrical service across a very small value resistor.
I had recently rewired the water heater to the correct 220V (it was improperly wired as 110V and the home inspector and I missed it before we moved in - duh) which is why it hadn't interfered before.

I added two more couplers and it boosted the signal enough to restore reliability.

So you may have a similar attenuation issue with something on the line.

 

[Desert_AIP]

Is the Omni in UPB or HLC mode?

 

[tmbrown97]

I was doing some troubleshooting a couple years ago - I've never seen noise on my lines, but was troubleshooting lost links...  one of the things I did was rearrange some breakers in my panel.  When I did, I actually managed to create a ton of noise and problems.  From what I can tell, when I pulled the breakers and reseated them something was a hair loose causing all new problems.  Once I went back and double checked and tightened everything, the noise completely disappeared and things were good.
 
I wonder if the process of messing around with the panel to install CT sensors over the wires could've loosened something - were breakers pulled out to make the wiring process easier?  Can you go through and make sure anything you touched is secure and perfectly aligned?

 

[pete_c]

Here too I had a similiar issue a couple of years back.  It was intermittant a bit and it wasn't noise so much as poor signals.   I did as above; reorganizing and labeling my circuits / breakers over the years.  I did initially have a bad HAI serial UPB PIM.  Once replaced my UPB was normal until the intermittant problem.  There I added a repeater which solved my issues.  I also today have a floating HAI serial PIM which I use in various parts of the home to check on the signals. 

 

[cmarcus]

Work2Play - The CT installation was done over a month ago well ahead of hooking up the GEM. I didn't notice any issue between then and the GEM install, but I won't rule that out. I'll double check all the breakers and hot wires to make sure all the connections are good.
 
Thanks for the ideas gents. I went ahead and ordered another serial PIM since I'm sure my other spare will eventually be put to dedicated use (if it's still good). Hopefully, by the end of this weekend I'll have a sense of what the problem is.

 

[kwilcox]

So, given:
no problems still with switches that activate links/switches that receive them
switches communicate between peers located on different circuits/legs of the house
Different devices using the 2 PIMs all can't transmit.

If everything above is true, there's only one solution: 2 bad PIMs.

However I would question the empirical accuracy of "switches communicating with other switches work fine". Typically these links activate other devices on the same branch circuit.

 

[tmbrown97]

I think I read this - but of course you tried the PIM in different locations around the house, right?

 

[cmarcus]

Okay - I've ruled out the two bad PIMs theory. I hadn't realized it at the time, but my SAI PIM when I took it from one room to another (which also meant connecting it to a laptop instead of the desktop), somehow went into message mode. Well, I believe this because when I went back to run the same test, Upstart would connect to the PIM, but I would get a "interface is out of date" message when trying to talk to a switch. I reset it and it's back in pulse mode and working. I reran the same test with my laptop in the living room on a circuit that has one switch. This worked. It not only detected that switch, but the other 4 as well. Everything seemed fine. Took it back to the office circuit and FAIL.
 
I then went into the panel and started inspecting wires and how well the breakers were seated. Everything was good. There was one questionable neutral wire connected to an AFCI breaker, but I fixed that and it made no difference. I did, however, start correlating the phases that each switch were on. The switches are on the foyer circuit and the living room circuit. Both of these are on the same phase, in the same panel. So the signals seem to fine on this phase, in this panel. The office circuit is in the same panel, but on the opposite phase. So this leads me to believe the phase coupler is not doing it's job. The noise measurements seem fine and oddly the PIM in the office seems to receive signals fine, it just can't seem to successfully transmit.
 
My Omnipro PIM is on the basement circuit in the other panel. It is on the same phase as the office which is not the phase the switches are on. This panel also has a phase coupler. I just tried sending a UPB link from PCAccess and I did not receive that message on the PIM in my office. So either 1) the HAI PIM is fubar'ed on the transmit side or 2) the signal between my two panels is extremely attenuated even on the same phase. I would tend to lean towards option 2), but I'm on my second HAI PIM so I have not confidence in these things.
 
My new serial SAI PIM has not arrived yet so I can't test that yet. Thus far, I am suspecting an attenuation issue. I just wish I could gain more confidence in that before I shell out for a repeater. Are those strong enough so that I only need to install it in one panel?
 
One thing I found during my testing, was that two switches that always work (one controlling a load, the other setup as a slave) just stopped working. The load controlling switch always worked, but the slave stopped. I scratched my head for a bit and then my wife said dinner was ready. Just realizing we were having leftovers, I hit the slave switch again and it worked. The damned microwave was the problem. This isn't the core problem I'm having, but definitely something I hope the repeater will help alleviate as well.

 

[cmarcus]

Well, I just had one of those ah-ha moments while reading through some old UPB posts. I never go in our front basement because we don't store anything there except for a little project I have that is out of sight, out of mind. About 5 months ago, I created a dehumidifier kiln to dry out two huge blocks of black walnut. Someone else's post about a suspect fan made me think about what fans I might have. I remembered the dehumidifier. I went downstairs and turned it off and VIOLA!!! Upstart in my office sees everything just fine. It appears the the dehumidifier is on the fritz. There's a little bit of water on the floor which is not normal so it may have started malfunctioning. What a hell of coincidence that this started after my GEM install. I spent a lot of time questioning that. Oh, and I should mention the dehumid is on the same circuit as the Omnipro so that whole phase was probably drowning in noise. I still don't know why the noise didn't register higher in Upstart.
 
I really do appreciate all the help. Not a total waste of time though - I found a couple of setup issues that needed to be fixed. No big ones, but they would have killed a couple of hours if I were just debugging those.
 
Humbly appreciative,
cmarcus

 

[Desert_AIP]

Glad you found it!

 
Probably not noise but signal attenuation.
The fan was sucking up the signals.
Much harder to isolate.