Premise ADI SECU-16s suddenly fail to open

[ckindel]

After 12+ years of stable operation I'm suddenly experiencing problems with my digital IO subsystem in my house.
 
I have two separate digital I/O systems in the house. One is in my equipment room, the other in the upper garage.
 
Both are connected to Premise via Lantronix UDS-10 devices using RS-485. 
 
* In the Equipment room there is a single SECU16 on the RS-485 bus.
* In the Upper Garage there is a SECU16 plus two RLY8XA 8-port relay devices.
 
These are SEPARATE RS-485 buses. The only commonality is they are both on my home network via wired Ethernet and the UDS-10 devices.
 
In both cases I can still view the web interface on the UDS-10s so I know network connectivity to them is fine.
 
But in both cases, the "comms" LED on the SECU16 are on solid (although the Equipment room occasionally flickers). I don't think they used to be always on, but I don't really remember. The "comms" LED stays lit even if the UDS-10 is powered off.
 
In Premise, both AppliedDigital ADNet devices report PortStatus = "Port(OpenFailed)".
 
I have powered everything off and back on. I have checked the wiring. 
 
Occasionally in the past few days of troubleshooting I've had periods where one or both sub-systems worked...for a while. But I've not been able to figure out a pattern or any clues.
 
I found this post about failing RS-485 chips in the SECU16s. Maybe both of my chips failed at the same time?!!?
http://cocoontech.com/forums/page/articles/_/tutorials/home-automation-tutorials/how-to-repair-your-secu16-r125
 
Or, is it possible the problem is in the UDS-10 devices? They both failed at the same time?
 
I'm stumped. 
 
I'm going to order a few of the RS-485 chips just to try that, but I'm not optimistic.
 
Hopefully someone can offer some insight.
 
Thanks!

 

[BraveSirRobbin]

You might want to check the DC power on the board if possible.  Did anything change temperature wise lately?  Try running it with the cover off.

 

[ckindel]

You might want to check the DC power on the board if possible.  Did anything change temperature wise lately?  Try running it with the cover off.

 
The Equipment room has AC, the upper garage is not air conditioned. 
 
The 2 UDS-10s and the 2 SECU16s are each on completely independent power supplies.
 
After posting my message last night I cold-booted my Premise Server (actually, I cold-booted the machine that hosts the VM that Premise runs in) as well as my main Ethernet switch.
 
After doing this, I got both sub-systems to connect and the SECU16s were working.
 
This morning, the Equipment room SECU16 is "Port(OpenFailed)" and the Upper Garage unit is still working.
 
More stumped than before...

 

[BraveSirRobbin]

You might want to take the Lantronix units out of the equation and wire directly to the RS-485 bus from the Ocelot.  Don't know how much of a pain that would be just to troubleshoot though.
 
The other thing to think of is maybe also post on Adi's forums and see if Guy Lavoie can help.

 

[ckindel]

You might want to take the Lantronix units out of the equation and wire directly to the RS-485 bus from the Ocelot.  Don't know how much of a pain that would be just to troubleshoot though.
 
The other thing to think of is maybe also post on Adi's forums and see if Guy Lavoie can help.

 
I don't use an Ocelot. The only RS-485 bus controller I have are the UDS-10s. I wonder if there are any other diagnostic tools availalble?
 
Thanks for the pointer to the Adi forums.

 

[chucklyons]

Charlie...don't know much about the SECU16s (actually, I know NOTHING about those), but please elaborate more on the VM side of things. Platform and whys would be great. I was thinking of doing that myself, but if there are some lessons learned, that would be a great help...

 

[ckindel]

Well, I replaced the RS-485 controllers on the SECU16 and that didn't help. 
 
I'm grasping at straws, but am going to try to replace the UDS-10s next. I wish I had another SECU16 lying around; I used to have a SECU16-IR but I can't find it. :-(

 

[georgejm]

Is your network flooded too?  I had a similar problem with my Barix Exstreamers and GC-100's.  Similar in the sense that I got communication errors.  Turned out to be Alureon and Z-Bot on one of my kid's computers.  It was flooding my network and Premise couldn't talk to network devices.   After I cleaned the machine, everything went back to normal.

 

[ckindel]

I don't think my network is flooded. 
 
But I am convinced it is not related to the SECU16s/RS-485 bus.  I can reproduce the problem with the SECU16s powered off.
 
In other words, the 'Port(OpenFailed)' is due to the failure for the driver to open the socket (port 10001 on the UDS-10s).
 
Using a telnet terminal trying to connect to port 10001 I can get "Winsock Error (274d): Connection Refused".
 
If I cycle power on the UDS-10, I can (sometimes) get it to connect.

 

[ckindel]

I think I just figured it out!
 
I started disconnecting things from my main Ethernet switch to see if maybe some other device in the house was causing interference.
 
I noticed that a port on the switch was live that shouldn't have been...my old physical Premise server.
 
Then I noticed that server was powered on. It should not have been. I retired it 2+ years ago (but left it plugged in).
 
Here's the theory: Sometime recently that machine got powered on. I'm not sure how that happened because the power switch is behind a cover.
 
In any case when it's on there are two instances of Premise trying to connect to the UDS-10s! It was working sometimes, because sometimes my new server (in the VM) was winning.
 
I've now turned it off and have done a little testing and I think we are back to normal. 
 
Needless to say, I've unplugged that old server.