OmniPro II "sudden" hard failure - SYSTEM RESETS and generally going berserk

[jcd]

Looks like your Ethernet port on the OP2 panel is "constipated". Have you tried the following
A - disconnect the Ethernet port, reboot and only utilize your serial connections to the panel. Your serial devices will not hiccup and your time will stay stable with no Ethernet connectivity (if it is an Ethernet issue).

UPDATE:
(busy week at work and I hadn't gotten updates to the thread but thank you again for everyone who's contributed time/attention/ideas!)

Okay, so the short version free of hysteria (and the past couple weeks have been full of hysteria!) is that I started with @pete_c comment above and just as suddenly as everything fell apart, things slowly started working again!
- I have been able to connect via Ethernet AND maintain a persistent connection for days.
- I have been able to download the entire .pca file at once and the connection has maintained.
- The OPII serial ports didn't work for a few days: #1 (VRCOP Z-wave) worked, but #2 (HiFi2) and #3 (URTSII) didn't work at all for a few days and then suddenly started to work.
- I've noted that my OMNISTAT2 has "reset" a few times - the backlight setting keeps getting reset to the default but otherwise everything seems okay.
- I've lost date/time settings on the keypad consoles a couple times.
- I'm still getting "clusters" of SYSTEM RESET errors in the PCA log file. 6-8 at a time all together but at random times and I cannot see any pattern.

So there still seem to be some lurking problems, but as a simple fact my system is otherwise up-and-running and everything seems functional. I have reconnected all of my iPads running Myro:Home and each has maintained a connection, shown proper statuses/refresh, all functions work - including (my most important feature) CCTV camera "pops" triggering on each screen where motion is detected.

I cannot by any means say I feel comforted that whatever happened is "solved" because aside from the intentional reboot sans Ethernet cable, nothing else is changed. And if all of this was caused by my OPII Ethernet port being "constipated"... well, I have no words for that.

My plan is to keep running "as-is" for another period of time (week or two?) and report back on what I'm noting, as I did above. I'm so grateful to be back from the Stone Age, but quite traumatized by the experience (#FirstWorldProblems, right?) because I don't feel settled on what happened and why. I don't ever want to go through this again!

Trauma aside, after my "settling period" above, I'll go back through the very helpful notes/recommendations and get to the very strong advice to "wall" the OPII behind a "lesser/slower" segment of the LAN. Again, @pete_c thank you for the history links. As of now I know how to throttle-down the port to a 10Mbps half-duplex speed, and I started to tinker with the "storm control" bits lightly but that's about where my networking knowledge ends. I'll invest the time to understand the steps shared before I come back with questions on that, and hopeful that might solve the issue more permanently. I still can't believe a noisy Ethernet port could bring down the whole shebang so hard!

So so grateful for how generous you guys are with your time and experience and expertise, THANK YOU! I'll update again soon...

 

[pete_c]

I tried with an old 10Mbs hub and a managed L2 switch and it did not work. It did work for some users.



What only worked for me was routing. The clock goes off time because of the serial bus. The serial bus gets messed up because of the Ethernet chip / port. The thermostat, X10 TW523, VRCOP and Zigbee ZIM are all connected to the serial bus. The keypad consoles and old Omnitouch 5.7's. The network connected devices just start to drop off until you are unable to get to the panel via IP.

Baby steps and do not panic.

Here is a snapshot of the traffic that goes to the Ethernet Port on the OP2 via the router.


Will connect the OP2 ethernet port to a managed L2/L3 switch port and post a graph of all of the traffic on my LAN going to the Ethernet port non routed.

 

[sjht]

One more thing: I did change the port settings to 10Mbps half-duplex, and also tried to "tuck" the HAI and HiFi2 onto a small 100Mpbs mini-switch to see if I could "throttle down" the traffic they were seeing - but neither helped and that arrived about at the end of my network knowledge.

Where did you make this setting? I have a UniFi USW 48 Pro switch and made the setting there. Setup as 10 Mbps HDX. I have a large network and have never had an issue.

 

[jcd]

Where did you make this setting? I have a UniFi USW 48 Pro switch and made the setting there. Setup as 10 Mbps HDX. I have a large network and have never had an issue.

I did indeed make it exactly here. Then as I was moving devices around on my switches I ended-up moving the OPII and the HIFI2 ethernet to the LAN ports on my Peplink router, where I also made the change there.

Fingers crossed things seem to have stabilized. As noted I'm going to let this run for a bit more - maybe another week - and then assess next steps. I expect that I'll start down the micro router path at that point to really set the OPII in a place where I can filter what traffic it "sees". I'm starting to map out "what needs to see what" so when I plan this next step, I'll have a view and questions to present to ensure I'm doing things right!

Thanks!

 

[jcd]

Soooo....

It's been a while and I promised I'd update my results from all the super-helpful guidance this group has shared. Here's the update: had a lightning strike on our home and the multiple layers of protection (1: at the panel, 2: in the racks, 3: in front of most devices) did an exceptional job protecting our AC electrical system...

Not so much with the low-voltage stuff. The surge came in through the exterior LV cabling and anything connected to a Ethernet cable was obliterated. In my house that means all HDTVs, A/V (HiFi2 + HT receivers, dvd players, etc.), OmniPro II and all connected expansions, NAS, printers, PCs, PoE tablets, CCTV, network switches, etc. Our extensive landscape lighting too. All gone.

Literally everything low voltage is being rebuilt from scratch, so the choice was forced and I'm now migrating to an Elk M1/HomeSeer world.

My new network architecture now includes fiber between switches and anywhere else I can insert a non-conductive link as a break-stop. All exterior LAN gear (IP cameras and WAPs) are on a fiber-isolated switch. I'm not sure if it's worth it - or even effective - to attempt to protect non-Ethernet low-voltage lines (PIRs and other security sensors) or if I'll just take my lumps if Nature spites me again. I know it's pretty hard to argue with 30,000 amps of lightning that has traveled over miles of free air to think I'll stop it with a millimeter air gap or some sacrificial SPD.

Anyway, my OmniPro II problems were "solved"

 

[electron]

I've experienced a very similar event years ago, took out my ELK M1 & accessories, PCs, cameras, switches. Ignoring the replacement cost, I think the most frustrating part was discovering components which partially failed, so it took me several weeks to figure out the total damage.

 

[jcd]

I've experienced a very similar event years ago, took out my ELK M1 & accessories, PCs, cameras, switches. Ignoring the replacement cost, I think the most frustrating part was discovering components which partially failed, so it took me several weeks to figure out the total damage.

Yep, I definitely feel for you my friend! I've already invested a couple HUNDRED hours in all of the diagnosis, assessment, and what little bit I've been able to rebuild to-date. Hundreds of hours more to come as I redesign and "re-learn" how to replicate my Omni knowledge into Elk. And like you said, there is this lingering undercurrent (no pun intended) of things I fear I'll "find out" later.

I am grateful that no one was hurt and the house didn't burn down. As annoying as it is, this is just "stuff" and living in a Flintstones house with no automation, etc. is definitely a First World problem!

 

[electron]

Totally agree and a great view to have!