Rebooting Several Boxes

[MobileMe]

I ran into a strange event today involving the webcontrol boards rebooting every few seconds.  I have three of them on the same network.  They have been working just fine for a while now. 
 
I put a NVR on the same network with four ip cameras.  Once I did this, all three of the webcontrol boards started rebooting every few seconds.  After unplugging the NVR and manually power cycling the board, everything went back to normal. 
 
Has anyone had any problems like this with any other devices?

 

[CAI_Support]

What if you only unplug NVR, not camera?
Something is injecting noise on power line when plugged in.  If you isolate which one can cause problem, that would help others.
What if you place those noisy devices on a separate filtered power strip, would that help?

 

[MobileMe]

It is very apparent that it is the NVR that is causing the boards to freak out.  Each board is located in a separate building with there own power strip.
 
I was just wondering if anyone else had ever experienced anything like this before.  Once I removed the NVR from the network, the boards are fine.  If I keep the NVR on the same network, the boards will just keep rebooting even if I power cycle them manually. 
 
The only theory I can come up with is that the NVR is sending out some kind of network packet that the boards do not like. 

 

[CAI_Support]

If the network switch is a real switch, it will not route data packets to WC's socket, if it does not just broadcast to every IP address all the time.  Modern network switch sends data packets based on MAC address destination address in the packet.  One way to find out is to setup on your computer a Wireshark software and capture if computer always receive datapackets from NVR, even if computer does not do anything with it.
 
WC can operate by itself without network.  But I guess without connecting to network, it is hard to tell if board restarted or not.

 

[BTCAD]

Network cable loop issue?

 

[Tschmidt]

Network cable loop issue?

Interesting problem. As CAI_Support mentioned normally an Ethernet Switch will only forward the frame to the port the MAC address is attached. Do you know if the NVR is doing a lot of broadcast packets? Otherwise the only reason the WC should see NVR traffic is if the switch does not know which port the specific MAC address is connected in that case it floods the frame to all port to learn which one responds. Since the WC only has a 10Mbps interface there could be subtle FW issue where it crashes under heavy load.
 
To BTCAD's suggestion it might be network loop issue, if that were the case ought to cause more trouble then just the WC. This sounds like a large commercial network so I assume they are running spanning tree on the switches.
 
At this point run Wireshark and see what network traffic looks like.

 

[MobileMe]

The NVR does have a "find camera" feature built into it which does send out broadcast packets, but no where near the amount it would take to bring down the WC.  I have about 150 or so devices on this network.  The only devices that were affected by the NVR, were the WC boards. 
 
I've never seen the boards behave this way either.  I would be watching them through the browser with polling on and watch them reboot.  The temperature would read "0.0F" and the time would go back to 2011.  It would do this every 5 to 10 seconds.  It was very consistent.  All three boards would do the same thing.  As soon as I unplugged the NVR, the boards would stop rebooting.  Plug the NVR back in and the boards would start rebooting again.  Like I said, it was very consistent. 
 
Other than the WC boards rebooting, the rest of the network was running fine.  No latency or lost packets on the network at all. 

 

[Efried]

Interesting problem. As CAI_Support mentioned normally an Ethernet Switch will only forward the frame to the port the MAC address is attached. Do you know if the NVR is doing a lot of broadcast packets? Otherwise the only reason the WC should see NVR traffic is if the switch does not know which port the specific MAC address is connected in that case it floods the frame to all port to learn which one responds. Since the WC only has a 10Mbps interface there could be subtle FW issue where it crashes under heavy load.
 
To BTCAD's suggestion it might be network loop issue, if that were the case ought to cause more trouble then just the WC. This sounds like a large commercial network so I assume they are running spanning tree on the switches.
 
At this point run Wireshark and see what network traffic looks like.

 
wireshark reveals that there are frequent "who has IP..." broadcasts, but that might not be what you are referring to.
BTW I have a newly firmware updated board connected to a WLAN access point which seems to be more unstable when connecting. This might be a point for integrating WiFi into the WC.

 

[CAI_Support]

MobileMe,
 
What kind of NVR do you have?  If that is so consistent, we like to see if we can get hold a NVR to run some tests.
At the mean time, does your switch allow blocking certain packets on per port bassis?  It probably sends some large data packets that so large, WC board ran out RAM still could not finish receiving the packet.
 
If we can find out from your captured data what it was sending, we could try to implement something to block or drop that kind of requests. If you have the wireshark capture, please send it directly to us, or let us know where to download it.

 

[CAI_Support]

Hi MobileMe,
 
We have opened a case with Microchip support engineers about this possible network stack overflow problem.
We do need to submit the data for them to analyze to see where is the problem.  If you could please help us to capture the data, so that we can get this problem resolved, we would really appreciate your help.
 
Thanks.

 

[CAI_Support]

MobileMe,
 
We do need to obtain captured data to analyze, so that we and Microchip engineers can work out a solution for it.  We do want to resolve this for you ASAP.  However, without captured network data, we can not do anything.  Thanks for your attention.

 

[CAI_Support]

Hi MobileMe,
 
Maybe you are busy not see this and our direct message through this board.  However, we do need to have captured data, or at least the model number of the NVR to test the problem reported.  We had opened case with our supplier, Microchip, for this reported problem.  Microchip requires us to provide captured data, so that they can continue work on this case.  Without supply that captured data, Microchip thinks this problem is not software related.  We believe you did see the problem, however, without captured data, we are in a bad spot of rock and hard places.