Help tracking down a possible cause for robot disconnects

[adciv]

When you lose coms, is:
1) The laptop losing connection to the radio WIFI or?
2) Is the roborio no longer responding over Ethernet?
And what is required to get coms back? Power cycle, time, or what?

Related wiring issues we saw this year:
1) Battery Anderson Power Pole CRIMPS did not create a good connection. Wiggling the battery connector & battery wiring could induce loss of power.
2) Radio to RIO Ethernet cable would momentarily lose connection long enough to reset the Ethernet on the RIO resulting in lost coms. This was despite the retaining clip always being secured.

[scca229]

Couple of things I would be checking, particularly since you mentioned the roboRio lights going out and back. Did the radio fully reboot too (50 seconds or so of dead)?

Check the fuses in the PDB, yellow and red, and make sure they are fully pushed in. If you are pressing on them and your thumb/finger isn't hurting, you aren't pushing hard enough.

Also check the 6ga wiring completely from battery to PDB. There should be zero movement on any connection point (battery terminals, breaker terminals, PDB terminals). Was the correct crimper used on the lugs going into the PDB? Those lugs work amazingly well when crimped properly (they are used in Telecom POPs for just about everything) but terribly when not crimped properly (get the right tool...no vice and a screwdriver for them). My favorite one (personal experience from installing telecom gear for ~17 years) is the Panduit CT-1700.

Do a test on the breaker. Power up and then tap the red button a little. You aren't trying to press it to kill power, but "flick" it a little like you are trying to loosen a stuck needle in a gauge (you know how that always works in the movies). Some breakers are very sensitive and will temp open even though the squeeze switch is still closed and cause a full power cycle.

I ask if both roboRIO and radio rebooted to try to narrow down where to look for potential power problem. If both do, problem is upstream (Battery to PDB). If only one does, problem is local. roboRIO would be fuse & roboRIO power wire. Radio would be fuse, wire to VRM, radio power wire. Make sure all small gauge wires in Weidmullers are clean (no whiskers) and long enough stripped to FULLY engage in the spring (and that the spring works...one of ours failed on the roboRIO CAN terminals).

Many times trying to simulate the motions of real driving just does not cause the failures. It might be something that only happens with movement in a particular direction/velocity change. Maybe have someone look at the wiring that was not present when it was connected. A set of eyes that haven't seen it might be able to spot something that is invisible to someone that knows "there is nothing wrong with it".

Since you say the roboRIO lights go out, including the RSL, without flickering being present, I'm pretty sure you are going to find a "Layer 1" issue (to steal a networking term) and not a brownout issue. Comms normally stay up during brownouts even though movement goes to pot.

[ahartnet]

I think we were finally able to find and address the issue today.

The rio was definitely doing a full reset (all lights off, when power came back on, the rio would have the status light orange during it's POST, etc). I don't think the radio was resetting (typically) because I would only say two solid lights (momentarily) and I think when the radio reboots, you'll get three solid lights (momentarily).

Things that we did in order (all things that may have contributed):
1) Replaced the main switch (testing/flicking/shaking/tapping couldn't cause a failure, but was one of the few remaining components not replaced)
2) Replaced the 10 Amp fuse. It was one of the first things that I started wiggling this past weekend, but maybe the vibe loads were in a different direction than how I would wiggle it.
3) restrip a CAN cable that only had a couple of wire strands
4) Test run - still lose comm
5) Replace cable from battery to main switch
6) Add some mounts to reduce the vibration of the electronics board (it's mounted on some fairly thin lexan that could bounce significantly. It snapped near one of the mounting points which allowed more vibration on the side with the rio
7) Test run - drive without loss of comms for a while
8) Practicing - lost comms once, but we've decided that was likely due to a loss connection on that particular battery rather than anything on the robot anymore.

So I don't know if it's due to the battery cable or due to reducing the vibration that we're working better now. If it's just due to stiffening up the electronics board, I'm worried we're still just hiding an issue - but I suppose I don't know what kind of shock & vibe loads the rio is designed to with stand. I'm sure it's designed for more impact loads so maybe for some reason the bigger oscillations we were getting with a floppier than designed electronics board was causing an issue? Or maybe the strain relief we added wasn't enough. Either way, we're glad to have a running robot again.

We did get a lot of dropped packets. but that may be normal in our school setting, I haven't looked closely at that before (attached for reference).

[scca229]

Originally Posted by ahartnet

We did get a lot of dropped packets. but that may be normal in our school setting, I haven't looked closely at that before (attached for reference).

Your packet loss is at a very set interval, every 15 or 16 seconds start-to-start with a duration of 5 or 6 seconds each time, excepting the larger one which looks like a double event, but the one after starting right on schedule. I'm thinking environmental in the school, almost like something is updating via wifi and hogging your spectrum.

Good to hear that the robot is working better.

[Jonathan L.]

You said in your initial post that you replaced a power cable between the switch and PDP. However, the radio should be powered though the VRM. Is this the case?

[ahartnet]

Originally Posted by Jonathan L.

You said in your initial post that you replaced a power cable between the switch and PDP. However, the radio should be powered though the VRM. Is this the case?

Yes the radio is powered through the VRM

[frcguy]

Originally Posted by scca229

Your packet loss is at a very set interval, every 15 or 16 seconds start-to-start with a duration of 5 or 6 seconds each time, excepting the larger one which looks like a double event, but the one after starting right on schedule. I'm thinking environmental in the school, almost like something is updating via wifi and hogging your spectrum.

Good to hear that the robot is working better.

One thing I can think of is that many commercial access points used in places like schools have some sort of "threat management" technology, like this. To implement this so called "threat management", access points may contain a radio that can block, jam and/or attempt to force disconnects on any wifi network it is told to attack. Your school's access points may be programmed to do that to any network that they aren't told not to, which may include your robot. I suggest maybe talking to your network administrator at school to see if that is the case. Hope that helps!

[Alan Anderson]

Originally Posted by ahartnet

I think we were finally able to find and address the issue today. ...
4) Test run - still lose comm
5) Replace cable from battery to main switch
6) Add some mounts to reduce the vibration of the electronics board (it's mounted on some fairly thin lexan that could bounce significantly. It snapped near one of the mounting points which allowed more vibration on the side with the rio
7) Test run - drive without loss of comms for a while

Did you save the cable that you replaced? Someone might be able to look at the APP SB50 end of it and determine whether it has a defect that could cause your symptoms.

[ahartnet]

Originally Posted by Alan Anderson

Did you save the cable that you replaced? Someone might be able to look at the APP SB50 end of it and determine whether it has a defect that could cause your symptoms.

Yes, we have the cables that were replaced. the SB50 end is not the suspect end, but rather the PDP side. We had some ultra flexible 4 AWG wire that we had wanted to use due to some tight spacing, but we had to trim some of the strands out to fit it in the ring terminal.