Help tracking down a possible cause for robot disconnects

[ahartnet]

Hello,

This past weekend, we took our robot to a demo and found that we could not maintain a radio connection for very long. We did what we could there, but today we went back to the school to try to track down the issue, and I can not find anything that seems to reliably cause the issue. A run down of observations and attempted resolutions:

Observations:
1) Robot loses comm.
2) Can not recreate by wiggling wires, fuses, or components.
3) Only loses comm when driving. Doesn't even have to be up rockwall or moat. Sometimes it loses comm driving up the batter or over the ramp for the low bar. Sometimes it drives fine over the rockwall or moat and does not lose come.
4) If robot is enabled and we pick up and drop the robot from any "worst case" angles, robot still maintains comm
5) Robot does not lose comm if we manually push the robot over the moat/rockwall/back and forth on the batter
6) Robot does not lose comm from driving back and forth aggressively/full speed. Seems to have to be at an angle
7) Comm loss does not appear to come solely from a flakey radio. I watch almost all the lights on the rio turn off then on. RSL turns entirely off/dark.
8) No stuttering of robot motors - I do not believe that this is from brownouts (which rio logs support)
9) Robot does not need to encounter an aggressive hit for loss of comm. It's happened going up the ramp to the low bar at a slow speed.

Attempts to resolve:
1) trying to recreate loss of comm/rio power cycles by wiggling wires, shaking components, etc
2) Try to recreate loss of comm by lifting the end that goes over the moat/rockwall and dropping. Does not recreate loss of comm
3) Try to recreate loss of comm by shaking robot aggressively, no luck.
4) Replaced PDP (we tried using ferrules and I think the weidmuller terminals lost some of their effectiveness. We have since changed back to not using any ferrules
5) Replaced RIO
6) Replaced a potentially bad power cable from the switch to PDP
7) Replaced a potentially too short (insufficient strain relief) 18 awg wire from the PDP to the Rio.
8) Added ramprate for drive code. THe fact that this only seems to happen while being driven points me towards a current/programming issue
9) Replaced code with a very simple project that only drives with a ramprate
10) enable robot, wiggle one set of wires, disable robot, check rio logs, repeat with a different set of wires. I was hoping that maybe the rio logs would catch some quick disconnects that are quick enough that the rio maintains connection.
11) Wiggling wires above includes wiggling the 10 amp fuse on the PDP for the rio.

I've attached rio logs (I hope). I haven't encountered a problem like since the rio logs have come out, but I would suspect that the fact that I lose data about the voltages/currents/etc prior to the actual radio disconnect is suspicious. It made me wonder if the can bus cable could be bad. I'm fairly certain there has to be something in the power path to the rio that is the cause because it looks like the rio resets when we lose comm. Do the rio logs help point anyone to an issue?

[simpsonboy77]

Each time you lose comms, PDP ports 12-15 draw a decent amount of current. I assume these are your 4 cim motors. You have a few brownouts, but none of them seem related. I'd open up the rio and check for any metal shavings. Since you replaced the Rio, this is unlikely to be the issue.

It could be grounding or static issue. With the robot OFF measure resistance from the frame to the PDP ground (any ground will do, they are all connected). Then check the frame to a ground on the rio. I usually measure it to the metal shield on the USB port. You should ideally get open circuit, but anything under 1M should be investigated in my opinion.

The last thing I could think of is maybe it's your battery. Make sure the terminals are tightened down hard so you can't even rotate the terminal around the bolt. You can toss a star washer between the wire terminal and the battery terminal to cut through any surface oxidation.

[tr6scott]

Suggestion, download a wifi analyzer for your phone.
I have been using one called wifi analyzer open source, and it works nice.

Review the wifi environment and see what channels are open, and set radio to use one of those channels.

I would also suggest to use the old dlink radio for demos, as the setup pages are less confusing as they have less options.

We have had better success using the 5GHz band. Which is doesn't seem intuitive, as the 5GHz has less range, but we have found that there is typically less existing access points using 5Ghz band.

This is problematic if you are using classmates as drivers stations, as it does not have a 5GHz radio, so we have purchased some 5Ghz USB adapters from newegg, specific model is no longer sold, but this:

http://www.newegg.com/Product/Produc...-037-_-Product

seems similar.

Our school is the worst 2.4 GHz environment we typically run into, so we are well versed in making these adjustments.

[ahartnet]

Originally Posted by tr6scott

Suggestion, download a wifi analyzer for your phone.
I have been using one called wifi analyzer open source, and it works nice.

Unfortunately since I watch the rio reset, I don't believe it's because the 2.4 Ghz is overcrowded. It's worth checking however.

Originally Posted by simpsonboy77

Each time you lose comms, PDP ports 12-15 draw a decent amount of current. I assume these are your 4 cim motors. You have a few brownouts, but none of them seem related. I'd open up the rio and check for any metal shavings. Since you replaced the Rio, this is unlikely to be the issue.

It could be grounding or static issue. With the robot OFF measure resistance from the frame to the PDP ground (any ground will do, they are all connected). Then check the frame to a ground on the rio. I usually measure it to the metal shield on the USB port. You should ideally get open circuit, but anything under 1M should be investigated in my opinion.

The grounding/static issue is a good thought that I've neglected to check. All of our electronics are mounted to lexan, but it's worth checking. Maybe we have a stray wire somewhere that wasn't cleaned up/removed. It also reminds me that our lexan electronics board is awfully wobbly. I wonder what the rio can accept vibration load wise? Additionally, you're correct - we have 4 CIMs on PDP12-15.

Originally Posted by simpsonboy77

The last thing I could think of is maybe it's your battery. Make sure the terminals are tightened down hard so you can't even rotate the terminal around the bolt. You can toss a star washer between the wire terminal and the battery terminal to cut through any surface oxidation.

Not a bad thought, but we've tried several different batteries. However, I don't know how closely I've checked the anderson connector that the battery connects into. We'll have to check that out tomorrow.

[ctt956]

It sounds to me like the motors are drawing too much current when climbing. What type of surface were you driving on at the demo? Do you have any practice defense you could try driving on? Maybe make a ramp the same size as the defense and batter ramps(if you don't already have them) and drive on/over them. Put a bigger platform at the top if necessary so you can park on it, then drive down. Try different ramps that are longer, shorter, steeper, not as steep, different surface, etc. and see how that affects it. If possible, watch the wheels when you test it to see if any slow down. Assuming you have two CIMs on each side(four total), try it with just one(two total). Running one on each side and switching each and/or testing the CIMs outside of the robot couldn't hurt. What gear ratio are you using? Changing it if possible might help.

[adciv]

When you lose coms, is:
1) The laptop losing connection to the radio WIFI or?
2) Is the roborio no longer responding over Ethernet?
And what is required to get coms back? Power cycle, time, or what?

Related wiring issues we saw this year:
1) Battery Anderson Power Pole CRIMPS did not create a good connection. Wiggling the battery connector & battery wiring could induce loss of power.
2) Radio to RIO Ethernet cable would momentarily lose connection long enough to reset the Ethernet on the RIO resulting in lost coms. This was despite the retaining clip always being secured.

[scca229]

Couple of things I would be checking, particularly since you mentioned the roboRio lights going out and back. Did the radio fully reboot too (50 seconds or so of dead)?

Check the fuses in the PDB, yellow and red, and make sure they are fully pushed in. If you are pressing on them and your thumb/finger isn't hurting, you aren't pushing hard enough.

Also check the 6ga wiring completely from battery to PDB. There should be zero movement on any connection point (battery terminals, breaker terminals, PDB terminals). Was the correct crimper used on the lugs going into the PDB? Those lugs work amazingly well when crimped properly (they are used in Telecom POPs for just about everything) but terribly when not crimped properly (get the right tool...no vice and a screwdriver for them). My favorite one (personal experience from installing telecom gear for ~17 years) is the Panduit CT-1700.

Do a test on the breaker. Power up and then tap the red button a little. You aren't trying to press it to kill power, but "flick" it a little like you are trying to loosen a stuck needle in a gauge (you know how that always works in the movies). Some breakers are very sensitive and will temp open even though the squeeze switch is still closed and cause a full power cycle.

I ask if both roboRIO and radio rebooted to try to narrow down where to look for potential power problem. If both do, problem is upstream (Battery to PDB). If only one does, problem is local. roboRIO would be fuse & roboRIO power wire. Radio would be fuse, wire to VRM, radio power wire. Make sure all small gauge wires in Weidmullers are clean (no whiskers) and long enough stripped to FULLY engage in the spring (and that the spring works...one of ours failed on the roboRIO CAN terminals).

Many times trying to simulate the motions of real driving just does not cause the failures. It might be something that only happens with movement in a particular direction/velocity change. Maybe have someone look at the wiring that was not present when it was connected. A set of eyes that haven't seen it might be able to spot something that is invisible to someone that knows "there is nothing wrong with it".

Since you say the roboRIO lights go out, including the RSL, without flickering being present, I'm pretty sure you are going to find a "Layer 1" issue (to steal a networking term) and not a brownout issue. Comms normally stay up during brownouts even though movement goes to pot.

[ahartnet]

I think we were finally able to find and address the issue today.

The rio was definitely doing a full reset (all lights off, when power came back on, the rio would have the status light orange during it's POST, etc). I don't think the radio was resetting (typically) because I would only say two solid lights (momentarily) and I think when the radio reboots, you'll get three solid lights (momentarily).

Things that we did in order (all things that may have contributed):
1) Replaced the main switch (testing/flicking/shaking/tapping couldn't cause a failure, but was one of the few remaining components not replaced)
2) Replaced the 10 Amp fuse. It was one of the first things that I started wiggling this past weekend, but maybe the vibe loads were in a different direction than how I would wiggle it.
3) restrip a CAN cable that only had a couple of wire strands
4) Test run - still lose comm
5) Replace cable from battery to main switch
6) Add some mounts to reduce the vibration of the electronics board (it's mounted on some fairly thin lexan that could bounce significantly. It snapped near one of the mounting points which allowed more vibration on the side with the rio
7) Test run - drive without loss of comms for a while
8) Practicing - lost comms once, but we've decided that was likely due to a loss connection on that particular battery rather than anything on the robot anymore.

So I don't know if it's due to the battery cable or due to reducing the vibration that we're working better now. If it's just due to stiffening up the electronics board, I'm worried we're still just hiding an issue - but I suppose I don't know what kind of shock & vibe loads the rio is designed to with stand. I'm sure it's designed for more impact loads so maybe for some reason the bigger oscillations we were getting with a floppier than designed electronics board was causing an issue? Or maybe the strain relief we added wasn't enough. Either way, we're glad to have a running robot again.

We did get a lot of dropped packets. but that may be normal in our school setting, I haven't looked closely at that before (attached for reference).

[scca229]

Originally Posted by ahartnet

We did get a lot of dropped packets. but that may be normal in our school setting, I haven't looked closely at that before (attached for reference).

Your packet loss is at a very set interval, every 15 or 16 seconds start-to-start with a duration of 5 or 6 seconds each time, excepting the larger one which looks like a double event, but the one after starting right on schedule. I'm thinking environmental in the school, almost like something is updating via wifi and hogging your spectrum.

Good to hear that the robot is working better.

[Jonathan L.]

You said in your initial post that you replaced a power cable between the switch and PDP. However, the radio should be powered though the VRM. Is this the case?

[ahartnet]

Originally Posted by Jonathan L.

You said in your initial post that you replaced a power cable between the switch and PDP. However, the radio should be powered though the VRM. Is this the case?

Yes the radio is powered through the VRM

[frcguy]

Originally Posted by scca229

Your packet loss is at a very set interval, every 15 or 16 seconds start-to-start with a duration of 5 or 6 seconds each time, excepting the larger one which looks like a double event, but the one after starting right on schedule. I'm thinking environmental in the school, almost like something is updating via wifi and hogging your spectrum.

Good to hear that the robot is working better.

One thing I can think of is that many commercial access points used in places like schools have some sort of "threat management" technology, like this. To implement this so called "threat management", access points may contain a radio that can block, jam and/or attempt to force disconnects on any wifi network it is told to attack. Your school's access points may be programmed to do that to any network that they aren't told not to, which may include your robot. I suggest maybe talking to your network administrator at school to see if that is the case. Hope that helps!

[tr6scott]

Originally Posted by ahartnet

Unfortunately since I watch the rio reset, I don't believe it's because the 2.4 Ghz is overcrowded. It's worth checking however.

So did you?
Are you using 2.4GHz?
Did you try 5GHz?

[Team34Guy]

The barrel power connector can sometimes momentarily lose connection when it's jarred. This can cause your comms to shut down for close to a minute while the radio reboots. We had this happen in competition.

[phargo#1018]

Can you elaborate on the conditions at your demonstration even?

I have seen communications drop-outs at demonstration events for a number of years. Most years, our team has been forced to tether the robot to operate the system. This year's game makes tethering nearly impossible.

I have a thread, under 'Technical Discussions', that describes the problems I have seen, comments from others, and the corrective action we took for a demonstration event last weekend.

Hope that this helps!