If your robot is jumpy, has watchdog timeouts, loses connection to the field, etc.
 

I just wanted to pass on a discovery we made about a problem with our robot, in case it helps some other team that is in the same boat, either this year (in off-season tournaments) or in a future year.

Twice this season (fortunately, not during a tournament match), our robot has suddenly developed a vexing problem where the driving is jumpy, the robot starts and stops, and there are watchdog timeouts (visible in the error display on the driver station). Very weird in both cases, because just minutes before, the robot was working great, with no change in software or wiring.

In both cases, we eventually discovered that the problem was an intermittent power connection on the robot. The first time, we found that one of the wires going to the robot's main breaker was getting loose, and making an intermittent connection. We fixed that, and the problem completely disappeared. We got lucky, and noticed a small spark at the main breaker that clued us in to the problem.

The second time, we discovered that the lug nut holding one of the wires to the battery had gotten loose, and the battery was making an intermittent connection. Swapping batteries fixed the problem. We tried the previous battery again, jiggled the battery wires, and found the problem. We then checked all of our batteries, and found multiple batteries with loose connections (yikes!).

Last year, there were several tournament matches where our robot lost connection with the field during the match for 30 seconds or so, especially when another robot collided with ours during the match. Field personnel told us, from their diagnostic screens, that our cRIO was rebooting itself during those 30-second blackouts. We had all kinds of theories about what might be happening -- perhaps we had a component that was not electrically isolated from the robot frame (especially the camera and the cRIO -- see <R36> in the 2011 rules), or perhaps the Ethernet connector to the robot radio was not a solid connection, or maybe the cRIO itself needed better shock mounting. We never did figure out that problem last year, but based on our experience this year, I'll bet you dollars to donuts we had some kind of loose electrical connection providing power to the cRIO and the rest of the robot.

So, if you find your robot is jumpy, starts and stops, has watchdog timeouts, or is losing connection with the field, I would strongly encourage you to check your robot's main electrical connections. In fact, these are excellent checks to make periodically in any case:

1. Check the connection of the wires to the battery, on all of your batteries. Make sure they are nice and tight.

2. Check the connection of the wires to the main breaker on the robot.

3. Check both ends of the wires from the power distribution panel that provide 24V power to the cRIO itself. Make sure the wires are firmly inserted into their connectors on both ends.

I hope this helps someone. These types of symptoms can be maddening, and the above checks are easy to make.

Re: If your robot is jumpy, has watchdog timeouts, loses connection to the field, etc
 

Randy,
A simple fix for your battery connections is something I have suggested here on CD over the years. It is something WildStang has used for years. When you are assembling the terminal to the battery post, add an external tooth star washer between the two mating surfaces. This, in addition to the locking hardware supplied, prevents the terminals from rotating with vibration or mishandling. Once the two terminal faces are locked in this way, the supplied hardware cannot loosen. The star washer also serves to cut through any surface corrosion or dirt on either the cable terminal or the battery terminal.
The loose hardware on the main breaker or PD is another common problem. It is an issue on the main breaker as the high resistance causes heat and that heat is conducted into the main breaker. This added heat derates the trip point of the main breaker and may cause the breaker to open during a match.

Re: If your robot is jumpy, has watchdog timeouts, loses connection to the field, etc
 

That is a very cool idea, thank you!

Re: If your robot is jumpy, has watchdog timeouts, loses connection to the field, etc
 

After fixing the above problems, and our robot behaving well for a while, our robot has twice more developed these same jumpy, start-and-stop symptoms, for other reasons. I wanted to share these as well, in case this info is useful to another team:

WIRELESS COMMUNICATIONS PROBLEMS

In our robot lab, we discovered we apparently have a lot of wireless communications interference in the 2.4GHz band that the Classmate PC on our driver station (and our programming laptop PC) uses to communicate with the radio on the robot.

We had two symptoms of this. The "Communications" indicator on the driver station kept flashing from green to red to green, and driver station commands to the robot were delayed, sometimes by a couple of seconds, which led to weird and dangerous jumpy robot behavior while trying to control it from the driver station.

In addition, our programming PC, which also uses wireless communications to transfer updated software to the cRIO, was weirdly slow when transferring code to the cRIO.

Using a long Ethernet cable to tether the robot radio to a wired Ethernet hub for both the driver station and programming PC completely fixed both of these problems, which clued us in that there was something wrong with the wireless communications. And the fact that both the Classmate and our programming PC had trouble communicating wirelessly with the robot either implied that the wireless capability on the robot radio was somehow defective, or that the 2.4GHz communications band had a lot of interference in our robot lab.

The FRC 2011 robot radio uses 802.11n wireless, operating either at 2.4GHz or 5GHz. On Page 7 of "How to Configure Your Radio," it recommends setting your robot radio to 2.4GHz, because the Classmate PC only has built-in wireless capability at 2.4GHz. However, in the footnote on that page, it goes on to say: "If a team's home facility has a significant amount of 2.4GHz wireless traffic/interference, the team may want to select 5GHz." We decided to follow this advice, and try using 5GHz in our robot lab.

To do this, we had to add 5GHz wireless capability to the Classmate PC we use for our driver station. (If your team uses an alternate laptop that already has 5GHz wireless capability, you can skip this step.) We bought and installed the Cisco-Linksys AE1000 High-Performance Wireless-N Adapter (about $50), which we attached to a spare port on the USB hub that we use with our Classmate PC. This is a high-end option; there are less-expensive USB/Wireless-N options out there (including the one below). You need to make sure you choose one that offers 5.0GHz, and that has Windows 7 drivers (since Windows 7 is the standard OS for the driver station this year).

It turns out that our programming laptop also only has 2.4GHz wireless, not 5GHz. So we tried out the less-expensive Cisco-Linksys WUSB600N Dual-Band Wireless-N USB Network Adapter (about $40) for our programming laptop.

Both of the above USB/Wireless-N adapters offer 5.0GHz and Windows 7 drivers (as well as drivers for Windows Vista and Windows XP).

We followed the radio configuration instructions in "How to Configure Your Radio" to set the robot's radio to use 5GHz rather than 2.4GHz.

So now, the robot, driver station, and programming PC are all communicating at 5GHz, and our wireless communication problems with the robot have vanished. The robot no longer stutter-stops or has delayed reactions, and the "Communications" indicator on the driver station is a solid green all the time. The programming laptop is now able to transfer new software to the cRIO quite fast, the same speed as when we have a tethered connection. I think we must have wireless interference at 2.4GHz in our robot lab, so this 5GHz workaround has fixed the problem.

NOTE: At an FRC event (or, at this time of year, an off-season event), you of course don't need these USB/Wireless-N adapters for your driver station or your programming laptop. The event organizers will of course provide you with a station where you encrypt your robot radio and put it into bridge mode so that you are using the wireless capability on the field. And in the pits, as you know, teams are only allowed to use tethered wired communications to the robot radio anyway. The above 5GHz workarounds would just be for use in your robot lab at home.

CHECK ALL ROBOT POWER CONNECTIONS, NOT JUST THE ONES TO THE MAIN BREAKER, BATTERY, AND cRIO

We checked and fixed all of the robot power connections mentioned in my original post on this thread (main breakers, battery, and power to the cRIO). Although those checks and fixes cleared up our problems earlier this season, we were again (as of a few days ago) getting jumpy behavior from the robot, even though we double-checked all the latter connections.

On careful investigation, we discovered that, over the course of the season, the screw had loosened on a power lead going into one of our Jaguars. As a result, it was making intermittent contact, which made that motor (and our whole robot) jumpy. We then checked and tightened the screw terminals on all of our Jaguars, and checked all of our other robot's power connections as well.

Tightening that Jaguar screw terminal (and the other Jaguar screw terminals) completely fixed the jumpyness of the robot.

So, I just wanted to add the above two items (checking and tightening ALL power connections on the robot including the Jaguars, and trying 5GHz network communications if your wireless connection from the driver station to the robot is iffy) as additional possible checklist items to look for if your robot is jumpy and unreliable. Hope this helps someone!

Re: If your robot is jumpy, has watchdog timeouts, loses connection to the field, etc
 

We had a similar issue with interference in the 2.4 GHz band from our schools local wireless system.

I connected a bridge to a computer to do a scan of all local networks to find that our robot network and the schools network were both attempting to use the same channel, in this case 10. I just set the bridge to a different channel that was not listed. That corrected the interference issue.

I was very surprised that we had 12 local systems. The lesson learned was to know what is in your air!

-Hugh

Re: If your robot is jumpy, has watchdog timeouts, loses connection to the field, etc
 

We had a problem that turned out to be that we mistyped the name of one of the joystick variables which was causing a delay and making the robot jumpy.