Looking for Last Minute Fixes on a Possible "Comm" Error

[darkspade]

Hey everyone, I'm with team 4979, and we are in some desperate need of some help with a "comm" error on our robot. We noticed through all of our matches that every single time on the competition field about 2/3 of the way through the match, sometimes sooner, sometimes later, our driver loses his ability to control the robot. At first, it's a dead stop then eventually it's slow moving, and it will reconnect sometimes but then be immediately lost again. We've replaced our Ethernet cable between the Radio and RoboRio, replaced the Radio, and even moved the Radio (due to possible CIM interference). We've had every control system adviser look at it with us, and it seems we've hit some sort of stopping point.

Some symptoms that keep contradicting what could be the issue are that we've never been able to replicate the issue on a WIRED connection from our computer to the Radio. We're not seeing any browning out with our battery or problems in our amp, voltage, or bandwidth draw. We only see this issue occur on the field during the actual matches. We've cut it down to possibly either coding (which we've cut significantly but needs to be tested tomorrow in our first match) or maybe even a thermal issue with our CIMS heating up significantly and causing some sort of fail safe.

We're at the Midwest Regional, and we haven't been able to contribute much with our current status, but we really want to figure this issue out.

[ctt956]

Does that problem occur when wirelessly driving it from your driver station(not connected through the FMS)? Try putting it on blocks and running it for a few minutes wirelessly(if it's legal to do so). What are you doing in the game when it happens, or what were you doing right before losing coms?

[darkspade]

It's not really what we're doing but when. Autonomous and the first ~45 seconds always run smoothly. However, after that, its quite literally when we are trying to move. No shots, nothing. Its just moving after some time on the playing field. Even rough housing with the obstacles in the first 45 seconds to a minute works fine before we start losing it. Thanks so much for the quick response btw.

[Kevin Sevcik]

Does your driver lose all control of every system on the robot, or just drivetrain? What do you mean by "eventually it's slow moving"? Do you mean the robot moves slowly no matter the driver's input? Or do you mean it responds slowly, like joystick full forward, then 1 second later robot is tearing across the field?

[ctt956]

Originally Posted by darkspade

It's not really what we're doing but when. Autonomous and the first ~45 seconds always run smoothly. However, after that, its quite literally when we are trying to move. No shots, nothing. Its just moving after some time on the playing field. Even rough housing with the obstacles in the first 45 seconds to a minute works fine before we start losing it. Thanks so much for the quick response btw.

Do other functions(climber(if equipped), grabber/boulder pickup thing, arm, shooter, etc.) work when you can't move?

[darkspade]

Yeah, it seems as if our driver looses all control of every system on the robot. I'm not 100% sure about the other aspects though (i.e. shooter or our feeder for our shooter), but its definitely something I would like to play around with during our first match tomorrow. I want to say its the entire robot.

As for the disconnecting part, what happens is that our driver is able to move it one second and the next he isn't able to move at all. He can push the joystick down all the way and it won't budge. If he continues to attempt to do so, then occasionally it'll move like inches forward. It sometimes reconnects and other times disconnects.

Here's a video of it: https://youtu.be/USfCu9z7Rok. Sorry for the poor quality of the video. By the last 45 seconds you can see us stop completely, move a little bit, then attempt to move again. I'm the drive team coach btw.

[Kevin Sevcik]

That's really weird. Your light never goes solid yellow, so you're not losing comm. I think the frequency of the interruption is too high for it to be the old Autodesk Updater problem. Are you getting Motor Safety Helper errors in your Driver Station logs? That'd be a good clue to something bogging down your teleop loop. The timing of the motor movements seems about right for that.

EDIT: If you physically have the Driver Station available, you can pull up the Driver Station Log Viewer from the start menu and look through the logs. Help on all that here. I'm about to hit the sack, so I won't be more help tonight.

[darkspade]

I guess that we're fairly sure we've ruled out motor safety issues, but we simplified some of the programming to test tomorrow.

[Kevin Sevcik]

Originally Posted by darkspade

I guess that we're fairly sure we've ruled out motor safety issues, but we simplified some of the programming to test tomorrow.

Motor Safety is absolutely what you're going to see if you have "complex" code that's bogging down the roboRIO. The Motor Safety helper objects are like watchdogs for each motor controller. If you're not sending a command value often enough to a controller, you'll get a motor safety error and the controller will be disabled till you set it again.

[darkspade]

Originally Posted by Kevin Sevcik

Motor Safety is absolutely what you're going to see if you have "complex" code that's bogging down the roboRIO. The Motor Safety helper objects are like watchdogs for each motor controller. If you're not sending a command value often enough to a controller, you'll get a motor safety error and the controller will be disabled till you set it again.

We'll know for sure whether or not motor safety is the issue when we're testing tomorrow. We can post updates then. Thanks for your help.

[rich2202]

Just for kicks, check that your radio is wired into the 12V2A socket, and nothing is plugged into the other 12v2A socket.

They are supposed to check for it during inspection, but it is easily missed.

[philso]

Originally Posted by darkspade

Some symptoms that keep contradicting what could be the issue are that we've never been able to replicate the issue on a WIRED connection from our computer to the Radio. We're not seeing any browning out with our battery or problems in our amp, voltage, or bandwidth draw. We only see this issue occur on the field during the actual matches. We've cut it down to possibly either coding (which we've cut significantly but needs to be tested tomorrow in our first match) or maybe even a thermal issue with our CIMS heating up significantly and causing some sort of fail safe.

Where is your radio installed? Is it "buried" amongst a lot of large pieces of metal that make it more difficult for the wifi signal to reach the radio? Can you post pictures showing your overall layout and the area around the radio?

[ctt956]

Originally Posted by philso

Where is your radio installed? Is it "buried" amongst a lot of large pieces of metal that make it more difficult for the wifi signal to reach the radio? Can you post pictures showing your overall layout and the area around the radio?

Something I just thought of: is there a lot of metal or things that could interfere around the field? If this isn't your first competition, has it happened at other competitions? Did you change anything between them?

[philso]

Originally Posted by ctt956

Something I just thought of: is there a lot of metal or things that could interfere around the field?

The metal parts of the field would be far enough away that they will not block the signals from getting to the radio. It is more likely that large metal surfaces (ie. the diamond plate in front of the Driver Stations) on the field reflect the radio waves back onto the field.

[rich2202]

Did you ever figure it out?

My guess it is a code issue.

When you say it works "wired", did you ever take it through a "practice match" simulation while wired?

It could be in the FMS signals to the RoboRio, which continuously activates various routines (teleop_continuous, teleop_periodic, etc.). It may not be the same set of activations that you get when you merely "enable" when wired.

Some things that might be happening:
1) Two or more routines start conflicting, and it takes time for the conflict to manifest.
2) Memory leak somewhere, and the roborio has to stop and do some garbage collection.