Robot Communication Lost for odd reason

[engbrech]

Hi all,

Sophomore team here getting ready for our initial competition during week 6. Today we were practicing with our shadow bot and ran into a problem that is baffling me.

We have a catapult that uses two CIM motor that draw full power when launching the ball. Over weeks of testing this has caused zero problems.

Today we deployed the program to the roborio such that is would start up on power on. Everything works fine, robot can drive, auto aim, etc... until we launch the catapult. When we do that, the robot shuts down, loses communication and restarts.

Other symptoms seem to indicate that the robot and roborio are simply drawing much more power when program is run at power on than we we deploy it from the computer. voltages drop significantly when the robot is driving though they don't turn it off. but when the catapult is launched, game over. Obviously, this will kill us at competition so I'm hoping for quick and helpful suggestions from some veterans.

Thanks all

Jason Engbrecht
Team 5434
Lead Mentor

[Alan Anderson]

Make sure you have a good, fully-charged battery installed. Look for loose power connections between the battery and main breaker and Power Distribution Panel. But my first thought is that you might have the output of one of your motor controllers shorted to something that it shouldn't be. Or maybe you have the wires crossed between the two CIM speed controllers, and the difference in timing when the program is run at startup vs. being run interactively makes it a problem.

[engbrech]

Thanks for the advice. Working on it right now and I see communications seems to come back but the robot code seems to stop running. Is this the symptom of a RoboRio brownout. I am getting voltage drops that seem particularly large and I think I'll start trying to track that down. But why would this be different in the interactive mode of deploying and the power on start up mode?

[nighterfighter]

Do your communications come back almost instantly? If so, that indicates that your code is actually crashing. (maybe you inadvertently divided by zero, referenced an object that wasn't there, etc)

A full reboot if you actually lost comms should take 30 seconds to a minute, depending on what rebooted (roboRIO vs radio).

[kevin.li.rit]

Double check to make sure the power to the RoboRiot is wired into the proper slot on the PDP. If its wired into the slots for the VRM or PCM, I believe those connections are unregulated.

[engbrech]

Thanks for more suggestions.

Here is more info as I trouble shoot.

The roborio was in the wrong spot. However, changing didn't seem to help.

Communication going down is variable. Sometimes yes sometimes no. It doesn't come back when it goes down even after a couple minutes.

Robot code always breaks until the power on the robot is reset.

I could certainly have a robot code issue but I can't be sure where to look for it. The issue never shows up when I deploy the code from the computer, only when I boot it on power up on the roborio.

I see voltage drops of about 3 volts (from 12 to 9) with current draws of about 100A as measure by the DS. Seems high to me, maybe others can direct if that is out of the ordinary. Doesn't depend upon the battery I put in.

I'm going to keep trying things. Suggestions are welcome.

[nighterfighter]

Originally Posted by engbrech

Robot code always breaks until the power on the robot is reset.

Do you mean that the Driver Station says "No Robot Code" until you powercycle the rio? And then it works until you fire the catapult, and it crashes back to "No Robot Code"?

If so, that's a code issue. (Unless it's something to do with LabVIEW deploy vs run, I'm not familiar with LabVIEW.)

If you're using a camera to autoaim, make sure when you do any calculations on the camera data, it actuslly exists. For example, in C++, we are using GRIP to get camera information via NetworkTables. If the camera doesn't see a target, GRIP doesn't publish any information to the network table.
Attempting to use those non-existant numbers results in the rio crashing and rebooting, while showing "No Robot Code". To fix that, we simply have an if block around that code saying "if (!table.empty())"


However, if you are actually losing power to the rio or Radio, it sounds like you have a short circuit somewhere. 9V is low, but not reboot everything low.

I'd suggest pulling breakers out of the PDP, one by one, until you isolate the issue.

Edit: When pulling breakers, be careful that you don't overload a motor. For example, if your catapult requires both motors to draw back, and having only one will stall the motor pull both breakers.

[engbrech]

That is what I mean nighterfighter. Those are messages on the DS.

I wasn't suspecting a code issue but I'm going to look into that. The whole "works when deployed but now when run from power on" absolutely baffles me. There are a couple reasons to think it could be related to vision so your post is leading me in that direction for a bit. We do program in Labview.

As to the drop to 9 volts, it happens with both the catapult and drive motors as far as I can tell. So I think if it is a wiring problem it is happening before the PDP. I'm also going to check that out as time allows.

Thanks for the feedback. Hopefully I'm on the right track.

[nighterfighter]

No Robot Code definitely means that something code wise is broken.

it might be a deploy setting. Someone more versed with LabVIEW will have to help with that part. When you imaged the roboRIO, was the little checkbox for "run code at startup" (or something similar to that name) checked?

It also sounds like you might have 2 separate problems. The robot code crashing is one, and the losing comms for long periods of time is the other. Losing comms sounds like a wiring/power issue.

Try performing a "tap test" on the main breaker. With the robot powered on, tap on the red button on the breaker (not hard enough to fully press it down though). If your lights on the electronics flicker, the main breaker needs to be replaced.

If both your drive train and catapult drop you to 9V separately, trying to run both at the same time will most likely cause a brownout. Your drive train should not drop you to 9V. What is your drive train (wheels, treads, etc) and how is it geared? How many motors per gearbox? (If more than 1, make sure they aren't fighting each other in the same gearbox)
I don't think that your catapult should drop you to 9V either, but CIMS draw quite a bit of current. Maybe surgical tubing/pneumatics might help alleviate some load off the CIMS? (Just spit balling an idea for that)

Finally, just for a sanity check: Leave all the hardware on your bot alone. Go back to the way it was working (deploy instead of startup I believe), and see if the robot still loses comms/code. If nothing fails, then we know it's something to do with the deploy/startup settings, assuming the order of robot operations remains the same. (Ex: Arm moves here then this motor turns for 5 seconds. If order of operations change, and something fails then you might have a shorted lead that only makes contact under certain conditions.) Or it might crash because an object/variable is being initialized a different order than what you expected.

[engbrech]

Have done the sanity check a few times. Can go back and forth between deploying and run on boot repeatedly and the problems comes and goes with that. Seems like a coding issue but a baffling one. I don't have a clue where to start on how the code might be executing differently in the two cases. But obviously I ought to look at the catapault code as a starter.

As for the voltage drop. Seems like that might be A problem but not necessarily THE problem. The drive (four mecanum wheels with a CIM each on 8.65:1) does not operate at the same time as the catapult so it isn't that the two combine for a large voltage drop.

Tap test causes no problems.

[thatnameistaken]

If you'd like another person to check the code, I'm decently experienced in LabVIEW and am willing to give it a look.

[rich2202]

If your RoboRio wiring was wrong, then check your Radio wiring. It should be plugged in to the 12v2A socket, and nothing else on the other 12v2A connection.

If launching the catapult causes the robot to crash, then that is where I would start to look for code problems.

If it works when deployed, but not when rebooting, then look for:
1) access to devices that may not be there.
2) access to devices that may not have yet been initialized.
3) access to variables/network tables/etc that may not yet been initialized.

What sometimes causes the "works when deployed, but not when rebooting" (or some other similar type combination) is that you are referencing a variable with an undefined value. Sometimes you are lucky, and sometimes you are not. On reboot, the memory addresses are usually in a default state that doesn't seem to work for you. On Redeploy, the deploy process may leave memory in a state that just happens to work for you.

Something as simple as a while loop conditioned on a new variable that is not first used until inside the while loop.

[engbrech]

I decided a little sleep was important but no I am back and troubleshooting.

Just ran a test with the voltage on the catapult reduced by a factor of 10 in order to learn if this was a power or coding issue. The code still showed the "deploy works, boot on power does not" problem making me quite certain now this is a coding issue.

I have a good bit of Labview coding experience so I'm going to spend some time looking for the issues (which I didn't do before because I thought is was a power issue). I'll post again when I've either located the issue or post the code if I feel lost again.

Thanks for the suggestions.

[Alan Anderson]

Just a shot in the dark -- have you recently copied or moved your LabVIEW project to another folder? If so, you might not be sending the program you think you are to be run at startup. Make sure the FRC Robot Boot-up Deployment settings have the right Local destination directory. I can't think of a simple way for things to get messed up so you're building to one folder and "deploying" from another, but it might be possible.

[engbrech]

Well I seem to have alleviated the issue but I don't really understand the reason. The catapult was being launched in the periodic task loop in a Labview subVi. It's launched was controlled with a timed loop structure. Switching that loop to a regular while loop with a "wait until next 10ms" control seemed to fix the job.

No more crashing the code.

Think are most working now except a vision problem that may or may not be related. I'm going to post that separately as I think that is now a new question.


Thanks for the advice. It was a heck of a problem to troubleshoot and being pushed away from a power problem into a code problem was what I needed.

Cheers