Robot heart attack
 

So shortly before a demonstration of our 2013 robot, I had jokingly mentioned that we had a transport cart in case the robot suddenly had a heart attack and couldn't move on it's own. Unfortunately, that is exactly what happened.

Just before the demonstration began, we ran a test and the robot seemed to work fine, with the exception of low battery power. We changed out the battery and removed the fuse for the pneumatics relay (had an air leak) and the fuse that lit the camera lights, just to try and conserve a little battery for the demonstration. When we tried running the robot again, every motor controller was dual-lighting (orange), and we can't seem to figure out where the issue is. Our experienced members have left, so no one is really sure what to do, including our advisor. Anyone think they can help out?

If you need more information, please ask specifically.

Notes:
Robot coded in Java
Seemed to be operational right before the demonstration
Driver Station indicates code and communications are normal
Code can be deployed onto the robot, but fails to control the robot systems


Will add more as people ask


Thanks for your time.
--Tim the Enchanter, Team 3950, Robo Gym

Re: Robot heart attack
 

That is indeed strange. Beyond the motor controllers dual-lighting, were there any other significant and visible issues you noticed? Was the robot signal light responding correctly? Was the driver station reporting communication and code? Is the issue persistent or was it a one and done kind of error? Was your pneumatic breakout showing signal changes if you tried to actuate any of the solenoids?

My apologies on the large quantity of questions; anything from the driver station to the PWM connection to the motor controllers could have failed (unlikely but possible). The more information the CD community has, the faster the issue can be identified.

Re: Robot heart attack
 

When 192 was working on one of our test drive bases, we removed the camera (the breaker to the camera) and the crio freaked out. It was looking for the camera and would not let us run anything, we had to remove the camera stuff from our code for the crio to respond correctly.

Re: Robot heart attack
 

If you turned off/unplugged the camera, and you have any sort of vision code on board, that is certainly your problem. The failure to receive camera image error will cause the cRIO to effectively DDoS itself, and may not allow certain other processes to execute.

Re: Robot heart attack
 

I just looked, and the camera code in LV has a two second timeout on the TCP operations. This can take place for each property set or get. So this can cause some delay in startup, but won't halt the program entirely.

If Java and C++ don't behave that way, I'd consider it a bug. In fact, I don't think the LV timeout needs to be that big either.

Greg McKaskle

Re: Robot heart attack
 

If the issue is not the camera, check all the cables on the cRIO. The serial cables may have been pulled when you pulled the fuses.

Re: Robot heart attack
 

What does "dual-lighting" mean?

Re: Robot heart attack
 

I knew I'd have more information to put in, but I couldn't think of it at the time.
First off, the dual-lighting simple means the Motor controllers light green and red at the same time, so it looks a bit orange.

Anyway, our signal light is currently not operational, but that's an unrelated issue. The light on the digital sidecar seems fine, though. I don't think it indicated any issue, but I could be wrong, I just don't remember.
The driver station indicates communication and code as normal, and we were able to re-deploy code onto the robot, but it moving the joysticks didn't control it.
The issue is persistent.
We did not test the solenoids individually, so I cannot say they caused any change in signal.

The camera itself remained operational, there was no issue retrieving images from it. Only the ring of LED lights was disabled.

Any more questions or ideas?

(Thank you all for replying)

Re: Robot heart attack
 

Were there any messages being reported on the Diagnostics tab of the driver station?

Re: Robot heart attack
 

Quite a few. I don't remember them off of the top of my head, and they weren't very clear error messages. One of them seemed to be about the joysticks, but I can't confirm that they are the issue just yet. Our testing will resume on Tuesday, and I will update if we get any further developments. I will also try to post the diagnostics from the Driver Station.

Re: Robot heart attack
 

If you want help figuring out what the problem is, there's something simple you can do to make that help more forthcoming: tell us what the system is telling you. You're going to have to give plenty of detail about what's happening in order for people to be able to interpret the situtation and give helpful advice about what to do next.

Are the yellow LEDs on the motor speed controllers steady, or blinking? Blinking indicates they're not receiving a PWM control signal; steady means they're being command to neutral (stop). Blinking could be caused by many things, with the error messages and status lights able to narrow down the possible causes. Steady could indicate a problem with the joystick or the robot code.

Re: Robot heart attack
 

Maybe a dead digital sidecar, or even worse, a dead cRIO. Some thing similar happened to our robot last year. During the last elimination match, our robot executed the autonomous "properly" and we, lost full control of the robot. When we checked our robot, we found that our cRIO was toast, but a computer instead of a piece of bread. Thankfully, we were carrying an extra cRIO with us. That saved us and got us into, I believe, the quarterfinals.

Re: Robot heart attack
 

Have you reverted to the original configuration (plug everything back in) with a fully-charged battery yet? Set a baseline and work from there. Maybe your diagnostic messages also existed prior to unplugging the systems.

Re: Robot heart attack
 

Some information I can answer, however. The motor controller lights are blinking; they do not give a solid light.

Re: Robot heart attack
 

Last year, week 5 of build season we had a similar problem happened to us. When the robot was enabled the cRIO wouldn't respond to inputs or print out data. We removed the cRIO and tried another one but had the same problem. Next we checked the digital sidecar. We went one by one removing the PWM cables to see if there was any small piece of metal shorting out the board. We eventually found the shaving and once it was removed the robot once again functioned properly.

Re: Robot heart attack
 

We immediately put everything back into place and used a fully-charged battery, but it has made no difference. I suspect the diagnostic errors were probably there, but I can't be sure, as I never checked before.

The cRIO might be the problem, but I can't be sure. I thought that there might possibly be a shaving or other metal debris interfering, as I noticed quite an accumulation of dirt and dust. I'll check the PWM cables tomorrow, but I wouldn't be surprised if there's a short somewhere. I'm no expert, but it just seems odd to me that every motor controller would give the same signal if there was only one short.

Re: Robot heart attack
 

So our problem has been resolved!

We did a lot of troubleshooting today, and after going through many of the electronic systems one at a time, we finally go to the point of switching out to our old Digital Sidecar. Once we switched to the old Sidecar, everything suddenly worked (sort of). A few tweaks to the wiring and some cleanup work, and the robot was working almost perfectly (just a pneumatics leak we intend to fix).

The Problem
The Digital Sidecar was broken. It didn't seem like it was, though. It gave us the lights we expected, but wouldn't delivier signals. Turns out the Voltage Regulator at Q1 was fried. I'll explain more in a little while.

Re: Robot heart attack
 

Broken is the normal mode of operation :) I think they had quality escapes that somehow worked :D

It's great to hear you got it working. My team has been through at least 4 in the past two years; I feel your pain.