cRIO troubleshooting

[Sconrad]

So just recently, our 2010 robot here at FIRST Team 122 ceased to respond to our driver station. It said it found communication and robot code, but displayed voltage at 00.00, did not send signals to the pwms and caused the analog breakout chip to overheat and melt itself. After a couple of hours of reimaging the cRIO, redeploying the code, and switching out parts, we determined that the issue lay in the cRIO To make matters worse, we plan to ship this robot in a week or two. However, after long hours of toil, research and experimentation, we managed to make it so out cRIO works, but is not legal in competition. This thread is to help anybody who has similar problems with their robot. In order to keep this thread relatively short, I am going to post what we did as comments

[Sconrad]

To determine where the problem lay, we tried different batteries, to no effect, different digital sidecars, also changed nothing, different module and wire to connect digital sidecar to cRIO slot 4, still did nothing, and different cRIO, which solved the problem of unresponsive pwms and motors.

[Sconrad]

For the unresponsive pwms without using a different cRIO: In Labview, you have a choice of DIO slot. It defaults to 4, but can be set to 6. By plugging our digital sidecar into 6 rather than 4 and using motor vis instead of drive vis, we managed to regain motor control. Therefore, we determined that Slot 4 was busted, but not the whole cRIO.

[Sconrad]

For the battery voltage: the battery voltage is measured at slot 1 on the analog breakout chip. For more info:http://www.usfirst.org/uploadedFiles...reakout(1).pdf When we first discovered the problem, this chip was overheating and it actually melted itself in some places So, the first thing we tried was a new chip. While the new chip did not overheat, the battery voltage was still not being displayed. We tried a new module, but this did not solve the problem. After triple checking that we configured the chip correctly, we determined by process of deduction that the problem once again lay in the slot. We do not use any analog inputs, so we determined that we would just leave out the slot 1. While this is not legal in competition (see R61 of last year's rules), it works for demonstrations, which is all this bot will be used for.

[Greg McKaskle]

If the Communications and Robot Code light are on, that means that the cRIO has a startup app and image that is self consistent, and that the code is running. If the PWMs aren't being updated, that could be due to a number of reasons.

The analog breakout in slow 1 needs to have power and a breakout in order to return something other than 0.0. I don't know of anything a cRIO can do alone to melt a breakout board, but I could see how a faulty breakout, one that had power and ground pins shorted together might build up enough heat to melt, and clearly wouldn't measure anything useful while that was going on.

If you did something to your cRIO so that it is no longer valid for competition, I'm not sure that sharing that approach with other teams is the most helpful thing to share. Swapping cRIOs actually swaps lots of stuff including FPGA images and other libraries. Did you check what error messages were on the Diagnostics screen? Did you verify that the versions of the cRIO image and FPGA image were as expected?

Finally, would you please explain what was done to the cRIO to "fix" it but make it illegal?

Greg McKaskle

[Sconrad]

Although I can't say this would work for sure (haven't tried it yet), plugging the analog breakout chip into another slot of the cRIO might give us analog input. If this does work, battery status could be displayed in user messages by accessing analog input 1, which is devoted to battery voltage monitoring. Anywho, hope this helps anybody who runs into the same problems as we did. If you have any questions on how we did what we did, or general comments on things we might try to get it to work, please reply

[Sconrad]

Originally Posted by Greg McKaskle

If the Communications and Robot Code light are on, that means that the cRIO has a startup app and image that is self consistent, and that the code is running. If the PWMs aren't being updated, that could be due to a number of reasons.

The analog breakout in slow 1 needs to have power and a breakout in order to return something other than 0.0. I don't know of anything a cRIO can do alone to melt a breakout board, but I could see how a faulty breakout, one that had power and ground pins shorted together might build up enough heat to melt, and clearly wouldn't measure anything useful while that was going on.

If you did something to your cRIO so that it is no longer valid for competition, I'm not sure that sharing that approach with other teams is the most helpful thing to share. Swapping cRIOs actually swaps lots of stuff including FPGA images and other libraries. Did you check what error messages were on the Diagnostics screen? Did you verify that the versions of the cRIO image and FPGA image were as expected?

Finally, would you please explain what was done to the cRIO to "fix" it but make it illegal?

Greg McKaskle

Thanks for the reply! This is meant to help teams that would use the robot in demonstrations, as we will be using ours, where FIRST rules don't necessarily have to be followed to the letter. The only thing that makes this cRIO ineligible for competition is the fact that Slot 1 is busted (the cRIO took quite a hit in competition in 2010 and so several things were bent, but worked until now). Otherwise, the cRIO works. The point was not to share how we busted the cRIO (we drove up a ramp and the circuitboard flipped over and landed on the cRIO ), but to share how we made it operational so that we would not have to spend several thousand dollars on a new cRIO (we already bought one at the FIRST price, so we would have had to buy one full price). This was not a software issue (I can verify that, the code we were using was a simple tank drive and I ran through it at least 8 times during the debugging process), but a hardware issue. The software was only modified to adjust to the adapted hardware configurations we used. cRIO image and FPGA version were as expected. The only errors on the diagnostics screen had to do with the analog inputs, which was to be expected, seeing as both the cRIO slot that they were attached to and the analog breakout chip were busted. After examining the cRIO, we think the issue is in the board in the cRIO, which is probably cracked or corroded due to the bangs it has taken over the years. I would imagine this is a rare issue based on the difficulty I had finding documentation on a lot of the problems we had. I hoped to offer some documentation through this thread.

[Sconrad]

Also, this is a temporary fix. We are looking in to fixing the broken cRIO slots so that the cRIO is once again competition worthy, but we needed a temporary placeholder solution until we can go about accomplishing this.

[Greg McKaskle]

What I got out of that was ...

Our robot wasn't working, and in debugging it, we discovered that our cRIO had some slots that didn't work. Could've been that those hard hits from last year really hurt something. Luckily we have a spare. We need to get the other one fixed.

------------
Since I actually posted after the 9:21 post, but before the others showed up, I couldn't tell what was going on, and envisioned team members with soldering irons or hack saws. It is indeed possible to use the alternate slots for digital and analog with a few changes to Begin.vi or your startup code, but as you note, the blinking robot light is expected to be connected to slot four, and the battery is expected to be read from slot 1. Without those, most FTAs will not let you on the field.

I'd encourage you to contact NI and send the cRIO back get it repaired or replaced. You may even get it back in time to use as a spare for the competition. Also, look at the bus connectors along the bottom. Look to see if there are bent pins or debris that is shorting it out. If so, consider what operations may have led it. You may also try wiggling the connector and see if slot 1 or slot 4 seem looser than the others. It may be that the hit busted the solder joint. You may want to consider a more protected mount of the cRIO for the upcoming season.

Greg McKaskle

[jhersh]

Originally Posted by Sconrad

Although I can't say this would work for sure (haven't tried it yet), plugging the analog breakout chip into another slot of the cRIO might give us analog input. If this does work, battery status could be displayed in user messages by accessing analog input 1, which is devoted to battery voltage monitoring.

Analog Channel 8 is the channel with the battery voltage monitor (voltage divider).

-Joe

[jhersh]

Originally Posted by Sconrad

The only errors on the diagnostics screen had to do with the analog inputs, which was to be expected, seeing as both the cRIO slot that they were attached to and the analog breakout chip were busted.

The Analog errors do not check the status of the hardware... they only come from software bugs. What specifically were the errors you were seeing?

-Joe

[Sconrad]

We hope to get the cRIO back in full working order at some point, but for now, all we need it for is practice, and we have a new cRIO incoming, so getting it totally fixed is not a 100 percent priority. Thanks for the feedback though! @GregMcKaskle: I can see how I might have miscommunicated there. Thanks for pointing it out so I could clarify it. @jhersh I can't honestly say I remember the exact diagnostics message. I was pretty sure that they had to do with the analog inputs, but if they have to do with the code then that seems unlikely, as we did not even refer to analog inputs. Maybe the errors were from the driver station code trying to get the battery voltage? Anyways, foolish me did not record the errors and I am basing this information on memory, so it could be inaccurate. However, I can say that we did look at them and they were not anything foreign or totally unexpected. Like I said, we resolved the problem to the point that we are satisfied with the results. Thanks for the help though guys!

[Al Skierkiewicz]

Conner,
I am guessing from your description that you fried a chip on the analog breakout board that plugs into the 25 pin connector on the top of the analog module in slot 1. From that description I believe you had reversed the battery leads to slot 1. In the process you damaged the five volt regulator on the board. That is the only chip on that board. The board is not reverse polarity protected. There is a single LED on that board that lights when correct voltage is present and the regulator is functioning. Unfortunately, simply replacing the chip may have not repaired everything on the board.
It is also possible that a slot on the Crio is defective. Often this occurs when foreign material drops into the connector or a pin on the connector is bent. While a bent pin is sometimes recoverable, generally this is not the case.
Please remember that the battery voltage on this board is used (through the jumper) to sense battery voltage for the Crio. Should it fall to less than 5.5 volts, the Crio disables output for motor control. The intent is to allow the battery voltage to rise and prevent the Crio from rebooting. The power supplied by the PD may fail when the battery falls below 4.5 volts. This voltage (as sensed) is also passed to the dashboard and is read during the match by the FMS.

[Mark McLeod]

Originally Posted by Al Skierkiewicz

The board is not reverse polarity protected.

One small update...

The Blue PCB Analog Breakout (circa 2010/2011) is reverse polarity protected.

The Red PCB Analog Breakout (circa 2009) is not.

But if they are dropping heavy objects onto their Breakout boards and modules then all bets are off.

[Sconrad]

Originally Posted by Al Skierkiewicz

It is also possible that a slot on the Crio is defective. Often this occurs when foreign material drops into the connector or a pin on the connector is bent. While a bent pin is sometimes recoverable, generally this is not the case.

Based on our attempts to isolate the problem, it seems that at least two of the slots are defective. We thoroughly checked both and there don't seem to be any pins bent or foreign matter. Therefore, we think that the board within the cRIO is damaged. Like I said earlier, this cRIO had the misfortune of being underneath a flipped robot, and so it is likely that this caused initial, unnoticeable damage and that, over the course of the years, the damage has increased until the slots malfunctioned. Thanks for your suggestions though!