cRIO cutting out 5 s after collision
 

Hi all -
Chill Out Team 1778 here. We are chasing an intermittent problem with our robot, and it is usually connected to a large collision. We've been troubleshooting the connections, but wondered if anyone has suggestions on how to isolate this problem effectively.

Robot config:
Basic kitbot chassis with 4 drive motor CIMs, and two additional motors for gate & rolller attachment. All 6 motors controlled by black Jaguars on a CAN bus wired to a 2CAN bus controller and terminated with a 100 ohm terminator. The 2CAN is also connected to a cRIO and on the network side to a wireless router.

Normally everything works fine, but if the robot gets hit hard on the gate, the robot will stop all response for 5 seconds before continuing. During this time the 2CAN shows a flashing red light, and then after 5 seconds returns to green flashing. We have checked the cRIO to 2CAN cable, and nothing seems to be wrong there. 5 seconds isn't enough time for the cRIO to reboot, so something else appears to be going on.

Thoughts?

Thanks in advance,
Rob

Re: cRIO cutting out 5 s after collision
 

I know that my electrical engineer mentor would hot glue the can connections on the jags to prevent them from coming out.

Re: cRIO cutting out 5 s after collision
 

Check your wiring. This is typically caused by some loose wires. They are typically in the least suspecting places and have a very high resistance. Something may have gotten disconnected after the collision!

Re: cRIO cutting out 5 s after collision
 

Well, some careful thought can tell us what it isn't: cRio power, for starters. Reboot would take longer than 5 seconds.

But you'll have to think about what else it can't be. When you have all those listed, see what's left: One of those has a loose wire somewhere.

Aside from hot glue, this is why we use Zip ties every 4-6" and bundle wires into a harness: Reliability and strain relief.

Good Luck.

Re: cRIO cutting out 5 s after collision
 

Check your Ethernet cables and the condition of your router ports. Also check the power connection to the 2CAN.

Red flashing means the 2CAN lost the cRio. As Don mentioned, it takes longer than 5 seconds for the cRio to reboot. It takes ~35 seconds for the router to recover from losing power (that hit us last weekend). Now that I think about it, the 2CAN takes about five seconds to reboot...

We've had small bits of plastic break off of Ethernet plugs and receptacles. The result is a physically loose and electrically intermittent connection. Everything looks fine, until a jolt in the wrong direction breaks a connection.

-Edit- I just re-read the OP. Sounds like a direct connection from cRio (8-slot) to 2CAN? My suggestion was based on the four slot cRio, assuming a router was in play.

Re: cRIO cutting out 5 s after collision
 

We have seen issues with the ethernet connections on the 2CAN being touchy in the past. Try wiggling some of these connections and see what happens. We had multiple 2CANs that had the same issue, worth a check.

Re: cRIO cutting out 5 s after collision
 

Thanks all for the great suggestions! We will check all those areas, and consider more ways to stabilize and provide strain relief!
Rob

Re: cRIO cutting out 5 s after collision
 

We originally had some suspect cables, these have all been replaced and no amount of jiggling or tugging can cause this fault in the pit. Additionally, artificial vibration in the pit (read: whacking robot with a hammer) cannot replicate this failure.

We replaced the 2CAN today, and still experienced the fault. [just read the comment from 1736... doh!] We have tried powering the 2CAN from both the regulated 12V and 24V regulated sources, and still experienced the fault. I have a hard time believing it is current related, as we have done plenty of high current maneuvering on our practice field without observing it. Additionally, the drivers seem to be reversing direction no problem on the competition field.

At this point we are very good friends with the CSAs. I know there have been some weird fringe electrical issues in the past with the 120 amp breakers doing weird things, and with bad PDBs, but I assume I'll be saying "DOH! Can't believe we missed that!!" when this is all over.

We played lots of back to back matches today, so tomorrow we are changing our cabling such that if we lose the 2CAN the cRio will still be able to talk to the field. As Don said, I don't think we'll get much additional information out of this -- if the cRio was losing power we'd be dead in the water for a long time.

Re: cRIO cutting out 5 s after collision
 

Al has been adamantly against hot gluing electrical connections because it's apparently "not an industry standard" (though I've seen it a thousand times).

Re: cRIO cutting out 5 s after collision
 

Should anyone stumbled upon this in the future, the issue stopped happening today. We had a tightly timed loop with no wait in our teleop code that was driving 100% CPU usage. We added a 10ms delay and our CPU usage dropped down to ~75% (on an 8 slot cRio). We also changed our cabling, so that instead of cRio -> 2CAN -> Router, it went cRio -> Router -> 2CAN. Not sure if it was one of these or the combination, but it stopped happening!

We may try to isolate it further in the off-season.

Re: cRIO cutting out 5 s after collision
 

We use this http://eclecticproducts.com/ag_adhesives.htm made by one of our sponsors.

And one of our other sponsors uses to secure electrical connections on the pharmaceutical robots they make.

This can certainly starve the code

Re: cRIO cutting out 5 s after collision
 

Glad you got things working (if not entirely solved). As an aside for anyone else reading, the original configuration as described should not have passed inspection as it violates R56.

Re: cRIO cutting out 5 s after collision
 

I know just enough about electricity and software to get myself in a whole bunch of trouble. Is there a plausible explanation why 100% CPU usage would make the 2CAN go out temporarily in a high energy collision?

I can see how our new routing could maybe be the fix. Even though we couldn't replicated it by tugging on cables, maybe there was something specific to the on-field collisions that made certain connectors in certain ports temporarily disconnect.

Re: cRIO cutting out 5 s after collision
 

I'm not sure if the two are connected but 100% CPU usage is never good. Maybe some "experts" can chime in about if they is any connection between the two.

Last year we put while loops and sequence tasks in TeleOp and it killed our code. Wasn't until this year that we discovered Periodic Tasks.

Re: cRIO cutting out 5 s after collision
 

I am glad you pointed that out. I think that we may have our 2CAN wired wrong too. We haven't had any problems but it seems we may be able to prevent some.