cRIO cutting out 5 s after collision

[rtrahms]

Hi all -
Chill Out Team 1778 here. We are chasing an intermittent problem with our robot, and it is usually connected to a large collision. We've been troubleshooting the connections, but wondered if anyone has suggestions on how to isolate this problem effectively.

Robot config:
Basic kitbot chassis with 4 drive motor CIMs, and two additional motors for gate & rolller attachment. All 6 motors controlled by black Jaguars on a CAN bus wired to a 2CAN bus controller and terminated with a 100 ohm terminator. The 2CAN is also connected to a cRIO and on the network side to a wireless router.

Normally everything works fine, but if the robot gets hit hard on the gate, the robot will stop all response for 5 seconds before continuing. During this time the 2CAN shows a flashing red light, and then after 5 seconds returns to green flashing. We have checked the cRIO to 2CAN cable, and nothing seems to be wrong there. 5 seconds isn't enough time for the cRIO to reboot, so something else appears to be going on.

Thoughts?

Thanks in advance,
Rob

[Kevin Phan]

I know that my electrical engineer mentor would hot glue the can connections on the jags to prevent them from coming out.

[yash101]

Check your wiring. This is typically caused by some loose wires. They are typically in the least suspecting places and have a very high resistance. Something may have gotten disconnected after the collision!

[DonRotolo]

Well, some careful thought can tell us what it isn't: cRio power, for starters. Reboot would take longer than 5 seconds.

But you'll have to think about what else it can't be. When you have all those listed, see what's left: One of those has a loose wire somewhere.

Aside from hot glue, this is why we use Zip ties every 4-6" and bundle wires into a harness: Reliability and strain relief.

Good Luck.

[Levansic]

Check your Ethernet cables and the condition of your router ports. Also check the power connection to the 2CAN.

Red flashing means the 2CAN lost the cRio. As Don mentioned, it takes longer than 5 seconds for the cRio to reboot. It takes ~35 seconds for the router to recover from losing power (that hit us last weekend). Now that I think about it, the 2CAN takes about five seconds to reboot...

We've had small bits of plastic break off of Ethernet plugs and receptacles. The result is a physically loose and electrically intermittent connection. Everything looks fine, until a jolt in the wrong direction breaks a connection.

-Edit- I just re-read the OP. Sounds like a direct connection from cRio (8-slot) to 2CAN? My suggestion was based on the four slot cRio, assuming a router was in play.

[jeremylee]

We have seen issues with the ethernet connections on the 2CAN being touchy in the past. Try wiggling some of these connections and see what happens. We had multiple 2CANs that had the same issue, worth a check.

[rtrahms]

Thanks all for the great suggestions! We will check all those areas, and consider more ways to stabilize and provide strain relief!
Rob

[Ian Curtis]

We originally had some suspect cables, these have all been replaced and no amount of jiggling or tugging can cause this fault in the pit. Additionally, artificial vibration in the pit (read: whacking robot with a hammer) cannot replicate this failure.

We replaced the 2CAN today, and still experienced the fault. [just read the comment from 1736... doh!] We have tried powering the 2CAN from both the regulated 12V and 24V regulated sources, and still experienced the fault. I have a hard time believing it is current related, as we have done plenty of high current maneuvering on our practice field without observing it. Additionally, the drivers seem to be reversing direction no problem on the competition field.

At this point we are very good friends with the CSAs. I know there have been some weird fringe electrical issues in the past with the 120 amp breakers doing weird things, and with bad PDBs, but I assume I'll be saying "DOH! Can't believe we missed that!!" when this is all over.

We played lots of back to back matches today, so tomorrow we are changing our cabling such that if we lose the 2CAN the cRio will still be able to talk to the field. As Don said, I don't think we'll get much additional information out of this -- if the cRio was losing power we'd be dead in the water for a long time.

[Michael Hill]

Originally Posted by Kevin Phan

I know that my electrical engineer mentor would hot glue the can connections on the jags to prevent them from coming out.

Al has been adamantly against hot gluing electrical connections because it's apparently "not an industry standard" (though I've seen it a thousand times).

[Ian Curtis]

Should anyone stumbled upon this in the future, the issue stopped happening today. We had a tightly timed loop with no wait in our teleop code that was driving 100% CPU usage. We added a 10ms delay and our CPU usage dropped down to ~75% (on an 8 slot cRio). We also changed our cabling, so that instead of cRio -> 2CAN -> Router, it went cRio -> Router -> 2CAN. Not sure if it was one of these or the combination, but it stopped happening!

We may try to isolate it further in the off-season.

[geomapguy]

Originally Posted by Michael Hill

Al has been adamantly against hot gluing electrical connections because it's apparently "not an industry standard" (though I've seen it a thousand times).

We use this http://eclecticproducts.com/ag_adhesives.htm made by one of our sponsors.

And one of our other sponsors uses to secure electrical connections on the pharmaceutical robots they make.

Originally Posted by Ian Curtis

We had a tightly timed loop with no wait in our teleop code that was driving 100% CPU usage. We added a 10ms delay and our CPU usage dropped down to ~75% (on an 8 slot cRio).

This can certainly starve the code

[RufflesRidge]

Originally Posted by Ian Curtis

We also changed our cabling, so that instead of cRio -> 2CAN -> Router, it went cRio -> Router -> 2CAN. Not sure if it was one of these or the combination, but it stopped happening!

Glad you got things working (if not entirely solved). As an aside for anyone else reading, the original configuration as described should not have passed inspection as it violates R56.

[Ian Curtis]

Originally Posted by geomapguy

We use this http://eclecticproducts.com/ag_adhesives.htm made by one of our sponsors.

And one of our other sponsors uses to secure electrical connections on the pharmaceutical robots they make.

This can certainly starve the code

I know just enough about electricity and software to get myself in a whole bunch of trouble. Is there a plausible explanation why 100% CPU usage would make the 2CAN go out temporarily in a high energy collision?

I can see how our new routing could maybe be the fix. Even though we couldn't replicated it by tugging on cables, maybe there was something specific to the on-field collisions that made certain connectors in certain ports temporarily disconnect.

[geomapguy]

Originally Posted by Ian Curtis

I know just enough about electricity and software to get myself in a whole bunch of trouble. Is there a plausible explanation why 100% CPU usage would make the 2CAN go out temporarily in a high energy collision?

I can see how our new routing could maybe be the fix. Even though we couldn't replicated it by tugging on cables, maybe there was something specific to the on-field collisions that made certain connectors in certain ports temporarily disconnect.

I'm not sure if the two are connected but 100% CPU usage is never good. Maybe some "experts" can chime in about if they is any connection between the two.

Last year we put while loops and sequence tasks in TeleOp and it killed our code. Wasn't until this year that we discovered Periodic Tasks.

[orangemoore]

Originally Posted by RufflesRidge

Glad you got things working (if not entirely solved). As an aside for anyone else reading, the original configuration as described should not have passed inspection as it violates R56.

I am glad you pointed that out. I think that we may have our 2CAN wired wrong too. We haven't had any problems but it seems we may be able to prevent some.