cRIO cutting out 5 s after collision

[rtrahms]

Hi all -
Chill Out Team 1778 here. We are chasing an intermittent problem with our robot, and it is usually connected to a large collision. We've been troubleshooting the connections, but wondered if anyone has suggestions on how to isolate this problem effectively.

Robot config:
Basic kitbot chassis with 4 drive motor CIMs, and two additional motors for gate & rolller attachment. All 6 motors controlled by black Jaguars on a CAN bus wired to a 2CAN bus controller and terminated with a 100 ohm terminator. The 2CAN is also connected to a cRIO and on the network side to a wireless router.

Normally everything works fine, but if the robot gets hit hard on the gate, the robot will stop all response for 5 seconds before continuing. During this time the 2CAN shows a flashing red light, and then after 5 seconds returns to green flashing. We have checked the cRIO to 2CAN cable, and nothing seems to be wrong there. 5 seconds isn't enough time for the cRIO to reboot, so something else appears to be going on.

Thoughts?

Thanks in advance,
Rob

[Kevin Phan]

I know that my electrical engineer mentor would hot glue the can connections on the jags to prevent them from coming out.

[Michael Hill]

Originally Posted by Kevin Phan

I know that my electrical engineer mentor would hot glue the can connections on the jags to prevent them from coming out.

Al has been adamantly against hot gluing electrical connections because it's apparently "not an industry standard" (though I've seen it a thousand times).

[Ian Curtis]

Should anyone stumbled upon this in the future, the issue stopped happening today. We had a tightly timed loop with no wait in our teleop code that was driving 100% CPU usage. We added a 10ms delay and our CPU usage dropped down to ~75% (on an 8 slot cRio). We also changed our cabling, so that instead of cRio -> 2CAN -> Router, it went cRio -> Router -> 2CAN. Not sure if it was one of these or the combination, but it stopped happening!

We may try to isolate it further in the off-season.

[RufflesRidge]

Originally Posted by Ian Curtis

We also changed our cabling, so that instead of cRio -> 2CAN -> Router, it went cRio -> Router -> 2CAN. Not sure if it was one of these or the combination, but it stopped happening!

Glad you got things working (if not entirely solved). As an aside for anyone else reading, the original configuration as described should not have passed inspection as it violates R56.

[orangemoore]

Originally Posted by RufflesRidge

Glad you got things working (if not entirely solved). As an aside for anyone else reading, the original configuration as described should not have passed inspection as it violates R56.

I am glad you pointed that out. I think that we may have our 2CAN wired wrong too. We haven't had any problems but it seems we may be able to prevent some.

[Al Skierkiewicz]

When you see a 5 second gap after a hard collision, expect that the radio power supply was interupted. Not long enough for a complete reboot but certainly long enough for the radio to make a recovery mode and reestablish comms with cRio and field. Just enough will corrupt some of the data. With the 2Can in the loop, check to see how long it takes for the 2Can to recover from a power fault. That is more likely the issue. You moved the comms link so you likely moved the power as well. This is just an accident waiting to happen again. Check this when you can.
I recommend against hot glue for two reasons. If you have applied it correctly, you might not be able to change something if it has failed. That might forever damage the device. If it does pull apart easily, then the hot glue was not applied correctly in the first place and you are only fooling yourself that you fixed the issue. Just because you think you see adhesive applied in an industrial device, do not think it was meant to come apart. The manufacturer might just think the part is expendable.

[MrRoboSteve]

Any chance your radio connector has a custom cable? I've seen teams use the wrong barrel connector and see intermittent issues on the radio.

I've worked with enough teams with random issues caused by continuous* 100% CPU usage to know that pretty much anything can happen. The DS Log Viewer makes this obvious.

* I'm distinguishing "continuous" here from occasional 100% CPU usage, for example when complex imaging logic is running.

[kmusa]

Originally Posted by Ian Curtis

Not sure if it was one of these or the combination, but it stopped happening!

Great to hear!

We just spent 2 days at Buckeye tracking down Jag-related issues, which ultimately turned out to be a loose wire in a power feed to one of them. (We have the black ones on our robot.)

While we've used the additional features of the Jags to great benefit in the past, the fact that any one of a number of issues can shut down the entire string is problematic (and explains why a number of teams won't use them.)

There are CAN repeaters that would isolate the segments, but they seem pretty pricey for FRC applications.

[RufflesRidge]

Originally Posted by kmusa

We just spent 2 days at Buckeye tracking down Jag-related issues, which ultimately turned out to be a loose wire in a power feed to one of them. (We have the black ones on our robot.)

There are CAN repeaters that would isolate the segments, but they seem pretty pricey for FRC applications.

In the particular circumstance you are describing, the issue was likely a result of talking to a jag that was (at least temporarily) not there. Which means that a star configuration would have the same issue.

The CAN signal will pass right through an un-powered Jag, they use a short tap on the bus internally, it is not passed through a chip on the board.

The other possibility is that your power issue was on the Jag doing the RS232->CAN bridging which does require powered circuitry.

[kmusa]

Originally Posted by RufflesRidge

In the particular circumstance you are describing, the issue was likely a result of talking to a jag that was (at least temporarily) not there. Which means that a star configuration would have the same issue.

The CAN signal will pass right through an un-powered Jag, they use a short tap on the bus internally, it is not passed through a chip on the board.

The other possibility is that your power issue was on the Jag doing the RS232->CAN bridging which does require powered circuitry.

Thanks for the clarification, especially WRT how the bus is connected internally.

We are using the RS232-CAN bridge capability, I'll have to verify that the Jag in question was performing that function.

-Karlis

[geomapguy]

Originally Posted by Michael Hill

Al has been adamantly against hot gluing electrical connections because it's apparently "not an industry standard" (though I've seen it a thousand times).

We use this http://eclecticproducts.com/ag_adhesives.htm made by one of our sponsors.

And one of our other sponsors uses to secure electrical connections on the pharmaceutical robots they make.

Originally Posted by Ian Curtis

We had a tightly timed loop with no wait in our teleop code that was driving 100% CPU usage. We added a 10ms delay and our CPU usage dropped down to ~75% (on an 8 slot cRio).

This can certainly starve the code

[Ian Curtis]

Originally Posted by geomapguy

We use this http://eclecticproducts.com/ag_adhesives.htm made by one of our sponsors.

And one of our other sponsors uses to secure electrical connections on the pharmaceutical robots they make.

This can certainly starve the code

I know just enough about electricity and software to get myself in a whole bunch of trouble. Is there a plausible explanation why 100% CPU usage would make the 2CAN go out temporarily in a high energy collision?

I can see how our new routing could maybe be the fix. Even though we couldn't replicated it by tugging on cables, maybe there was something specific to the on-field collisions that made certain connectors in certain ports temporarily disconnect.

[geomapguy]

Originally Posted by Ian Curtis

I know just enough about electricity and software to get myself in a whole bunch of trouble. Is there a plausible explanation why 100% CPU usage would make the 2CAN go out temporarily in a high energy collision?

I can see how our new routing could maybe be the fix. Even though we couldn't replicated it by tugging on cables, maybe there was something specific to the on-field collisions that made certain connectors in certain ports temporarily disconnect.

I'm not sure if the two are connected but 100% CPU usage is never good. Maybe some "experts" can chime in about if they is any connection between the two.

Last year we put while loops and sequence tasks in TeleOp and it killed our code. Wasn't until this year that we discovered Periodic Tasks.

[yash101]

Check your wiring. This is typically caused by some loose wires. They are typically in the least suspecting places and have a very high resistance. Something may have gotten disconnected after the collision!