Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[jhersh]

Originally Posted by drakesword

Its my understanding that this has been modified for safety (which is understandable) to require the "trusted" heartbeat. Does anyone know the interval that the heartbeat must be receive in order to keep it alive? and Does anyone know the actually rate in which the heartbeat is being sent?

Either a heartbeat or a trusted set output command must be received by a Jag within 100ms of the last one to keep the Jag running. Every 15ms, the cRIO checks if the user has sent a trusted set output command within 75ms of the last heartbeat or trusted set output command. If not, it sends a heartbeat. This means that the cRIO will send a heartbeat to keep the Jag running somewhere between 75ms and 90ms (if needed).

[jhersh]

Originally Posted by mjcoss

One thing that I have seen which is causing no end of issues is that if you get a timeout on the messages, the API is return no indication of the failure. So, for example, if the GetForwardLimitOK() function is called, and times out, you get back false. There is no way to know that that has happened and if you are making decisions based on these results... We have an encoder on our lift mechanism. To zero the encoder, we drive to the bottom limit switch, and when we get there, we set the encoder to 0. This works fine until we lose the message due to timeout. From that point on the lift is offset by where ever the timeout occurred. There really needs to be a way within the API to detect that the transaction timed out.

Yes... that's a definite shortcoming of the C++ API. Both the LabVIEW and Java CANJaguar APIs have a mechanism for the user's program to know that there has been an error. I intended to remedy that this season, but didn't have the time. It will be fixed next year.

Originally Posted by mjcoss

All in all, I'm really regretting the decision to use the CAN bus. And for the most part all of the features that I really wanted to use, that were provided by the CAN bus, proved to be unusable.

I'm sorry to hear that you aren't happy with the CAN implementation. What features did you want to use that you aren't using? What made them unusable for you?

[mjcoss]

Here's what we were planning:

1) We are using a two speed transmission from AndyMark. It has uses 2 motors, and has a single optical encoder built into the the gearbox (on the output shaft). I had originally intended to use speed & position control on the drive subsystem. Position control for autonomous, and speed control during teleop. We first tried just using a y splitter to feed both Jaguars, and then tried a master/slave arrangement. While we got it to work, it seemed unstable. I'm sure that with more time, we might have been able to get it to work reliably but there really wasn't enough time.

2) Our lifter mechanism had the same problem. We had intended to use position control to put the arm at preset heights. Again, we had two banebot motors driving a custom gearbox with an encoder of the output shaft. The position controls would work for some positions, but would oscillate and not find the set point at others. I tried tuning it several times and couldn't get something that was 100% stable. Again we tried a Y split, and a master/slave configuration. We ended up moving the encoder to the CRIO and ran the PID loop on the CRIO. This gave me better control on the loop, and was immediately stable.

3) We have limit switches directly connected to the Jaguars, for the lifter mechanism, and the claw. The bottom one on the lift mechanism was used to home the encoder. We drop to the bottom limit switch and set the encoder count to 0. This works fine unless we timeout on the get limit message. If that message times out, the API returns at limit, and we stop. The premature stop of the motor made zero be in the wrong place, throwing all subsequent offset off by upwards of 6 inches. Apparently, in Java you can catch the exception, but I didn't see anything in the C++ code to allow the caller to know that the results of the call was bogus. I've modified my version of the limit switch calls to return an error status so that I can decide what to do if an error occurs.

Now add into the mix that, under some unknown set of circumstances, we get a flood of CAN bus timeout messages making the robot completely unusable. Given all that, one has to wonder if using the CAN bus is a good idea. In addition, we get some number of intermittent timeouts, which while not fatal causes weird behavior because the code can't determine that a timeout has occurred since the caller is returned a valid return value even if the call times out.

I have updated the firmware to the latest versions. I've tried two different bus terminators. And new cables.

One item that we've seen that has lead me to point a finger at the 2CAN is that on our robot we have an onboard compressor. We have noticed that if during autonomous if the air pressure in the system is low and the compressor needs to run immediately, we get the flood of CAN bus messages. It is as if the run all 4 CIM motors, plus rehome the lifter, plus drop the arm, plus run the compressor is a trigger for the failure My theory is that the voltage drop is causing a failure in the 2CAN. Of course, it could be something completely unrelated.

We are going to St. Louis and I'm seriously debating whether we should rewire the bot to use PWM cables.

[drakesword]

Originally Posted by mjcoss

Here's what we were planning:

One item that we've seen that has lead me to point a finger at the 2CAN is that on our robot we have an onboard compressor. We have noticed that if during autonomous if the air pressure in the system is low and the compressor needs to run immediately, we get the flood of CAN bus messages. It is as if the run all 4 CIM motors, plus rehome the lifter, plus drop the arm, plus run the compressor is a trigger for the failure My theory is that the voltage drop is causing a failure in the 2CAN. Of course, it could be something completely unrelated.

We are going to St. Louis and I'm seriously debating whether we should rewire the bot to use PWM cables.

Its my understanding that the 2CAN wont drop out till about 6 volts. Regardless are you wiring it to the termination blocks or to the 12 regulated supply or the 24 volt regulated supply?

We had ours wired on the 24v line parallel to the solenoid bumper. We had dropouts fairly often but when it happened we saw the PHY lights go out on the connector. We switched the 2CAN to port 2 and had another dropout some time later. We then swapped out our radios (we had issues) and never had a dropout since. We also developed a pre-match procedure where one of the drivers logged into the 2can and checked to see that all the jags were online. Once we started and only one jag showed up. Quick power cycle and all of them were back online.

I'm going to write a quick java applet that will display the 2CAN status information without the need to open a browser (aka have it pop up with the driver station on load)

[mjcoss]

Off hand I don't know for certain how the 2CAN is connected (I'm more of a software guy), but I believe that it powered via the termination blocks. Should it be moved?

Just as an aside, how are people connecting the 2CAN? We have the CRIO connected to the DLINK via the CRIO's port 1 and the 2CAN via the CRIO's port 2. The second connector on the 2CAN is unused. We had though of adding a camera to the robot and I was planning on plugging the camera into the unused port of the 2CAN.

However, when I first wired it up, I had the 2CAN connected to port 1 of the CRIO and the DLINK was connected to the second port of the 2CAN, but it seemed to me that in this configuration, the 2CAN was going to have a lot more work to do that was unnecessary, as it would have to bridge all the traffic to and from the CRIO and driver station.

[drakesword]

Preliminary release for the app is available. Go to this post

[rrossbach]

Our robot has 8 Jags (4 black, 4 grey) running off of 2CAN, and we do a fair amount of runtime data logging on the cRIO for post-match analysis. We see occasional timeouts (approx every 20sec or so) but catch them so it has never really caused a control problem.

Interestingly, even when seeing timeouts we have never seen any CAN errors reported on the 2CAN webdash - assuming that the webdash is working correctly, that leads us to suspect that the occasional timeouts not caused by any wiring/comms issues with the CAN bus itself, but rather something else going on in the software stack within the cRIO.

Once nationals are over we're planning to test a variety of scenarios (running Jags off the 2CAN both with and without the cRIO, requesting tons of status info from the Jags continuously, etc) to try to collect very specific data on what scenarios provoke the timeouts. It's a great learning exercise for the students, and hopefully the data will help narrow down exactly what's going on. Can't promise exactly when we'll dig into this but we plan to post the "scenario design" we come up with and seek input from the community here.

- Ron
Team #2607 controls mentor

[mjcoss]

So I finally got a chance to run some tests on our second bot that was made in parallel with our competition bot. It is not exactly the same but we were seeing similar CAN bus timeouts.

I replace the bus termination with a new one using a 120 ohm resistor (our old ones were 100 ohm), and we move the 2CAN power supply to the 24volt regulated power off the power distribution block (I think that what the electrical guy said0. Over a 10 minute session, cycling thru teleop, autonomous, and disabled states, we saw only 2 timeout messages.

Unfortunately, I can't recreate the one "known" failure case that we have seen on our competition bot. That is, if in autonomous, we need to run the compressor at the start of autonomous mode, we get a complete CAN bus failure. The problem is that our second bot was partial disassembled, and so not all of the components are running/attached.

But it seems like a good sign, although why it would loose even one is a bit of a mystery. Maybe if we increase the wait time a small step to see if the message was just delayed or was really lost.

[jtdowney]

Originally Posted by mjcoss

we move the 2CAN power supply to the 24volt regulated power off the power distribution block

Just a heads up, this is not a competition legal setup per R38A.

[techhelpbb]

We rebuilt the frame for our 2nd robot and once again, with an entirely rebuilt electrical board and new mounting for the cRIO (and some heavier wiring in some places)...we still have CAN timeout errors occasionally...though we can still manage in voltage mode as we have done so far. So this is now 3 entire reworks of that system all producing similar results. I should like to point out however, that we can literally stall this current robot and we can't demonstrate that the timeouts we see match that stall event.

After addressing some issues with crimps of the RJ connectors on this build, I'm adding to my list of projects a Jaguar CAN cable tester, maybe one that can produce a 1 or 2 MHz test signal for a testing of the wires. Maybe with a nice counter to see if the pulses are being dropped.

We have set up a collection of test equipment that can be mounted on our robot (while it drives around freely) and when we get done with competition hopefully we'll get some nice readings of the signals at various points in this system. Clear up some of possible issues that have been suggested. I'd really like to get a nice protracted recording of the CAN bus communications prior to the timeouts and an analog recording of the power supply voltages, the CAN bus and a few other things. At very least it'll be highly educational for the students and handy to have the tools (we'll share as we get it going). Hopefully what we'll work out will be budget conscious enough that it can be replicated by other teams without undue cost (so far I feel that's the case).