CAN on the entire 2012 robot

[steelerborn]

FIRST team 1671 the Buchanan Bird Brains used CAN on the entire robot, I know our electrical mentor went around in the pits at nationals and was surprised to see the lack of CAN and the Jaguars in general (We used the Black Jags). Just wondering if other teams that used all CAN could comment on their 2012 CAN experience, and why did some teams not run CAN and the Jaguars. From what I understand running CAN helps by letting the Jags run some of the loops and not in the C-rio which helps with processing power, maybe this could be related to some of the FMS and Communications issues on the field. Just a thought.

[duggerd]

The problem with using Jaguars is that if you put too much load on it, or change motion, or even change direction quickly, they reset and take a while to reboot and do not re init with the cRIO. This has killed our robot in the past and, even we had problems with it this year. In our Gallileo division we has a team run over our flipper witch ran on a jag, and when it was going up and applied too much force and reset our jag and killed the flipper for the remainder of the match. Using a jag does not affect your field connection issues. That is your bridge and their FMS.

[Levansic]

We at team 585 used CAN exclusively, at first. At our first regional in Seattle, we had some connectivity issues, and a few mentors from other teams were adamant that we had too many jaguars on our CAN bus. We use a 2CAN, and these mentors ignorantly (and belligerently) believed that the hard limit of CAN jaguars was only four nodes. They also suggested that using CAN was pushing our CPU utilization to 100%. Our small onsite team was pushed by them into changing four over to PWM.

In fairness, the problem in reality was the lack of robustness in the RJ-11/12 on the jaguars. Losing a PWM connection only takes out that single motor. Breaking the daisy chain on CAN takes away the termination, and usually loses everything. When this happens, acknoledgements don't come back to the cRIO. This may hang up the networking code on the cRIO. I'm only speculating on this last part.

Between Seattle and Las Vegas, our students were split on whether to go back to all CAN. The student who did the most CAN specific work decided to go back to all CAN. I supported her reasoning. We went through our LabView project, and combed everything. We enabled reentrant code, disabled debug code, and enabled in-lining on every sub-vi we could. This greatly optimized our code.

We also put together a backbone for the CAN bus, that had high quality RJ-12 modular connectors attached to continuous wires, similar to this. This changes the topology to a star, instead of a daisy chain. Our backbone had three modules with four ports each, rather than the two rows of six.

In Las Vegas, we were back on all CAN. We had found that one of our older beige jaguars had changed its address after a few matches (thoroughly befuddling). By the end, everything was working great. All 10 of our jaguars were working without issue on CAN. We did have a few communication drops, but I'm unconvinced that CAN had anything to do with that. Bad luck had our two final qualification matches against the thoroughly impressive one-robot alliance that is the 987 High Rollers.

Our robot shipped to the championship with everything working. When our team unpacked it, our shooter no longer worked. We found (again) that it was the RJ-11/12 connection on the Jaguar. They got it working pretty quick, as they had seen this problem enough, by now. Had our best performance all season at the championship, balancing every match, I'm told.

-- Len

[gracie.]

We had a lot of success with CAN this year. At first, it took a while for the programmers to sort out the needed modifications to the code, but after that it made things a lot easier from a wiring standpoint. We had 11 motor controllers this year, but our team only had 6 jags that were fit to put on the robot. So we ended up using PWM anyways for the 5 victors needed to fill the gap. However, CAN made my life a lot easier when putting the board together, and made it so we didn't have to worry about cables falling out on the field. We didn't have any connection issues from it. It was great!

[mikets]

We have 9 motor controllers on our robot and 7 of them are Black Jags. The other two are Victors just because we ran low on space. We don't have any problems with them at both regional and championship. There are a lot of posts on this topic. For example, one of them is:
http://www.chiefdelphi.com/forums/sh...d.php?t=103977

[Levansic]

One thing that did frustrate us, was the ease of going into a protected stop condition. Not really sure what to call the state, but we hit it often when trying to optimize our bridge tipper, while using closed loop current control.

Stall currents generally follow Ohm's law, since the motor armatures are not moving, at least when in steady state. We had several trips of the jaguar, before hitting 20 Amps (24 A stall current at 12V), and never saw voltages approaching the battery voltage. At lower currents, performance was reliable, but not enough torque for what we needed.

Now, we dont have an oscilloscope, so the back current spikes due to rapid voltage increases may have been huge, but the jaguar would sometimes failsafe out before there was any discernable movement in the attached motor. If we decreased our PID gains, the jaguar would not source enough power to get the motor moving. Not sure this is clearing much up, but we found the tuning to be finicky, and not robust when the temperature of the attached motor changed. If there was filtering on the feedback signal, or if a voltage output ramp rate could be enforced along with the PID, then this problem could be minimized or eliminated.

There was an earlier thread on PID speed control here. We thought we could tame that beast, but the fast sampling rate on the encoder, coupled with lots of backlash in our tough box nano gear boxes, made the feedback too noisy, and the control loop far to unstable to use. Not sure if there is any filtering on the feedback signal, in the jaguar PID code, but there should be.

We intend to do a lot of testing and characterization work this summer.

-- Len

[taichichuan]

Team 116 had a mixture of CAN and PWM on this year's robot. Jaguars on the ball guide and shooter motors (one on each side) and Victors to run the wheels due to size constraints and the annoying habit of the Jaguars resetting when the drivers do a full forward to full reverse thing. You could address that with motor ramping, but we wanted to give our drivers the ability to do whatever they felt was necessary to drive the robot on the field.

2010 and 2011 were pretty much all CAN except where there wasn't enough room to mount a Jaguar and a Victor was used. We had a couple of issues in 2011 based on a bad Jaguar firmware release. But, those problems appear to have been addressed with firmware 101.

However, to many, the Jaguars are damaged goods. Problems with aluminum falling into the Jags and shorting them out, problems with resets because of rapid full forward/full backward control movements, their sheer size, the relative ease with which students (and mentors) could reverse wire the input power and toast them and problems with the RJ-12 connectors have all contributed to the perception that the Jaguars are not a good choice. The fact that CAN can only be used for control with a Jaguar means that CAN gets a bad rap as "guilt by association".

I think that TI/FIRST putting the Jaguars out for re-competition is a good thing. The RFP is open enough to support some crucial modifications that have been long needed in the Jaguars. But, given the perception that the Jag are bad, I'm not sure that you'll find any manufacturer who'd be willing to take that project on with a design they didn't control the IP for with special firmware specifically designed for FIRST and in a really limited market like our robotics applications.

The Stellaris micro-controller is an ARM Cortex M3 -- a very stable chip that is similar to the controller used in the VEX control unit. The Stellaris line is very successful for TI with sales going up by a factor of 2x this past year. But, they just can't make any money on the Jaguars. So, this RFP is a way for TI to dump the Jaguar and move on to more lucrative ventures.

For folks that say CAN is unreliable, I ask if you drive a car made in the past 10 years. If so, you've likely got CAN bus. CAN is *very* reliable. But, the implementation that we see in the Jaguar can be prone to problems because of all of the failure modes of the Jaguars themselves.

However, the ability to get better low speed control and the ease of wiring the robot really speak well for the Jaguars in CAN mode. Add to it the closed loop support modes, and I think that CAN is ultimately a winner. We just need a better implementation, so we can put the CAN is unreliable issue to rest. So, I really hope that the RFP process will be successful and we can get a manufacturer for these new motor controllers that will stay with us while they work out the issues with CAN control and we work out the learning curve to be better able to take advantage of the features that CAN brings to the table.

Mike

[mjcoss]

Team 303 used all CAN as well. We had 8 motor controllers, 6 Black Jaguars, and 2 Tan ones. There are some real issues with using CAN, as others have mentioned. Lost configuration on brown out, not stellar PID controls, and timeouts of messages on the bus - as reported by the cRIO. In addition, you have a new single point of failure in either the 2CAN, or the serial cable. Lost of that device or cable will render your robot dead in the water.

We also had a case where one of the Jaguars on the bus went crazy and flooded the bus, causing us to loose comm with all of the Jaguars.

One failure case that we had (and is the reason for the 2 Tan ones) is on our shooter motor. If you rapidly changed the setpoint on our shooter motors, the Black Jaguars would eventual stop working completely, and required a reset, and reflash of the firmware. The Tan ones would reset and this triggered code which reloaded the configuration and start working again.

Lots of people don't like the high current shutdown of the Jaguars, and I'd like to have it be tunable but should you really be running the motors at such a high current? The victors would reboot as well but they rebooted quickly, and had no configuration, and would keep trying to do what you told them until they burned out either the motor or themselves. Seems better to save the equipment, and fix the high current issues.

[Alan Anderson]

Originally Posted by mjcoss

Lots of people don't like the high current shutdown of the Jaguars, and I'd like to have it be tunable but should you really be running the motors at such a high current?

There's already a circuit breaker to protect the wiring from overcurrent situations. The Jaguar's shutdown is there only to protect the Jaguar itself, and I think it's too aggressive.

[FrankJ]

We had 7 jags on our canbus without issues. 4 were on the mechanum drive cims, 2 on the fisherprice shooter motor, & one on a banebot 500 conveyor. Only the mechanum drives would be what I consider to be heavily loaded. We used the serial port on the Crio, Anal about the canbus wiring, and careful not to overload the canbus network with traffic. Had zero problems once other issues were sorted.

I did find out that there is an issue with the First implementation of canbus. If you write to an non existent node (Read resetting as well) You will crash the canbus. As I understand it the canbus "driver" on the Crio waits for an acknowledgement that never comes.

[mswalker]

We used CAN in 2011 for four motors without any problem. In 2012 we linked 9 Jaguars (about half each black/tan) with CAN. We had quite a few problems with loss of motor function and increased latency. I believe that the connections and wire integrity were the main issues. We spent several hours one night testing all the cables and connectors to make sure we had good wires and good connections. That worked for a while but then the latency came back. The latency showed up in a lack of responsiveness to the controls - e.g. stuttering instead of driving ahead smoothly. In each case it would take a lot of time to trace the entire daisy chain to try to figure out where we were having a problem. When we watched the network when we had problems we would see a large number of communications errors, e.g. can't find the address. We could usually fix the problems by replacing a cable or working on a jack but it all took time.

After our practice sessions at regionals we replaced the CAN connections with PWM. We didn't have any issues with control latency after that.

CAN seems to be the way to go but doesn't seem sturdy or reliable enough at this point.

[Brian Selle]

We had 13 motors on our 2012 robot, 9 were controlled with Jaguars/CAN using the 2CAN. The biggest issue we experienced early on was the connector pins not sprung out far enough to make good contact. After fixing that, everything worked flawlessly all the way through the championships. I suspect the latency issues mentioned earlier may have been due to using the serial bus instead of a 2CAN. We experienced none. We ran one closed loop PID loop with a high quality potentiometer and it was actually much less sensitive to the gains than running it through the cRio probably because the 1000hz update rate is more forgiving. Having some options to filter the sensor inputs would be very nice.

We had 2 alliance partners fail during our regional finals due to PWM cables coming loose (one was glued in place). I like hearing the RJ-11 "click" in place.

[JDNovak]

Team 16 has used CAN since 2010. We started with the 2CAN from the beginning and ran 10 Jaguars the first year and 11 Jaguars in 2011 and this year. The biggest issue the first two years was an initialization problem at boot up that was hard to detect but would render the robot motionless at the start of the match. This risk caused us to convert to PWMs at the end of last season.

This year we started again with CAN intent on finding the issue and solving it rather than avoiding it. We haven't had the problem with this year's firmware and CAN driver. We did have problems with the original 2CAN due to static but we were able to overcome that until we could get the redesigned 2CAN just before the start of the season.

There are tradeoffs with every system. Every time we build another robot we have cable issues until all marginal ones are found and replaced. After that we have had very little trouble. As several have alluded to, a cable problem usually takes the whole system down due to daisy chaning and failed termination. I intend to create a bus system that reduces this weakness similar to the ones referred to above. The 2CAN web page is invaluable in finding problems quickly. It can even be monitored through the router while driving to see motor loading and intermittant problems.

One side issue I would like point out is the Jaguar overcurrent protection vs Victors. I have had a few motors fail in the last year and a couple with very low resistance. The Jaguars shut down fast enough to keep the battery voltage above the reset point of the Crio and D-Link. I have heard of shorted motors on victors in at least a couple of cases that left robots dead while the system rebooted. Sometimes the quick overcurrent protection is worth something. We have had no problem with overcurrent shutdown on properly geared systems.

One of the reasons we continue to come back to CAN is the inherent current monitoring on every motor. One use is to protect motors in parallel. Our winch this year uses worm gearboxes in parallel to eliminate backdriving. if one motor doesn't drive the other is stalled and fails quickly. current monitoring allowed shutdown fast enough to avoid this problem.

As for PID control on board. the testing we have done has worked well but we always wind up back in the Crio because of lack of features in the Jaguar. Syncing two motors is a big issue. If there were some filtering options and the configuration was non-volitile I think we would be using it.

It will be interesting to see what transpires with TI backing out of manufacturing Jaguars but if we have to go back to PWM I'll be dissapointed.

My opinion is that a lot of the challenge of robot design is in finding how much technology you can incorporate into your design without making it the weak link. This about science and technology, isn't it?