Your take on CAN...

[RyanN]

I'm curious as to what other teams think of CAN. Is your team using it? If so, why? If not, why not? Any problems that keep on popping up for no apparent reason?

I ask this because I know Team Fusion wanted to use CAN for the main sake of getting rid of the spaghetti. We were also interested in the different control features such as a configurable voltage ramp, current status, voltage status, fault status, etc...

We wanted to use it last year, but encountered too many bugs (which were later fixed with new cRIO images and firmware) during build season. This year is a different story.

We did have an issue, which I cannot comment on, where CAN stopped working, but I was able to easily fix it when I went home for the weekend to mentor. Basically the cRIO somehow got stuck into safe mode, the students reimaged it, but did not turn Net Console off properly.

The main issue that worries me is the RS-232 portion from the cRIO to the Jaguar, as motor noise, vibrations, etc... can interfere with the connection, but supposedly the firmware is built strong enough to combat these problems.

I'm fairly confident about the actual CAN portion though, I mean they use CAN everywhere in industry, right down to the car you drive (if you're old enough )

We aren't placing all bets on it though. It's only running our shooter, so if it does go out, we're still a really good defensive bot with the ability to heard balls and cause major chaos on the field.

So yea... what's everyone else's take on CAN? Who's using it? Who isn't? Why? How are you using it?

[techhelpbb]

CAN on the Jaguars is pretty functional. Exceptions:

1. It's a single point of failure, make sure the wiring is correct.
2. Without using the 2CAN you can overwhelm the CAN communications.
3. If the Jaguars brownout you'll end up back in the default modes and if you're tracking encoders with them you'll have a problem. So don't build a system that browns them out (I know that's a lot simpler to say than do).
4. You can detect the brownouts and thankfully the newer firmware won't lock up like it used to.

[gracie.]

From the electrical team's point of view, CAN is awesome. We got immobilized in a couple matches last year because the PWMs on our jags kept falling out. It got to the point where we were using hot glue and electrical tape, but the damage was already done. I know it took programming a while to figure out how CAN works, but in the end I think it was worth it.
A word of caution... crimp your connectors so the wires are in the right order... oops.

[mikets]

I can't comment on CAN from past experience because this is our first year using CAN too. But we have been researching CAN issues in order to avoid hitting them. So we should be able to tell you our experience after this season . However, we can share some of the things we learned so far. For example, the issue about Jags browning out.
http://chiefdelphi.com/forums/showthread.php?t=101871
Reliability issue due to daisy chaining CAN bus.
http://chiefdelphi.com/forums/showth...light=backbone

There are a few issues we are still researching. One of them is about the bandwidth of CAN bus. It may be more relevant to you since you are using RS-232 instead of Ethernet which is much slower. Depending on how your program is structured, you may be sending multiple CAN messages across the CAN bus in each iteration of your robot loop. My understanding is that each message involves a send and a receive and they are synchronous. In our case, we have seven Jaguars on the robot. In an ideal scenario, our robot main loop will be updating all 7 Jaguars. 5 of our Jaguars have encoders so our ideal scenario is one Set() call for each Jag and one GetPosition() or GetSpeed() call for each Jag that has encoder. So ideally, we will have at least 12 CAN messages sent for each robot loop. I say ideally because I am making an assumption that we do not send more than one Set() or Get() for each Jag in a single loop. But we already found a few places in our code that's not true. So we are in the process of adding perf code to benchmark the number of CAN messages sent per loop and the average time each CAN command uses and how much time the CAN bus operation consumes in a single iteration of the main loop. We are doing this because we occasionally hit the MotorSafety timeout issue which we can't explain except for CAN bus traffic taking too much time in a single robot loop. We are instrumenting a CANJag wrapper class to collect this information. We hope to have some results in a week or so.

[Phalanx]

If you're looking to save bandwidth on the CAN bus consider using Periodic Status for each JAG. Configure each JAG in begin, and then you ask each JAG in teleop to obtain it's most recent cached status update.

Be aware, there is a minor bug as described here:
http://www.chiefdelphi.com/forums/sh...d.php?t=103629

Also consider, 128K for the Serial to CAN adapter, versus 1M using 2CAN and Ethernet.

[mikets]

Originally Posted by Phalanx

If you're looking to save bandwidth on the CAN bus consider using

Periodic Status for each JAG. Configure each JAG in begin, and then you ask each JAG in teleop to obtain it's most recent cached status update.

Be aware, there is a minor bug as described here:
http://www.chiefdelphi.com/forums/sh...d.php?t=103629

As far as I can tell from reading that thread, it is a LabView only feature. (no?) We are using C++. I don't seem to see that in C++ WPILib. In any case, like I said, we are planning to implement a wrapper class that shadows all config and parameters of the Jag, then we will create a periodic task (say 100 msec period) that will communication with each Jag once in each period to communicate the "changes". So if the caller is calling more often than this period, it will just get/change the cached shadow copy. I suppose this is exactly what the LabView library is providing. It would be great if the CANJaguar class in WPILib will support this for next year (also implementing the brownout safeguard).

[mjcoss]

We are using CAN and used them as well last year. My concerns are the introduction of a single point of failure (2CAN), and the side effects of the safety features of the Jaguars. In a properly running system, the Jaguars run fine, and don't brown out. BUT this is a robotic competition, and bad things happen.

There has been persistent issues with timeouts. Some claim it's all just bad wiring on the part of teams, others claim that the CRIO/CAN driver is to blame, others that the 2CAN is the issue, and others that the Jaguars are bad. The upshot is that we have a number of components and lot a software in between.

One thing that I stumbled upon lst year was that if a CAN bus message timed out, the caller (in C++) receives no indication that an error occurred. This results in the CAN request returning valid (0) but incorrect data. So the test GetForwardLimitOK will return false on a timeout'd request. This can wreak havoc on code that is unaware of the issue.

[Aren Siekmeier]

Our team hasn't much extensive research here, but here's our experience anyway:

Last year, both 2175 and 3130 (the team we share a shop with) tried it out during the build season, but it consistently caused teleop to lag, and it seemed it was due to traffic jams on the CAN bus. Since we weren't getting that much benefit from the CAN bus anyway and it was near the end of build, both teams switched to PWM control because we needed something that worked, and the lag in Teleop went away immediately. Both teams were using the Serial to CAN capabilities of the black Jags. Since then, we have found that the lag issues on the CAN bus may have been due to loose connectors (the pins can get bent too far back), but haven't tested any of it yet.

This year, 2175 went with all Victors to save space and because they are significantly simpler electrically and don't fault. Also, we would have gained nothing from using CAN this year because all of the mechanisms for which we want closed loop control are on more than one speed controller, so offboarding the control loop is not an option. So 2175 hasn't pursued CAN since the end of the 2011 build season (we may play around with it more this offseason).

3130 gave CAN another shot, and as far as I know have had success. They are still doing Serial to CAN. But one of them would have to give all the details there.

[techhelpbb]

Originally Posted by mjcoss

We are using CAN and used them as well last year. My concerns are the introduction of a single point of failure (2CAN), and the side effects of the safety features of the Jaguars. In a properly running system, the Jaguars run fine, and don't brown out. BUT this is a robotic competition, and bad things happen.

There has been persistent issues with timeouts. Some claim it's all just bad wiring on the part of teams, others claim that the CRIO/CAN driver is to blame, others that the 2CAN is the issue, and others that the Jaguars are bad. The upshot is that we have a number of components and lot a software in between.

One thing that I stumbled upon lst year was that if a CAN bus message timed out, the caller (in C++) receives no indication that an error occurred. This results in the CAN request returning valid (0) but incorrect data. So the test GetForwardLimitOK will return false on a timeout'd request. This can wreak havoc on code that is unaware of the issue.

I still fight with this issue as well so I'll second this concern. There's so many things that can cause it and I'm sure quite often it's a subtle combination. A lot of work has been poured into working the various bugs out, and still, if we wait long enough it'll come up (in the default modes, it comes up much more often in other modes).

[johnmaguire2013]

I don't understand motors, electricity, etc. however I have a question on behalf of my team, and I think I'm wording this correctly.

Basically, we used CAN with Jaguars for our robot's drivetrain this year. Multiple times it stopped working during the match, and our team was seemingly able to determine that this was due to built-in protection on the Jaguar to stop, essentially, surges. It was happening when a large amount of voltage was sent through the Jaguar I believe, and our team said this wouldn't be an issue if we switched back to PWMs. It would usually happen if our driver was moving back and forth very quickly (e.g. the bridge.) Does anyone have an idea of what was happening or how this could be fixed?

[techhelpbb]

Originally Posted by johnmaguire2013

I don't understand motors, electricity, etc. however I have a question on behalf of my team, and I think I'm wording this correctly.

Basically, we used CAN with Jaguars for our robot's drivetrain this year. Multiple times it stopped working during the match, and our team was seemingly able to determine that this was due to built-in protection on the Jaguar to stop, essentially, surges. It was happening when a large amount of voltage was sent through the Jaguar I believe, and our team said this wouldn't be an issue if we switched back to PWMs. It would usually happen if our driver was moving back and forth very quickly (e.g. the bridge.) Does anyone have an idea of what was happening or how this could be fixed?

I think you mean current not voltage (the battery voltage will generally get lower as you drive the robot).

There's a bunch of issues at work here:

1. The Jaguars and the Victors both reboot when the voltage of the battery gets too low (the battery can be too low because it needs to be charged, or because you are beating it down with heavy electrical loads). Victors, which use PWM only, will reboot fairly quickly and in some cases you might not even notice. The Jaguars using CAN can use quite a few modes and those modes have to be selected if they are not the defaults. If you use the Jaguars with PWM or the default mode on CAN (which performs a similar concept) when the voltage drops to a certain point (what is called a brown out) they come back up and just start working again (at least if you have the newest firmware they usually do). If you use another mode on CAN, then when they brown out you need to detect it and reset the mode. Problem is if you put an encoder on the Jaguar and it browns out...unless it's an absolute encoder...you'll loose track of the position of that encoder. Keep in mind that the brown out condition is not going to come back if the circuit breaker trips and hasn't yet reset.

2. You don't mention what you've got for a drive train. I assume you mean a drive train because of the way you've described it. However, it's very easy with the weight limits of the FIRST competition to build a drive train that draws more than 40A from the 40A breakers powering the CIMs. I assume you've put the CIMs in the drive train on the 40A breakers if not...that's probably gonna need to be changed. If you start to draw more than 40A from the breakers you'll start getting into the area where the circuit breakers are starting to trip (they take a while to trip) and fairly randomly the breaker will open cutting the power till it cools off (the longer you wait, the longer generally it will work after that...if you start doing it all over again shortly after the breaker cools, it's already pretty hot, so you'll probably cut out again shortly).

To add to this you've get the safety limits of the Jaguar. The Jaguars are designed to basically provide 40A. They'll provide 60A for a short period, but then they'll start to cut you back. The circuit breakers will also start to cut off at some point after 40A. So sooner or later you'll have the Jaguars cutting you back or the circuit breaker cutting you off for a bit, but one way or another if your drive train is quite often passing more than 40A of current you'll be running on the edge of failure.

Keep in mind the Victors won't make the breakers stop cutting off if you draw more than the 40A either. However, the Victors won't reboot into the wrong mode or cut you back. With the Jaguars your driver might mistakenly think being cut back means try to push to go faster (running you right back into the very overload you want to avoid and possibly pushing the breaker into opening). Remember that your drivers probably can't see the cut off in progress unless you report it, so they'll assume they are slipping or something mechanical like that.

A simplistic indicator that you're pushing too much current is the CIM motors will get quite hot. I warn you now that the CIMs can get hot enough to burn you so be careful if you plan on touching them to check that.

3. You might have a circuit breaker(s) in your power distribution board that are a little more 'touchy'. I've seen some cases where a drive train is marginally close to the current limits and variance in the performance of the circuit breaker causes the breaker to open. Usually the breaker is just doing what it should (in fact better than normal), but for this purpose it might be undesirable. Might want to try some other breakers it's quick and simple, but remember that if you don't quickly isolate a few 'touchy' breakers you are too close to the limitations of the system and you need to address that instead.

4. Are you sure you wired the effected part of the system correctly? Are the connectors firmly connected? Was the right gauge wire used? Are the screws properly tightened (be careful not to remove or strip them, sometimes when you remove the Jaguar screws you get some SWARF! from the bottom of the hole the screw was in loose in the Jaguar). If you can put a load (like a light bulb) on the wiring and check the voltage drop you can check to see if your wiring can handle the load (obviously there's more to this check than what I've written here...I'd ask a mentor or ask for more information if you want it).

5. Are you properly ventilating the Jaguars? Remember they have a fan inside them and it's important they get ambient temperature air flow. If you don't provide this you can cook the Jaguars and that's gonna lead to trouble.


What I wrote about is probably what they were trying to communicate.
Below are some more reasons your Jaguars might lock up or stop working for a bit.


6. If you're using encoders on the Jaguar then you might be loosing track of them. That can cause issues so you might want to think about how fast your encoders rotate versus their resolution, whether you properly installed the U.S. Digital encoder wheels, or if the encoder is electronically producing a clean signal. I won't wander too far off on this, it's a possible problem but I have no idea if you have encoders at all.

7. It appears some of the CAN bus connectors on some of the newer Jaguars might have some quality control issues. You might want to check the pins to make sure they make good contact. Also, it's easy to make bad CAN wiring, you want to check that because if you have bad CAN wiring it can stop several or all of the Jaguars from communicating and that might be intermittent.

8. You might be communicating too quickly on the CAN bus. The 2CAN is able to sustain a higher communications rate than the RS232 to CAN bridge in the black Jaguars. It's possible to overwhelm the communications process with CAN. Obviously if you're not properly communicating with the Jaguars they'll probably not do what you expect.

9. You might have set up your PID loops with poor choices for the relevant settings. This assumes you are using the PID functionality. It's possible that if the PID loop was setup properly your electrical system might not pass so much current, or the result will be the movements you expect. I can think of quite a few systems where a properly functioning PID loop might avoid an overload, especially systems with finite mechanical limits (like arms where if you don't stop you hit the mechanical limit and overload the motor trying to go past it).

There are 9 hints hope that helps.

[JABot67]

Originally Posted by techhelpbb

I think you mean current not voltage (the battery voltage will generally get lower as you drive the robot).

There's a bunch of issues at work here:

1. The Jaguars and the Victors both reboot when the voltage of the battery gets too low (the battery can be too low because it needs to be charged, or because you are beating it down with heavy electrical loads). Victors, which use PWM only, will reboot fairly quickly and in some cases you might not even notice. The Jaguars using CAN can use quite a few modes and those modes have to be selected if they are not the defaults. If you use the Jaguars with PWM or the default mode on CAN (which performs a similar concept) when the voltage drops to a certain point (what is called a brown out) they come back up and just start working again (at least if you have the newest firmware they usually do). If you use another mode on CAN, then when they brown out you need to detect it and reset the mode. Problem is if you put an encoder on the Jaguar and it browns out...unless it's an absolute encoder...you'll loose track of the position of that encoder. Keep in mind that the brown out condition is not going to come back if the circuit breaker trips and hasn't yet reset.

2. You don't mention what you've got for a drive train. I assume you mean a drive train because of the way you've described it. However, it's very easy with the weight limits of the FIRST competition to build a drive train that draws more than 40A from the 40A breakers powering the CIMs. I assume you've put the CIMs in the drive train on the 40A breakers if not...that's probably gonna need to be changed. If you start to draw more than 40A from the breakers you'll start getting into the area where the circuit breakers are starting to trip (they take a while to trip) and fairly randomly the breaker will open cutting the power till it cools off (the longer you wait, the longer generally it will work after that...if you start doing it all over again shortly after the breaker cools, it's already pretty hot, so you'll probably cut out again shortly).

To add to this you've get the safety limits of the Jaguar. The Jaguars are designed to basically provide 40A. They'll provide 60A for a short period, but then they'll start to cut you back. The circuit breakers will also start to cut off at some point after 40A. So sooner or later you'll have the Jaguars cutting you back or the circuit breaker cutting you off for a bit...but one way or another if your drive train is quite often passing more than 40A of current you'll be running on the edge of failure.

Keep in mind, the Victors won't make the breakers stop cutting off if you draw more than the 40A either. However, the Victors won't reboot into the wrong mode or cut you back...which with the Jaguars your driver might mistakenly think means go faster (running you right back into the very overload you want to avoid and possibly pushing the breaker into opening).

A simplistic indicator that you're pushing too much current is the CIM motors will get quite hot. I warn you now that the CIMs can get hot enough to burn you so be careful if you plan on touching them to check that.

3. If you're using encoders on the Jaguar then you might be loosing track of them. That can cause issues so you might want to think about how fast your encoders rotate versus their resolution, whether you properly installed the U.S. Digital encoder wheels, or if the encoder is electronically producing a clean signal. I won't wander too far off on this, it's a possible problem but I have no idea if you have encoders at all.

4.

Thank you for giving us some detailed instructions. I will attempt to clarify our situation.

We use CAN and have been having a specific problem during our matches where some, most, or all of our Jaguars will cut out perhaps halfway to three fourths of the way through a match. This happens when we go from full forward to full reverse or vice versa.

We do believe this has something to do with the current spike that occurs when the Jaguars are trying to spin the motors in the opposite direction from where we're going. You said it was very possible to build a drivetrain that will draw more than 40 amps when switching directions. That may be so with our drivetrain, but we have tried switching directions on a single Window motor (with no load) attached to a Jaguar, and it still fails.

We set the Jaguars to brake mode in code, and we also tried setting the "automatic ramp" rate. This stops the errors from occurring; however, our drive becomes less maneuverable because when switching from forward to reverse, the robot's motors will still be running forward for a while. We have tried several ramp rate values, and none of them were good enough to be used in a match. (We got DQed in a match because we hit the opponents' bridge while they were balancing. Not sure if the lack of control caused that specific infraction, but there sure was a lack of control.)

Our drivetrain is a four wheel tank with AM supershifters. Nothing too ridiculous. We use encoders plugged into the Jaguars, but only use them in autonomous mode. Auton is not a problem for us.

Would a switch to PWM cables on Jaguars (perhaps with the jumpers set to ramp mode) allow us better control of the robot and at the same time allow us to compete for an entire match?

We just want to be able to go from full forward to full reverse and vice versa during a match without our speed controllers failing on us, in order to give our driver the slickest control possible. Is that too much to ask for?

[techhelpbb]

Originally Posted by JABot67

Thank you for giving us some detailed instructions. I will attempt to clarify our situation.

We use CAN and have been having a specific problem during our matches where some, most, or all of our Jaguars will cut out perhaps halfway to three fourths of the way through a match. This happens when we go from full forward to full reverse or vice versa.

We do believe this has something to do with the current spike that occurs when the Jaguars are trying to spin the motors in the opposite direction from where we're going. You said it was very possible to build a drivetrain that will draw more than 40 amps when switching directions. That may be so with our drivetrain, but we have tried switching directions on a single Window motor (with no load) attached to a Jaguar, and it still fails.

We set the Jaguars to brake mode in code, and we also tried setting the "automatic ramp" rate. This stops the errors from occurring; however, our drive becomes less maneuverable because when switching from forward to reverse, the robot's motors will still be running forward for a while. We have tried several ramp rate values, and none of them were good enough to be used in a match. (We got DQed in a match because we hit the opponents' bridge while they were balancing. Not sure if the lack of control caused that specific infraction, but there sure was a lack of control.)

Our drivetrain is a four wheel tank with AM supershifters. Nothing too ridiculous. We use encoders plugged into the Jaguars, but only use them in autonomous mode. Auton is not a problem for us.

Would a switch to PWM cables on Jaguars (perhaps with the jumpers set to ramp mode) allow us better control of the robot and at the same time allow us to compete for an entire match?

We just want to be able to go from full forward to full reverse and vice versa during a match without our speed controllers failing on us, in order to give our driver the slickest control possible. Is that too much to ask for?

When some or all of the Jaguars cut out, do they do it at the exact same moment? Could be an intermittent connection on the CAN bus. Are all the units you loose sequential in the CAN chain? Perhaps the forward and reverse shakes something loose.

When you tried the window motor did you use a fully charged battery? You know when you suddenly shift the direction of your possibly 130lb robot you have to fight all the mechanical momentum of the robot that was previously moving the other way. You absorb that extra force with a surge of power from your battery. The more you suddenly shift the direction as hard as possible the greater the surge. Hence why the automatic ramp helps because you slowly absorb that energy...not all at once. If you absorb too much energy (relate current to kinetic energy of electrons) all at once the potential energy of the battery (relate voltage to the potential number of electrons to move out of the chemical system in the battery) and then keep trying, your battery voltage will drop and that can cause a brownout which all or part of the system might notice and reset. On the other hand, that's a big rush of current and that current might trip the safety of the Jaguar or the circuit breaker as well. Of course it could be a combination of all those possible issues as well.

You know if there's nothing else wrong with the system you can write your own ramp in software.
You can even simulate the brake function in your own software.
The team I mentor does this frequently.

[JABot67]

Originally Posted by techhelpbb

When some or all of the Jaguars cut out, do they do it at the exact same moment?

Usually that is the case.

Originally Posted by techhelpbb

Are all the units you loose sequential in the CAN chain?

No, they can but are usually not sequential in the CAN chain.

Originally Posted by techhelpbb

When you tried the window motor did you use a fully charged battery?

Yes, we used a fully charged battery. We have experienced and characterized the issues that occur on the CAN bus when the battery power is low.

Originally Posted by techhelpbb

You know when you suddenly shift the direction of your possibly 130lb robot you have to fight all the mechanical momentum of the robot that was previously moving the other way. You absorb that extra force with a surge of power from your battery. The more you suddenly shift the direction as hard as possible the greater the surge.

The error occurs even when the robot is barely moving. Our driver knew about this issue in the competition yesterday and purposely tried to avoid throwing the weight of the robot around. During no point in the match did he try going full speed forward (actual robot speed, not the speed of the wheels) to full speed reverse. The error mostly occurred when he was trying to turn. Turning does not absorb as much energy as a sudden shift from full forward to backward.

Originally Posted by techhelpbb

Hence why the automatic ramp helps because you slowly absorb that energy...not all at once.

Is there a good value for the automatic ramp that allows us the maximum amount of control possible without causing damage or disablement of the Jaguars? We were unable to find a good value.

[RyanN]

Originally Posted by johnmaguire2013

I don't understand motors, electricity, etc. however I have a question on behalf of my team, and I think I'm wording this correctly.

Basically, we used CAN with Jaguars for our robot's drivetrain this year. Multiple times it stopped working during the match, and our team was seemingly able to determine that this was due to built-in protection on the Jaguar to stop, essentially, surges. It was happening when a large amount of voltage was sent through the Jaguar I believe, and our team said this wouldn't be an issue if we switched back to PWMs. It would usually happen if our driver was moving back and forth very quickly (e.g. the bridge.) Does anyone have an idea of what was happening or how this could be fixed?

I think our mentor, Steve Phillips, puts this situation in the best perspective...

Imaging you're in your car, driving down the freeway, at 70MPH. Then you decide you simply want to go into the opposite direction. You change from drive to reverse, and hit the gas. What's going to happen? Well, I'll tell you what isn't going to happen... you won't be going the direction you want to be going. What will happen is that you'll likely break a lot of stuff in your car.

Same concept applies here to robotics. You have a 120lb robot going at 15FPS. That's quite a bit of momentum built up. Your driver suddenly wants to go the other direction, and slams it into the other direction. First off, the motors are creating back EMF of about 12V, and plenty of current capability, you then tell the speed controller to apply -12V, and give it all the current it can. You have a difference of 24V now, right? Given that you're technically going over the stall current because of the back EMF, you're probably putting a load of 200+ Amps on the Jaguar. It simply cuts out for protection. The Victors don't have this protection built in. They'll try to back drive, and the only overcurrent protection they have are the 40A snap action breakers. If you didn't have those in the circuit, they would literally self destruct, and actually, I know we have had a few in the past self destruct because of the conditions you described.

The ramp mode will keep you from reaching the cut-out condition, but you'll lose some maneuverability. The best solution is to make sure the drivers remember this concept, or switch to Victors and use PWMs.