Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[Zme]

i can tell you that it is not loosing power as there were outputs added to the code that would trigger when the GetPowerCycled() function returned true. a very early version of our code actually used both faults and power cycles as a trigger to re-init but it was taken out later, why it was taken out i don't know.

[techhelpbb]

Originally Posted by jhersh

Are you certain you aren't tripping a breaker or browning out the Jaguars under high load (due to poor wiring to the power input terminals of the Jaguar)? What you described are all symptoms of the Jag rebooting.

-Joe

I can be certain we didn't blow any fuses or trip any breaker.

It's possible, I suppose that the load is too great and we are somehow browning out, we did consider that and try fresh batteries.

We don't seem to be consistently dragging down the batteries which would loosely indicate we aren't being overly aggressive in a general sense.

I can certainly try putting a datalogger on the power supply side of the Jaguar and see what I can like that. Not sure I have a current probe I can use in that configuration on 12VDC and a shunt is not a good idea given the load impedance.

The wires are clearly of sufficient gauge...I didn't wire that part of the system but it's better than 12AWG (so it's not 12AWG), and the problem doesn't seem to follow any particular wire so it's not likely the crimps. We've crimped plenty of other wires in the past with these tools and this make and brand of lug and they are firmly screwed down.

I could try putting a large 25V or 50V capacitor near the Jaguar on the input side to provide additional low internal resistance power if somehow the impedance of the wires from the power distribution board was suspect. It may not be legal on the competition floor, but for troubleshooting I think the battery will handle that.

[Mr. Lim]

We are also seeing an occasional CANTimeoutException being thrown under brownout conditions.

Strangely, it is one Black Jaguar in particular that is always the one that goes first. It is connected to a CIM in our 4-motor drivetrain, and it is a "master" Jaguar, in that we have an encoder hooked up to it, and run it under speed control.

Under a fresh, full battery, the exceptions don't occur, but within a 2 minute match, the voltage certainly drops enough to start throwing these exceptions. We feel it is voltage related, as when we are in low gear, the exceptions never occur, whereas in high gear, they happen very frequently.

Our workaround so far has been to catch the exception, and re-initialize the Jaguars when they occur.

These seems to work decently well.

We've gone through the wiring several times, and can't seem to isolate why one Jaguar is more prone than the others. Less tolerant Jaguar? A CIM that likes to draw more current than the others? Your best guess is as good as mine...

[techhelpbb]

Originally Posted by Mr. Lim

We are also seeing an occasional CANTimeoutException being thrown under brownout conditions.

Strangely, it is one Black Jaguar in particular that is always the one that goes first. It is connected to a CIM in our 4-motor drivetrain, and it is a "master" Jaguar, in that we have an encoder hooked up to it, and run it under speed control.

Under a fresh, full battery, the exceptions don't occur, but within a 2 minute match, the voltage certainly drops enough to start throwing these exceptions. We feel it is voltage related, as when we are in low gear, the exceptions never occur, whereas in high gear, they happen very frequently.

Our workaround so far has been to catch the exception, and re-initialize the Jaguars when they occur.

These seems to work decently well.

We've gone through the wiring several times, and can't seem to isolate why one Jaguar is more prone than the others. Less tolerant Jaguar? A CIM that likes to draw more current than the others? Your best guess is as good as mine...

I can say for certain we had one CIM in our pile of parts that had some issues with brushes. No matter what we did with that CIM it was trouble everywhere it went. It sort of worked on a Victor 884, but every time it touched a Jaguar of either model you'd either fault on startup or you'd get moving and shortly there after strange and bad things would happen.

We discovered it early on. I thought we disposed of it, or at very least marked it. Suddenly it 'reappeared' and made for another 2 or 3 hours of troubleshooting. I believe one of the other mentors showed it to the garbage can when I wasn't around so that it couldn't find it's way back into anything important again.

I can also say that one side of our robot drive train exhibits higher inertia than the other. While messing around with encoders before we isolated them that side was by far more likely to really have serious show stopping issues than the other which for the most part is the same exact design. Once we made the isolation board it stopped misbehaving and for the most part accepted the same PID tuning parameters for a setpoint of velocity (you could at least start tweaking the values from the side with less inertia and you'd be close). It drove nice and straight once we isolated the encoders until we started getting the CAN bus issues and then, after we trapped that error, it was good again. It doesn't seem to me that this particular part of our robot is any less well built than the other. We didn't, after all, fabricate the gear boxes themselves, and cursory examination doesn't indicate any obvious mechanical issue. However, it seems just a slight variation in the mechanism and even if we take the Jaguars from the side with less inertia they seem to have issues on the side with more inertia. Now I do grant the reader that in swapping the Jaguars we are using the wiring from the side of the robot that has problems and that might be why that side keeps being slightly more prone to problems. The wiring just does not itself seem bad.

As soon as I get a chance I'll rig a datalogger up on the robot while it moves. I'd really like to see what the power supply side actually looks like when this goes on.

[John Heden]

Originally Posted by John Heden
At this point I’m partial to the startup race condition theory of some type.

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

Greetings,

I was curious as to whether your testing was succesful and whether we have a fix at this point ? Would it be a fix on the FMS side or a new CRIO image ?

Thanks,

John

[jhersh]

Originally Posted by John Heden

Greetings,

I was curious as to whether your testing was succesful and whether we have a fix at this point ? Would it be a fix on the FMS side or a new CRIO image ?

Thanks,

John

It has nothing to do with the FMS. We are testing a new image that should fix it. We have had no failures yet. However, FIRST wants more testing before making it public.

Are you aware of any teams having trouble at any regionals using v28 this weekend?

-Joe

[ozrien]

Originally Posted by Mike Copioli

I will give you a preliminary answer until Omar has a chance to provide a more detailed one. In short the problem is caused by a lack of sychronicity between the cRIO CAN transactions and the 2CAN dashboard transactions. This is a simple explanation of the problem, it is actually a bit more involved as Omar has explained it. Omar has written some management code that is intended to deal with this problem, however the web dashes ability to interact with the CAN bus is second chair to the user code. If the user code is sending can throttle requests to frequently, for example, the time the web dash has to interact with the bus is limited. This is not an issue with the Cross-link Control System because the 2CAN performs all synchronization and has more of a 'master' role. But again this is Omars area of expertise.

The 2CAN keeps track of outstanding transactions from the cRIO and from the Web dash. The 2can knows which CAN ids represent requests that will have ACK responses and keeps track of when the cRIO or WD is waiting on an ack.
So if WD wants to request say bus voltage on Jag1 it first checks if cRIO has performed any requests (resync tokenization, throttle set, etc..). If cRIO has then WD will hold off on transactions to Jag1 until Jag1 sends an ACK (intended to go to cRIO).

cRIO Set Throttle
<--------------------------User opens Web Dash, WD holds off...
cRIO ACK
WB Get Voltage <--------------- Jag sends an ACK, it is now available
WB ACK

Similarly if the WD is waiting on Jag1 for a response/ACK and cRIO request comes in, it will delay transmission until Jag1 responds, OR a one millisecond timeout occurs (to ensure cRIO gets minimal latency to bus ).

Without this management code there would be problems where...
cRIO Sets Throttle
WB Gets Voltage <--------------------------User opens Web Dash,
ACK
ACK
...which confuses the Jag and sometimes causes Jag to not respond at all.

[John Heden]

Originally Posted by jhersh

It has nothing to do with the FMS. We are testing a new image that should fix it. We have had no failures yet. However, FIRST wants more testing before making it public.

Are you aware of any teams having trouble at any regionals using v28 this weekend?

-Joe

Thanks Joe!

No, I'm not aware of any other teams reporting CAN startup problems with V28 from this weekend. Perhaps the Team Update #17 suggestion to monitor for CAN errors was effective (or perhaps scared a few more teams into a PWM conversion). Maybe someone at this weekend’s competition might add their observations on this issue…

[ozrien]

New 2CAN firmware 2.5 available
http://www.crosstheroadelectronics.com/2CAN.htm

I found and fixed a few issues that could be related...
Major Fixes....
-Fixed a potential situation where 2can stops sending udp frames - only happens on higher Ethernet utilizations.
-There is an improvement to lagging Web dash browsing while robot is enabled.
See release notes for details on all fixes.

There is no FRC legal obligation to update from 2.1 to 2.5. Both function with the FRC_2CANPlugin (SVN rev66).

[PhilBot]

I’d like to throw a totally unreported problem into the CAN problem bucket.

This relates to closed-loop potentiometer-based arm control.

This happened to our team after the 2’nd to last qualifying match at the Pittsburgh regional this weekend.

Setup: We use 4 Jaguars running on Serial CAN. The first 2 “drive” Jags are run open loop voltage control (no encoders) but the last 2 “Arm” Jags are switched between open loop voltage control (for manual adjustment), and Potentiometer based Closed loop control (for recalling pre-learned arm positions). A single VI is used to do this switch by issuing new sets of setup commands each time the mode is changed.

We also have two VI’s in “Periodic Tasks” that do a “read back” of the current arm position for diagnostic purposes and to use as inputs to LEARN arm position presets. This has been working seamlessly for the last week of build, and the first 2 ½ days of competition. Note: This read-back isn’t needed for normal operation because the Jags are normally sent “Goto” positions and left to do their own closed loop control.

Just before the last match of Quals, we broke an arm chain while trying to adjust a position reset (using our tried and true “Learn” mode). The root cause for the mechanical failure turned out to be the fact that for some reason both Jags were reporting arm positions of “0.0”, instead of the actual arm position (so the learn process was recording bogus information).

We hadn’t made any software or hardware changes for several matches, and we tried multiple reboots, measured pot voltages, checked wiring etc., but couldn’t get the jags to read proper values. It was odd that both Jags were reading zero position although they had little in common physically (other than the CAN bus and the software). There were no more than the usual occasional drive Jag CAN errors on the diag screen. Eventually we loaded up the code in debug mode to see exactly what was coming out of the Arm Jags. To our surprise both JAGS were reporting no errors and positions of 0.0. We did a Highlight Execution to verify that the reads were being done… they were, but returning no valid data.

Through this entire operation, the Arm JAGs were still responding to voltage output commands and could drive the arms to any position. They just would not return valid “position” status on request.
We were stumped as to what to try next. If I’d had my USB to Serial adapter I would have reloaded the JAG code or run the COMM program (bad me for forgetting to bring that).

The very odd thing was the problem corrected itself as if by magic after an hour of debugging when we loaded some new arm presets onto the CRIO flash drive programmatically. I can’t explain this in detail as it would require a full understanding of our code, but suffice it to say that this change should not have changed the JAG operation in any tangible way…. all it would have done is sent more reasonable “value” into the Drive VI as pot setpoints.

Once things started working again, we just rebooted back to the original code (that had never been unloaded) and redid the presets using the normal “learn” mode and everything seems to be back normal (except my future failure fears)

Has anyone seen this reluctance of the JAGs to return pot position, especially once they have been setup correctly?

[jhersh]

Originally Posted by PhilBot

Has anyone seen this reluctance of the JAGs to return pot position, especially once they have been setup correctly?

Hi Phil,

That can happen if a Jag reboots or have a communication problem when you are configuring the Jag Position reference. I'm guessing if you had set the position reference again, you would have seen it fixed. Maybe your jags were browning out due to a low battery?

-Joe

[PhilBot]

Originally Posted by jhersh

Hi Phil,

That can happen if a Jag reboots or have a communication problem when you are configuring the Jag Position reference. I'm guessing if you had set the position reference again, you would have seen it fixed. Maybe your jags were browning out due to a low battery?

-Joe

None of the above.

When the problem first occured we changed batteries and re-booted several times. Changed batteries again later after matches were over. No effect.

The problem is that this "0.0 postion" error persisted for many reboots, debug download etc. and then just returned to normal and hour later.

As for the code...

The Jags are switched between "Voltage out" and "Position mode" each time a preset recall button is pressed (and held). Before the new mode parameters are loaded, the JAGs are disabled, and then re-enabled at the end. All of the JAG parameters (control mode, position source, pot turns, PI gains etc) are re-loaded, and the code checks for any errors and re-tries the command sequence 3 times if errors persist.

Note: We'd run into the occasional bad-command-write problem several days before shipping and had built in the retries. So it's pretty bullet proof.

Do you think accidentally sending the JAG an invalid "position value" (eg: a negative value for target position) could lock up the controller????

Phil.

[techhelpbb]

Adding to my list of troubleshooting tools that I guess I'll be fabricating...hopefully I can keep enough students engaged at this point...

It's far too easy to think that you have a brown out condition on this robot.

The power source is limited and the surge currents for the CIMs can exceed 120A each, with each robot having generally at least 2.

Then add the fact that the platform moves...and the brownouts could be *very* short durations and I have this picture in my head of an oscilloscope being dragged behind a robot on a wheeled cart like hospital equipment. Just never let us mind how many channels you might need in a system like this!

Luckily a whole digital sampling oscilloscope can be implemented in an FPGA up to 100 million samples per second at 8 bits and plug into a USB port (depending on what and how you trigger, and your samples per period required, this is at best a 50MHz logic analyzer at Nyquist rate/frequency or maybe a 5-10MHz oscilloscope...I assume there you want to reconstruct a waveform with at least 10 points-20 points of sampling per cycle at minimum which means that you need more than the Nyquist rate/frequency which just eliminates aliasing). Luckily F.I.R.S.T. is starting to let us put laptops on the robot. However, that's still at least 2 pounds, probes, and some current measuring apparatus.

I have a better idea. Let's call it a brown out detector circuit. It's analog (so it's not subject to sampling issues...just integration and when in doubt there's some pretty darn fast operational amplifiers around these days). It'll not need to store all the data either, so the robot won't have to cart around something to fill up...just so you can analyze only what you see when you get it off there. Plus this way you'll get instant feedback. Too bad I don't have a source for small DVBST tubes (how an analog storage oscilloscope works).

When we're done with this, I'll put the information out there for everyone as well. I hope such a thing will make the whole issue of a power brown out as simple as looking at the LED or small LED backlit LCD. I can't see people forking over a few hundred bucks for a digital sampling oscilloscope and possibly 2x-5x more than that for a good inductive current probe. This will be far more economical and mobile.

[Mike Copioli]

Originally Posted by techhelpbb

I hope such a thing will make the whole issue of a power brown out as simple as looking at the LED or small LED backlit LCD. I can't see people forking over a few hundred bucks for a digital sampling oscilloscope and possibly 2x-5x more than that for a good inductive current probe. This will be far more economical and mobile.

This is a good idea, maybe it is something we could make. One thing to consider is the fact that brown out is device dependent and not the same for each piece of hardware. So what are you trying to detect brown out on comes to question. The CAN Jaguar class has a method that allows you to determine if power has been cycled since its last call, GetPowerCycled (). This may be something that you could use to rule out Jaguar Brown out. Since the FRC control system has several voltages, 5v, 12v, 12v boost, 24v boost it is possible for only one device to brown out based on its location, wiring, and individual power requirements.

Your problem does not sound like brown out to me. Having said that I can state the a Jag will go into a non-responsive state if you bridge CANH to CANL several times or at just the right time, only a power cycle will fix this problem. This suggests either a problem with the Jag firmware or an issue with the Jaguar CAN controller itself. I have also observed Jags becoming non-responsive after a brown out causing the entire CAN bus to crash. This should not happen.

We try our best to make CAN a high performance option for teams that choose to use it. We test our products vigorously to ensure that they meet the expectations of FIRST teams and the CAN spec. Sometime there are changes made to the system that force us to retest and and re-validate. Unfortunately there is a lack of transparency that we are subject to that makes validation difficult. This lack of transparency causes us to spend most of our time testing when we could be adding new features. In a perfect world we would have full access to all of the things that affect CAN, this is not the case. I am confidant in saying that these issues that teams are seeing are not caused by the 2CAN. I do hope that they are resolved and want teams, FIRST, NI and TI to know that we are willing to do what ever is required to help find solutions to these issues.

[kamocat]

What about measuring the thing that actually causes the brownout?
The 3.3v power supply on the Jaguar. What voltage does the processor brown out at? 2.7v?
You can use an interrupt on an analog trigger to wait until a brownout occurs.
(You can get the 3.3v off of the brake/coast pins. If you're using CAN, you can override that jumper anyways)