Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[Mike Copioli]

Originally Posted by kamocat

What about measuring the thing that actually causes the brownout?
The 3.3v power supply on the Jaguar. What voltage does the processor brown out at? 2.7v?
You can use an interrupt on an analog trigger to wait until a brownout occurs.
(You can get the 3.3v off of the brake/coast pins. If you're using CAN, you can override that jumper anyways)

The brown out on the Jag is around 4.8 volts based on the TPS54040 datasheet. The Jag uses a 5 volt buck in series with a 3.3 v regulator. This is the rated lockout for that regulator, most likely it is above 5 volts. Also the CAN controller and RS232 convertor both operate on 5V volts not 3.3v.

[The Lucas]

Originally Posted by Mike Copioli

This is a good idea, maybe it is something we could make. One thing to consider is the fact that brown out is device dependent and not the same for each piece of hardware. So what are you trying to detect brown out on comes to question. The CAN Jaguar class has a method that allows you to determine if power has been cycled since its last call, GetPowerCycled (). This may be something that you could use to rule out Jaguar Brown out.

In addition to GetPowerCycled(), would it be useful to call GetFaults() (available in C++) and check for kBusVoltageFault to determine Brown Out conditions. I haven't tried reading the CAN messages, but I think it is possible to enter this fault mode without dropping to a full power cycle voltage (I have seen this indicated with the red flashing LEDs across all Jags on previous robots). I don't know if this fault mode affects sensors and closed-loop control, but it is worth a look.

Mike, can the 2CAN run on low enough voltage to log or display errors like the the Bus Voltage fault via web interface? I unfortunately don't have a 2CAN (yet) to test these things.

[techhelpbb]

Eh...I'm somehow repeating myself?

Epic atomic transaction failure.

[techhelpbb]

Originally Posted by Mike Copioli

Since the FRC control system has several voltages, 5v, 12v, 12v boost, 24v boost it is possible for only one device to brown out based on its location, wiring, and individual power requirements.

The idea in this case would be to rig the comparator to have either various references, possibilities include:

1. A unity gain buffered non-forced voltage out D/A converter (most voltage out D/A converters are buffered somehow).
2. One of the various reference voltage components already available for various conversion systems (some of them can be 'in-system' calibrated).
3. A 'digital potentiometer' in a voltage divider configuration possibly with a unity gain buffer.

One can measure current using a shunt resistor, and I have dug out my collection of 2 milliOhm 30W shunts because using a inductive current probe is just a double problem. A standard Rogowski coil probe would be slow and limit your reaction time to about 3uS using most off the shelf components (and it assumes the hall effect sensor is capable of measuring DC saturation of the coil). So even if you did want to sample, you'd probably top out around 100,000 samples per second before you alias. Obviously a professionally designed oscilloscope current probe responds faster than this, but it's coil size is small and it's amplification circuits expensive and more difficult to calibrate. I can't see spending $400-$2,000 on this...possibly per device.

I was concerned about using the shunt in other posts because I wasn't convinced I could find a ready source of this wattage and value shunt resistors, but I found one. Lucky enough the stall current of each CIM isn't much higher. I just don't think telling people to trim their own shunts is going to work out unless they have a precision meter or bridge.

As to the placement, I figure the device has to be designed to be connected at the device that you think is 'browning' out. This includes the shunt resistor which should be on the load end.

In case it's not apparent where I'm going with this, as the concern here isn't purely passive circuit dynamics, I'm gearing up to create a way to check the phase relationship and magnitude of the voltage and current by setting the levels to where you consider anything beyond or below that a failure. Then latching that state. So the instant something slips out of the range of 'acceptable' you merely get an indicator that you have a problem. If you really wanted you could use a S/R flip flop on that configuration (like a 555 timer) and then use a digital device like a cheap PIC/AVR to do some QOS analysis. For example...how many times did you slip out of range....how long approximately was it out of range?

[techhelpbb]

Originally Posted by Mike Copioli

The brown out on the Jag is around 4.8 volts based on the TPS54040 datasheet. The Jag uses a 5 volt buck in series with a 3.3 v regulator. This is the rated lockout for that regulator, most likely it is above 5 volts. Also the CAN controller and RS232 convertor both operate on 5V volts not 3.3v.

Thanks you just saved me some time.

I've been looking at various ways to put a little extra capacitance on the 5V and 3.3V sides to increase the noise rejection.

I thought I saw this in the Jaguar schematic before, but I haven't had time to look.

[techhelpbb]

I should like to note for the record, that we do not use the 2CAN on the cRIO.

We are using the RS232/CAN bridge in the black Jaguar.

I just want to point this out because there could be yet more complications depending on how you get to the CAN bus.

[techhelpbb]

Ultimately, even with the high performance of the Jaguar's embedded controller I am concerned about using a digital measuring system to measure 'brown outs' because the noise spikes are short and they could easily alias the conversion process. If you get a good shot of noise into the circuit and it's very brief you might not even know it was there.

I've worked on some integrated circuit design and there are a few ways to go about the idea of an internal reset and brownout detector and sometimes the nice clean digital approach isn't a great idea around motors.

In other robotics designs...for example the RB5X (circa the 1980s) they used to use 2 batteries. One for the control circuitry and one for the drive motors. Specifically because this meant the noise couldn't walk back to the control logic as long as you had some isolation. They used 'logic level relays' to operate those drive motors, so there was no physical connection between the motor battery and the logic battery.

In this case, we don't get many choices with the Jaguars because they steal their logic power from the same terminals that feed the H-bridge.

BTW, in case anyone here never heard of the RB5X robot:
Meet a blast from the past and a certain recent Pawn Star TV show episode....
http://www.rbrobotics.com/
http://www.rbrobotics.com/Specs/rb5x_specs.htm

[Mike Copioli]

UUUUmmmmm Yaaaah......... or the Jag could just handle brown out better. Im just sayin.

[techhelpbb]

Originally Posted by Mike Copioli

UUUUmmmmm Yaaaah......... or the Jag could just handle brown out better. Im just sayin.

Next you'll want it to motion control or something.
Maybe differential encoder inputs?



Seriously, you'd think at 3.3V from a 12V+ source it could find the overhead to handle the noise.
RB5X has a 6V battery...for 5V logic...and while it moves obviously it discharges to even less.
It has *much* less room to play from the days before DC-DC converter modules where everywhere.

I'm quite sure that this can be made to work. However, the question remains what to tweak.
Hopefully, the rules won't eventually conflict with whatever needs to be done to clear this up.

[EricVanWyk]

Originally Posted by techhelpbb

The idea in this case would be to rig the comparator to have either various references, possibilities include:

1. A unity gain buffered non-forced voltage out D/A converter (most voltage out D/A converters are buffered somehow).
2. One of the various reference voltage components already available for various conversion systems (some of them can be 'in-system' calibrated).
3. A 'digital potentiometer' in a voltage divider configuration possibly with a unity gain buffer.

One can measure current using a shunt resistor, and I have dug out my collection of 2 milliOhm 30W shunts because using a inductive current probe is just a double problem. A standard Rogowski coil probe would be slow and limit your reaction time to about 3uS using most off the shelf components (and it assumes the hall effect sensor is capable of measuring DC saturation of the coil). So even if you did want to sample, you'd probably top out around 100,000 samples per second before you alias. Obviously a professionally designed oscilloscope current probe responds faster than this, but it's coil size is small and it's amplification circuits expensive and more difficult to calibrate. I can't see spending $400-$2,000 on this...possibly per device.

I was concerned about using the shunt in other posts because I wasn't convinced I could find a ready source of this wattage and value shunt resistors, but I found one. Lucky enough the stall current of each CIM isn't much higher. I just don't think telling people to trim their own shunts is going to work out unless they have a precision meter or bridge.

As to the placement, I figure the device has to be designed to be connected at the device that you think is 'browning' out. This includes the shunt resistor which should be on the load end.

In case it's not apparent where I'm going with this, as the concern here isn't purely passive circuit dynamics, I'm gearing up to create a way to check the phase relationship and magnitude of the voltage and current by setting the levels to where you consider anything beyond or below that a failure. Then latching that state. So the instant something slips out of the range of 'acceptable' you merely get an indicator that you have a problem. If you really wanted you could use a S/R flip flop on that configuration (like a 555 timer) and then use a digital device like a cheap PIC/AVR to do some QOS analysis. For example...how many times did you slip out of range....how long approximately was it out of range?

Please remember that this forum has a wide range of audience. It is cool that you know all these words, but clarity is more valuable. These posts are effectively a smoke screen to those without the electrical background to parse it.

Allow me to translate:
We could detect brownouts by measuring current and voltage.


And please excuse my bluntness, but the rest of it is entirely superfluous and ridiculous. Seriously? DVBST? Rogowski coils? Hand trimming shunts?

This is the PID show all over again. Inspire students. Don't try to impress them.

[techhelpbb]

Originally Posted by EricVanWyk

Please remember that this forum has a wide range of audience. It is cool that you know all these words, but clarity is more valuable. These posts are effectively a smoke screen to those without the electrical background to parse it.

Allow me to translate:
We could detect brownouts by measuring current and voltage.


And please excuse my bluntness, but the rest of it is entirely superfluous and ridiculous. Seriously? DVBST? Rogowski coils? Hand trimming shunts?

This is the PID show all over again. Inspire students. Don't try to impress them.

I don't need to impress anyone at all. It has no impact on me either way...what so ever.

1. There are lots of kinds of coils, and even field measuring coils, I'm being specific about which one.
2. There are other ways to measure current, I'm being specific about why this way.
3. I had stated concerns about why a shunt resistor to measure current with Ohm's law was a bad idea in this case before (availability), I am adjusting my view based on the available materials.
4. I was looking for an even more simplistic and cheap way to get to this goal of this brown out detector, and I had hoped to seek out specifics so I was seeking the limits of the problem. Hence why I considered how an analog storage oscilloscope is made (it's a DVBST tube).
5. I provided an alternate example of how others solved this problem, using a robot that was targeted specifically towards an educational setting.

Further, for anyone that actually wants to know what those terms mean...

http://en.wikipedia.org/wiki/Rogowski_coil

http://en.wikipedia.org/wiki/Oscilloscope

http://en.wikipedia.org/wiki/Direct-..._Storage_Tubes

http://http://www.reuk.co.uk/Make-a-Shunt-Resistor.htm

We need to solve the problems and I am trying my best to find a cost effective way to do that with the help of these mentors in the spirit of community contribution.

Thanks.

[gblake]

Originally Posted by techhelpbb

The idea in this case would be to rig the comparator to have either various references, possibilities include:
...

Techhelpbb and other folks - My EE degree is a bit dusty; but recently there seem to be a lot of uncertainty about possible problems (and solutions) in this thread and very few clear, specific questions/answers in the last 10-15 posts.

I think the entire conversation would be improved by a clear, clean, concise list of simple observable symptoms linked to proven root causes. There have been a lot of "maybe"s, "sometimes", "somebody told me"s, and possible test method posts (techhelpBB - I have to say some of those have been way into the weeds...); but there have lately been few "Here is the checklist you follow to fix problem X", or "Here is the checklist you follow to separate problem X from problem Y".

Who can cut through the fog, drain the swamp, and help that majority of students & mentors who aren't EEs, and who don't have the time (right now) to immerse themselves in the aracane minutia?

Maybe someone could start a new thread to discuss the physics of the COTS circuitry and possible home-brew test methods, and this one could stick to proven results?

Blake

[techhelpbb]

Originally Posted by gblake

Techhelpbb and other folks - My EE degree is a bit dusty; but recently there seem to be a lot of uncertainty about possible problems (and solutions) in this thread and very few clear, specific questions/answers in the last 10-15 posts.

I think the entire conversation would be improved by a clear, clean, concise list of simple observable symptoms linked to proven root causes. There have been a lot of "maybe"s, "sometimes", "somebody told me"s, and possible test method posts (techhelpBB - I have to say some of those have been way into the weeds...); but there have lately been few "Here is the checklist you follow to fix problem X", or "Here is the checklist you follow to separate problem X from problem Y".

Who can cut through the fog, drain the swamp, and help that majority of students & mentors who aren't EEs, and who don't have the time (right now) to immerse themselves in the aracane minutia?

Maybe someone could start a new thread to discuss the physics of the COTS circuitry and possible home-brew test methods, and this one could stick to proven results?

Blake

Fair enough, I'll summarize our situation...can't speak for the others.

The criteria for the problem we have:

1. We are not using the 2CAN, we are using all black Jaguars with one as the RS232 bridge. For what it's worth we are programming in JAVA.

2. We are using them in one of 2 modes as it relates to the wheels, which most exhibit the problem most often (we do occasionally have some issues with the arm that are similar...but our arm is highly geared down).

A. In one mode, we have >no< encoders. We drive completely off vbus and we see a problem occasionally where the Jaguars just time out. A quick reset and they generally return...and that's being general....when they don't we have to reboot.

B. In the other mode we drive drive 2 Jaguars both with a PID velocity setpoint and we have a circuit to isolate the encoders using an optocoupler and TTL logic. This circuit is the subject of another topic on this very forum:
http://www.chiefdelphi.com/forums/sh...t=89282&page=4

The circuit shown is the work of the other engineer...functionally it is equivalent to what we made. With the only obvious difference being that our circuit has a pull up on each encoder channel (for testing the circuit without either the Jaguar or the encoders mostly to check for wiring problems to either). Also our circuit uses the extra inverters to provide test points for an oscilloscope...consider it proof of concept.

Even with this circuit our robot will drive straight as an arrow, spin, dance and do everything it is supposed to...for only so long. Then we'll suddenly develop the very same CAN errors (every few minutes). In fairness I'll say we see the problem more often when we use the encoders...but...we do see the problem without them. I personally think we can attribute some of that to the simple fact that if the Jaguar malfunctions on the CAN bus whatever caused that could also cause it to loose track of the encoder. The other reference is a different matter.

3. It has been noted by myself and other mentors that the circuit in 2B above becomes necessary because of various sources of electrical noise. The wiring of the Jaguars is prone to a specific source of noise called a ground loop, especially when the grounds are connected as in a more simplistic encoder split (see other topic for more detail). This circuit eliminates this source of noise from the encoders as it isolates them from each other and even the encoder by using light. It also lets you put some test equipment on there to see what's going on...minus the software. We found that invaluable because we have a bunch of bad encoders from years past and it's hard...with so many possible issues...to find them all. With this circuit (as we made it) we can see on an oscilloscope when the encoders are not working properly and we can do that test while the wheels spin at any test RPM. I can assure you by test myself that the encoders are working properly.

4. The circuit in 2B does not do anything about any electrical noise issues on the CAN bus, it has nothing to do with it. Other mentors and myself have noted that placing a capacitance across the various sensor power supply terminals on the Jaguar (the only place we can legally get to the internally regulated power sources) improves the issues with the CAN bus errors moderately (not completely).

5. It has been suggested earlier in this topic that we were either overloading the Jaguars or they were browning out. We have no evidence of overloading the Jaguars in that we have not blown or tripped any current limiting devices (fuse/breakers). The robot doesn't seem to overly drain any properly operating battery we put on it. The Jaguars themselves, via BDC-COM do not indicate any major overloads. This leaves us with the possibility of a brown out condition. A brown out condition we would need to test for on a robot moving at high gear around a playing field and that does not appear on any test we've been able to reproduce while stationary.

6. I then proceeded to attempt to figure out how to test for this condition, in the most practical, least expensive, most definitive manner. I did so in detail as it would seem that we are not the only one to experience this problem. Furthermore, even if we didn't have this specific problem...we all should know by now that a low battery condition can cause issues on the robot and sometimes people don't notice it because batteries are often best tested under load.

7. If you use the Jaguars as you would the Victor 884 (no CAN, just PWM cables from the digital side car) you'd side step the issues. To put it as simply as possible, the way the Jaguar responds to PWM unless you really do overload the Jaguar, it'll mostly ignore major sources of non-repeating noise that might otherwise trip up the CAN bus. In this mode, you can still use the encoders on the cRIO if you want it, however, in this mode we've sort of given up on the CAN bus.

7. To date...the only options I've considered beyond those discussed are to put capacitors across the CIM motors to try to snub some of the noise from the brushes. Put capacitors on the Jaguar's sensor power supply wires to try and add additional filtering to the Jaguar's internal power supply. Removing the ground wires from the CAN cables has been suggested in the other topic.

8. Other people in this topic have reported an issue from boot up, which until recently we never experienced. In their failure mode they turn on the robot and basically fail to initialize the Jaguars on the CAN bus. It has been suggested that this is a timing issue and can be addressed in software. We have seen this issue a total of once.

Conclusion:
What I am looking for, like you, is a way to standardize the process and eliminate these painful problems from the system. They are *too technical* and expensive and effectively form a barrier to getting this working properly for anyone that doesn't have the knowledge/skills/time to make it work. Given the state of my physical health right now....I more than anyone...do not want to be making anything or wasting money looking for phantom problems.

The tangent above was merely my way of looking for an economical test for making sure that the power quality on the robot is not causing adverse problems. If we can determine that the power quality is the problem...then we can at least define a reasonable process to find the problems. Right now...I'm looking for something I can't test...because I'd have to chase the robot with test equipment in high gear.

It's possible there's simply something wrong with the CAN communications, but honestly, when other mentors have had to troubleshoot that issue, they've had to confront the manufacture of other circuits because otherwise they'd need to buy something for a logic analyzer to test it.

[DonRotolo]

I think you're looking too deep, the problem must be simpler. Like fixing a car, 99% of the time it's simple, just go back to basics.

Put a 12 volt light bulb across the Jaguar inputs, you'll (literally) see any brownout or breaker issues as you drive.

You say the wiring is good, how do you know? Explain your tests and their results.

In all your posts I see a lot of assumptions and third-hand information. To really solve this, we need first-hand data.

Thanks.

[techhelpbb]

Originally Posted by DonRotolo

I think you're looking too deep, the problem must be simpler. Like fixing a car, 99% of the time it's simple, just go back to basics.

Put a 12 volt light bulb across the Jaguar inputs, you'll (literally) see any brownout or breaker issues as you drive.

You say the wiring is good, how do you know? Explain your tests and their results.

In all your posts I see a lot of assumptions and third-hand information. To really solve this, we need first-hand data.

Thanks.

I'll find some light bulbs to put on it and see what there is to see with that.
If it were a breaker issue, I would expect it to trip and stay that way.

The wiring was tested with multimeter for resistance from the lugs that would go on the Jaguar back to the power board.

A multimeter was applied to the Jaguar input power while it was run off the ground no major issues were noted (but multimeters are fairly slow to measure).

When the robot was misbehaving I did put an oscilloscope on the power supply once it was stationary and I saw no major issues that would account for the fact that the Jaguars at that time were basically locked up.

When we first encountered the problem weeks ago we just reboot the robot a few times. However, that was slow and painful. So we adjusted the software to detect the event of the Jaguar error and reset. For the most part that improved the situation as long as you ignore that we're still getting time out errors.

I don't have my numbers with me tonight. I'll have to swing by and get them and we don't have a DSO or an oscilloscope camera to work with, so I have no waveforms for anyone to look at.