[DFTF] We eat what we CAN, and what we can't, we CAN...

[Joe Johnson]

This is part of a series of posts called Drinking From The Firehose on getting Dr Joe back up to speed on All Things FIRST.

In particular, I just posted a question on Jags v. Vics

Today's topic:
We eat what we CAN, and what we can't, we CAN...

Yeah, I know the subject makes no sense, but there is that old joke I love about the farmer's answer to his city relative's question, "What do you do with all those tomatoes?"

The REAL topic is Jags and CAN.

Here is what I have heard from sources usually deemed reliable:

First, one danger of using CAN there are some cases where the CAN message blocks (i.e. halts) the code execution of the cRio. Threads can be used to help to contain the impact of the problem but it doesn't fully solve the problem. There are a bunch of smart folks working to try to fix this, but the end result is still uncertain.

Can anyone comment on this?

Second, if you get more than 3 Jaguars on the CAN bus, you get to the point where you can send out to the speed controller faster via PWM. From what I hear, you can send out all PWM signals at a rate of 200hz (5msec between updates) via PWMs. Note this is faster than your joystick data is coming over from the robot which I understand is 50hz (20msec).

On this topic, I am mixed. Yes there are times (in some special cases during autonomous mode, implementing a fast PID loop where sample times are critical, etc.) where 5msec update are an important feature, I can't see this being very often.

AND, if I am going to implement CAN, I am going to use PID on all but the most simple motor application, in which case, there is certainly no reason to update the Jaguars 200 times per second.

Does anyone have data on what the PID update rate is?

On another topic, I have a notion that I want to have my drive motors in PID mode but closing the loop not on position (via a pot input or via the encoders) but closing the loop the way motor control gurus have done for years, a fast loop based on velocity (in the Jaguar, hopefully) and a slow loop based on position.

There are several problems with this idea, the first of which is that the Jaguars do not support this mode.

I think it is worthy of a new thread...

All for now.

Joe J.

[Al Skierkiewicz]

Joe,
I can't speak to some of your post but the confusion may come from the need to control motor function via field controls. For this reason, CAN implementation requires some interaction so that E-stop, match start and stop, and automode commands take precedence over anything that might be commanded via the CAN bus. For safety, all robots must be able to be stopped when the filed controller or the personnel running the event (FTA) deems it necessary.

[Jared Russell]

To echo Joe's sentiments, I have been wondering for a while exactly what the PID implementation within the Jag looks like. Not just update rate, but also the form of their gains (there are dozens of perfectly valid ways of writing the PID equation, but they all do different things to the gains) as well as what anti-windup techniques and derivative term filtering techniques (if any) are in place. Does anyone have a document that provides this information?

[Ether]

Originally Posted by Joe Johnson

Does anyone have data on what the PID update rate is?

I believe the Jag's PID update rate is 1000Hz.

I'll dig up a source.

[Alan Anderson]

Originally Posted by Jared341

To echo Joe's sentiments, I have been wondering for a while exactly what the PID implementation within the Jag looks like...Does anyone have a document that provides this information?

The Jaguar source code (minus FRC-specific safety features) should be available for download from TI.

[Hugh Meyer]

Joe,

Your summary of the “Jag” situation is dead on. This is based on my experience from using CAN since FIRST allowed us to use it.

One aspect that you didn’t mention is the interface to the CRIO. There are two primary interfaces, the serial port on the CRIO which uses a serial converter within the jaguar to convert to CAN and the 2CAN device which is a bridge between Ethernet and CAN.

When we started out using CAN I said we would try the serial interface, mainly to save money. We like to log lots of data, so we were attempting to transfer lots of data both directions using 8 Jaguars. It wasn’t very long until we discovered that the system couldn’t keep up. Switching to the 2CAN increased throughput by at least a factor of 10. Even with the 2CAN we could still take longer to transfer what we wanted than our periodic loop in the main code would support.

As a work around we changed our periodic loop to 100 msec and multiplexed the logging data. As I recall we divided up the requests for data from the Jaguars into 2. One pass through our main loop we request data from half of the Jaguars, and the next time through we get the other half. This seems to work good enough. But we must keep a close eye on the delays to be sure the code run time does not take longer than the periodic loop.

We monitor the code run time by setting a digital output pin high at the start of our loop, setting it low at the end of the loop and a second pin is a toggle at the start of the loop. We monitor these signals with an oscilloscope to be sure the timing is what we expect.

The code does indeed block and waits for the communication to complete each cycle. This wastes most of the time just waiting. Based on our testing it always does this.

I think a simpler system where commands are sent without waiting for an ACK would be fine. This is done all the time with DMX control systems in the theatrical lighting industry and it works fine. That way we could send out data in sequence to all the Jaguars, then by the time we did that, the first one would be ready to respond again. I am hoping we see a change in this direction, but that is beyond our control.

This year our team is discussing using the Jaguar PID to close loop on speed, then use a slow loop around that for position with the CRIO. I think that would work well. Another approach we may try this year is to go back to PWM and make a separate thread for each loop in the CRIO. I really like the CAN and want to stick with it. The data we get back from the Jaguars is priceless to knowing what is going on.

As Allan mentioned the source code for the Jaguars is the way to find out what the PID implementation is doing.

I hope this has been clear. If not I will gladly expand if you have specific questions.

-Hugh

[Ether]

Originally Posted by Hugh Meyer

I hope this has been clear. If not I will gladly expand if you have specific questions.

I have a couple questions Hugh:

We monitor the code run time by setting a digital output pin high at the start of our loop, setting it low at the end of the loop and a second pin is a toggle at the start of the loop. We monitor these signals with an oscilloscope to be sure the timing is what we expect.

The set hi/low is great way to observe the timing. What's the toggle for? To test for re-entrancy?

The code does indeed block and waits for the communication to complete each cycle. This wastes most of the time just waiting. Based on our testing it always does this.

You said block waiting. Did you mean busy waiting?

[Hugh Meyer]

Ether,

The toggle is to measure the periodic rate and trigger the scope. The Hi/Low measures our total code run time.

We call a function to send data. It does not return until the data is sent and the ACK comes back from the CAN bus. I call that blocking because the thread is stopped from running while waiting for the function to return.

-Hugh

[Ether]

The toggle is to measure the periodic rate and trigger the scope. The Hi/Low measures our total code run time.

Bear with me? I'm still not clear on this. You said "We monitor the code run time by setting a digital output pin high at the start of our loop, setting it low at the end of the loop and a second pin is a toggle at the start of the loop." (emphasis mine).

It sounds like you are setting one pin high "at the start of our loop" and you are toggling a second pin at the start of the same loop. Am I reading that correctly? If so, couldn't you trigger the scope on the rising edge of pin1, measure the periodic rate as the distance between pin1 rising edges, and measure the loop run time as the distance between pin1 rising and falling edges?

We call a function to send data. It does not return until the data is sent and the ACK comes back from the CAN bus. I call that blocking because the thread is stopped from running while waiting for the function to return.

In the blocked waiting state, the thread releases the CPU so other threads can run. So if it's blocked waiting, would it be possible to put the CAN data gathering in a separate 200ms thread so you wouldn't have to multiplex it?

[Hugh Meyer]

Originally Posted by Ether

It sounds like you are setting one pin high "at the start of our loop" and you are toggling a second pin at the start of the same loop. Am I reading that correctly? If so, couldn't you trigger the scope on the rising edge of pin1, measure the periodic rate as the distance between pin1 rising edges, and measure the loop run time as the distance between pin1 rising and falling edges?

Yes, you are correct. That is what we are doing. Yes, the periodic rate could be measured that way, when all is well and working correctly. We did this 3 years ago when this was all very new. Our periodic rate was set much faster and we were attempting to transfer much more data. So, our code run time was longer than the periodic rate. We were in trouble shooting mode. When we finally realized the code was taking longer than the period things started to make sense. I just like the redundancy of measuring them independently so we still do it that way. Seeing both signals on the scope is a good visual confidence monitor.

Originally Posted by Ether

In the blocked waiting state, the thread releases the CPU so other threads can run. So if it's blocked waiting, would it be possible to put the CAN data gathering in a separate 200ms thread so you wouldn't have to multiplex it?

We are planning to try a scheme like that this year.

-Hugh

[Joe Johnson]

Originally Posted by Hugh Meyer

<snip> 2CAN device which is a bridge between Ethernet and CAN.

<snip> try the serial interface, mainly to save money. <snip> we discovered that the system couldn’t keep up. Switching to the 2CAN increased throughput by at least a factor of 10. Even with the 2CAN we could still take longer to transfer what we wanted than our periodic loop in the main code would support.

As a work around we changed our periodic loop to 100 msec and multiplexed the logging data. As I recall we divided up the requests for data from the Jaguars into 2. One pass through our main loop we request data from half of the Jaguars, and the next time through we get the other half.

<snip>
-Hugh

This is from a few days ago, but now I finally have enough clarity of thought to ask questions about it.

2CAN:
$200 from AndyMark. I am not saying it is overpriced, but it is a lot of money. Is this the device of choice? Are there other possible choices?

100msec loop time:
Is that you MAIN loop time? 10Hz seems pretty slow to me. At 10fps the robot covers 1ft per loop. That seems a bit too slow for my blood. I suppose that if you had your Jaguars all in feedback mode, the robot behaved fine but what did your drivers think? I think it may be okay because my main motivation in this is not to make high speed better behaved but to make fine motions at low speeds better behaved.

I may even consider modifying the gains of my PID loop based on the robot configuation -- the robot in a compact position may need different gains from the robot with its arm reaching over some field obstacle -- it would be great if the driver didn't see a difference in behavior.

Data logging:
What sort of data were you logging and where we you logging it?

This part of the 2CAN documentation seems interesting:

The 2CAN also provides a web host dashboard that reports
information on all of the Jaguars on the CAN bus. Information
such as voltage, current, temperature, faults, etc… are displayed
in an easy to read webpage accessible to any laptop connected to
the Ethernet bus. This allows teams to perform quick diagnostics
on their motor controllers and also debug hardware issues

If this is true, then I think that the logging could all be done back on the Win7 PC via some Java code that scrapes the data off the 2CAN's web host dashboard.

Did you try this?

I don't know if it is true but I have heard that there is some magic (magic in fast and easy to implement) method to get data between the cRio and the Win7PC. If so, then there is yet another twist in the plot to consider.

More questions but my message is too long already.

Joe J.

[Hugh Meyer]

Joe,

2CAN:
Last year the 2CAN and serial port were the two options. NI must support them. This year may be different.

Loop Time:
100 msec is fine. We would like faster, but it is a trade off between CAN communication time and periodic rate. Our drivers couldn't perceive much difference between 100 msec and 50 msec. Fastest Windoze can do is 35 msec anyway. Anything faster is just an illusion.

Data Logging:
We log just about everything we can think of. Attached is a typical log file from a match during autonomous mode. We send data back in the user packet. It is 934 bytes (or something like this), which allows for lots of data. We have yet for that to be a limit. This data shows up in the dashboard. That lets us view on the dashboard in real time and write it to disk for later review.

2CAN Web Page:
The 2CAN will show much data if you hit it with a web browser. We could not do that and control the Jaguars from the CRIO at the same time. It is one or the other. Also, the configuration to get to connect the web interface to the 2CAN is not a legal mode, since you must have a path directly from the browser to the 2CAN. Normally the 2CAN must be connected to the CRIO for normal robot operation. This year may be different, since the 2CAN will be connected to the switch.

PID with two CIMS:
Last year I made a significant effort to make this work by connecting one encoder back to two Jaguars. We solved the ground loop issues with optical isolators. Every once and a while for some reason we would see the jags driving opposite directions. As best I could tell it was from the backlash in the gears. I used lower line count encoder wheels to reduce the ability of the system to see the backlash, which helped, but it didn't solve the problem. This caused us problems at Purdue, so we abandoned the effort and changed to voltage mode. The drivers said it was much more responsive when it worked. There are several threads where I talk about what we did last year with much input from others. Seems the only way to do this is to make the loop in the CRIO and just drive the two jags with PWM.

Fast Reversals on Jags:
This has been mentioned several times. During testing last year I discovered that the jags would reset when a fast reversal was attempted. Connecting a larger capacitor on the power supply seems to help this problem. We did this by installing it on the encoder pins in the jag. If you review the schematic you will see there is not much filtering on the 3.3 volts. Again, there are threads from last year where this is discussed.

-Hugh

[Ether]

Originally Posted by Hugh Meyer

Fastest Windoze can do is 35 msec anyway. Anything faster is just an illusion.

I'm skeptical about that Hugh. Where did you get that info?

[Greg McKaskle]

The link shown below are actual DS timings as measured by Marshal Horn. I believe the statement about Windows timing is referring to the typical 55ms resolution which is a holdover from DOS and IBM compatible computers. This is improved by using the multimedia timers and setting it to a better resolution. LV has been doing this since '95 or so.

Greg Mckaskle

Link

[Hugh Meyer]

Ether and Greg,

I thought this statement might be questioned… :-)

Thanks Greg for the link. I would be interested to know more about how Marshal measured these. I am surprised to see so many points down in the 20 msec range, but notice there are several points in the 60 msec range, which is more where I would have expected. I would argue that the computer does in fact have a significant impact on performance.

Greg, if we are running C++ is it still using the multi media timers? Which ones are being used? Can you tell us the exact Windows system calls being used?

Below are some addition thoughts on this issue.

Windows is not a real time OS, so nothing is guaranteed in the time domain. Context switching is the heart of the issue. Windows has all kinds of processes running that do all sorts of functions. In the case of the FRC robot code getting the joystick data and dumping it out the network port is just another process that runs when it can.

Ultimately the fastest it can go depends on the number of processes running, the CPU speed, and the number of CPUs present. It is also related to “quantums” as defined in the windows world and what that is set to. It varies for different versions of windows.

Generally speaking the kb 259025 article refers to a quantum as 10 or 15 msec. Threads get 3 or 9 quantums. So that puts the thread run time up in the 30 - 135 msec range.

A glance at task manager will show many processes. While not all of them are ready to run, it doesn’t take many to get the time between “runs” to be longer than 50 msec.

So while a thread is running the service time could be shorter, when a context switch occurs the process is starved while every other process gets some time. During this pause the joystick data is just on hold, not really moving out to the network.

Another way to think about this is to imagine a snippet of software code that generates a periodic wave out a port. All will be fine, when the process is running, but as soon as a context switch occurs, the signal rate will stall until the process is switched back to run.

Here are some links with more information:

Context Switches
http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx
http://technet.microsoft.com/en-us/l.../cc938613.aspx

http://support.microsoft.com/kb/259025
Quantums 10 or 15 msec divided by all the processes running would not support very high resolution.

http://stackoverflow.com/questions/2...hes-on-windows
How long is the time frame between context switches on Windows?

-Hugh