2015 Control System Alpha Testing

[Tom Line]

Originally Posted by Greg McKaskle

There was the one FRC situation I know of ... see attachments.

Of course in the grand scheme of things, it doesn't do much good when the cRIO simply gets mashed through the bottom of the robot and crate. The energy has to go somewhere. The cRIO may survive, but you still have to build it a new body.

I hope Rick doesn't mind me bringing back painful memories.
Greg McKaskle

So... Um... should we be durability testing the rRio for that particular circumstance? I've got a couple team members who have an incredible knack for destroying robot parts...

[AustinSchuh]

I'm curious how the rRio deals with a second high bandwidth USB device being used at the same time as the usb radio. Are both ports connected to the same root hub so that they share the bandwidth? Having written a USB driver for work, I have learned enough to be skeptical of USB.

I'm also curious what the bandwidth is to the various digital inputs and outputs. How fast can you actually read them by polling? I have seen other controllers (Chumby Botball Controller, I'm looking at you) get this very wrong. Hook up an encoder, and count the edges in polling code. See how fast you can spin it before loosing counts.

What rate can the controller actually read encoders? Edges/second. How many encoders can it support?

We used (before it lost us a regional) a feature in WPILib which let us read an encoder immediately after a hall effect had a positive edge by firing an interrupt. This was awesome since we could use it to figure out when a joint passed the homing position without having a polling delay. Does this work in the new version?

How is the timing jitter? +- ?% Am I going to have to work really hard to get my timing accurate to avoid control loop problems like on the cRIO? I can't count how many control loop problems ended up being timing jitter problems...

[Greg McKaskle]

Tom, the shipping company has graciously volunteered to do that testing.

Austin, I was going to send you a URL to the specs, but what I could find was pretty light. I'll try to answer them here.

USB bus can be overloaded, but so can ethernet. The effects when you do so are different, and you are correct in being skeptical. It will be tested in this situation.

As for other I/O rates -- the roboRIO is a similar architecture to the cRIO and all other NI RIO devices. RIO stands for Reconfigurable I/O, and that means an FPGA is user-programmable to handle the hard-real-time duties like building a decoder from some digital inputs. The I/O speeds of cRIO are determined by the modules, so the roboRIO digital speeds are actually far higher.

I don't have the details on count and speed, but it I believe it is the same number of encoders at about 10x the speed.

As for jitter of the RT code reading the encoder, that is something you control. VxWorks is a realtime OS and you should be able to measure and control jitter, but the base templates and frameworks in FRC avoid many of those features in order to keep the code clean. The RT-linux OS will offer many of the same features, but if you don't use them, you are trading simpler code for jitter. How much jitter are we talking about, and do you know what was causing it?

Greg McKaskle

[Mark McLeod]

The symptoms generated by the new system under low voltage conditions (momentary dips as well as a dead battery) would be of interest. The effect of low voltage brownout on a maxed out USB power output, various communications types, PWM/Relay/Digital outputs, voltage regulators, peripherals like the pneumatics breakouts, in addition to how the system handles staged shutdown. The shutdown response to a short on the different types of power pins (what doesn't get cut), overloading the power draw of the USB ports, that type of thing.

Ability of software to detect and report or act on these power outages.

How the system responds to the overload of time unconstrained loops and other common rookie programming mistakes - grind to a halt or critical operations take priority.

Easy to write a test program that exercises every I/O at once. Identify limits, e.g. max number of encoders supported, and what happens when you try to go past those limits. Test and verify the limits they've told you the new system has.

[Ether]

Originally Posted by Mark McLeod

...

^This. Spoken like a true veteran of realtime embedded programming

[Michael Hill]

Originally Posted by Mark McLeod

The symptoms generated by the new system under low voltage conditions (momentary dips as well as a dead battery) would be of interest. The effect of low voltage brownout on a maxed out USB power output, various communications types, PWM/Relay/Digital outputs, voltage regulators, peripherals like the pneumatics breakouts, in addition to how the system handles staged shutdown. The shutdown response to a short on the different types of power pins (what doesn't get cut), overloading the power draw of the USB ports, that type of thing.

Ability of software to detect and report or act on these power outages.

How the system responds to the overload of time unconstrained loops and other common rookie programming mistakes - grind to a halt or critical operations take priority.

Easy to write a test program that exercises every I/O at once. Identify limits, e.g. max number of encoders supported, and what happens when you try to go past those limits. Test and verify the limits they've told you the new system has.

I'll third this. Also, what is the voltage drop-out? Also, do we have access to the FPGA at all?

[Tom Line]

Originally Posted by AustinSchuh

I'm curious how the rRio deals with a second high bandwidth USB device being used at the same time as the usb radio. Are both ports connected to the same root hub so that they share the bandwidth? Having written a USB driver for work, I have learned enough to be skeptical of USB.

I'm also curious what the bandwidth is to the various digital inputs and outputs. How fast can you actually read them by polling? I have seen other controllers (Chumby Botball Controller, I'm looking at you) get this very wrong. Hook up an encoder, and count the edges in polling code. See how fast you can spin it before loosing counts.

What rate can the controller actually read encoders? Edges/second. How many encoders can it support?

We used (before it lost us a regional) a feature in WPILib which let us read an encoder immediately after a hall effect had a positive edge by firing an interrupt. This was awesome since we could use it to figure out when a joint passed the homing position without having a polling delay. Does this work in the new version?

How is the timing jitter? +- ?% Am I going to have to work really hard to get my timing accurate to avoid control loop problems like on the cRIO? I can't count how many control loop problems ended up being timing jitter problems...

Austin, can you suggest a test that would determine if there are jitter problems?

I'd guess you're talking about high-speed velocity control, but give me specifics on what caused you trouble in the past.

[AustinSchuh]

Originally Posted by Tom Line

Austin, can you suggest a test that would determine if there are jitter problems?

I'd guess you're talking about high-speed velocity control, but give me specifics on what caused you trouble in the past.

Hi Tom,

Write a loop which sleeps for X msec, or runs code in a notifier at a specified frequency, and then reads the system clock and looks for how long it actually slept. Do this a large number of times and plot it. I would expect to see Gaussian noise of small magnitude if things are working right.

Yes, we had a problem with a high accuracy velocity loop. On the cRIO, I was seeing a 100 hz loop returning anywhere from 5 to 15 ms (I think it was worse, but I'll guess low). You could hear it in the loop. The state space observer made the assumption that the timestep was constant at 10 ms to simplify the math. The loop would throw the motor into full reverse occasionally on our flywheel to deal with the large apparent change in velocity. Our programmer dealt with this by registering a timer interrupt on vxworks so our code would get run every 10 system ticks, but that was a lot more work than it should have been. You would expect that "sleep" on a realtime system would sleep for the correct amount of time if the priority is right. 254 never got it working correctly with Java, and I had to detune the observer to deal with it.

Austin

[Greg McKaskle]

The link below digs into this a bit and shows the jitter of various timing mechanisms in LV. It is not safe to assume that all programming resources on a realtime OS are realtime.

http://team358.org/files/programming...Everything.PDF

Greg McKaskle

[marccenter]

I guess one really good way to test the hardware for potential issues is to beta test the hardware and software during the 2014 season. This is not directly related to the thread, sorry, but it would seem this would be the next logical step.

It would also seem if there are few perceived advantages given to a beta hardware/software test team over the existing cRIO2 system that this should be seriously considered. It also provides an excellent way for other teams to get a chance to see the system while at district and regional events.

[Jon Stratis]

Originally Posted by marccenter

I guess one really good way to test the hardware for potential issues is to beta test the hardware and software during the 2014 season. This is not directly related to the thread, sorry, but it would seem this would be the next logical step.

It would also seem if there are few perceived advantages given to a beta hardware/software test team over the existing cRIO2 system that this should be seriously considered. It also provides an excellent way for other teams to get a chance to see the system while at district and regional events.

That would certainly be a good test for the new control system, but there are a few problems with doing so:
- The new control system provides some significant enhancements, like increased processing power and USB host/device ports
- To the best of my knowledge the new control system has not been tested with the old FMS - there may be compatibility issues running it at an event
- The new control system could introduce bugs/issues that other teams wouldn't encounter, leaving those teams running it at a disadvantage in competition

That said, FIRST is doing a lot to test it. We've been asked to implement the new control system on our 2013 robot, and all the Alpha Test teams will be getting together in NH this fall for a weekend "event-like" testing. I imagine this testing would be similar to what they did for the communications issues last year, except obviously with a different focus. Regardless, we'll be running the control system on the FMS it's meant to be run on in a competition scenario to try and find issues.

We're also expected to bring the new control system with us to events January-April to help expose other teams to it. Since it won't be on our 2014 robot, it will be available for other teams to look at and ask questions about the entire competition, instead of being unavailable when the robot is on the field.

[Ether]

Originally Posted by marccenter

It would also seem if there are few perceived advantages given to a beta hardware/software test team over the existing cRIO2 system that this should be seriously considered.

What kind of serious consideration did you have in mind?

[Tom Line]

Originally Posted by Greg McKaskle

The link below digs into this a bit and shows the jitter of various timing mechanisms in LV. It is not safe to assume that all programming resources on a realtime OS are realtime.

http://team358.org/files/programming...Everything.PDF

Greg McKaskle

I have an email in to Mark asking for the LabVIEW code he utilized for those measurements. With any luck he'll have it laying around and we'll be able to test the new system in the same manner. Lacking that, we'll write a new one and test the cRIO and rRIO.

[Ether]

Originally Posted by Greg McKaskle

The link below digs into this a bit and shows the jitter of various timing mechanisms in LV. It is not safe to assume that all programming resources on a realtime OS are realtime.

http://team358.org/files/programming...Everything.PDF

Greg McKaskle

FWIW. I'm linking this post because it mentions Timed loops in the section about multi cores.

http://www.chiefdelphi.com/forums/sh...&postcount=172

Follow the thread for jhersh's response.

[Jon Stratis]

For those who are wondering, we had an un-boxing today!