2015 Control System Alpha Testing

[Tom Line]

Originally Posted by Greg McKaskle

There was the one FRC situation I know of ... see attachments.

Of course in the grand scheme of things, it doesn't do much good when the cRIO simply gets mashed through the bottom of the robot and crate. The energy has to go somewhere. The cRIO may survive, but you still have to build it a new body.

I hope Rick doesn't mind me bringing back painful memories.
Greg McKaskle

So... Um... should we be durability testing the rRio for that particular circumstance? I've got a couple team members who have an incredible knack for destroying robot parts...

[AustinSchuh]

I'm curious how the rRio deals with a second high bandwidth USB device being used at the same time as the usb radio. Are both ports connected to the same root hub so that they share the bandwidth? Having written a USB driver for work, I have learned enough to be skeptical of USB.

I'm also curious what the bandwidth is to the various digital inputs and outputs. How fast can you actually read them by polling? I have seen other controllers (Chumby Botball Controller, I'm looking at you) get this very wrong. Hook up an encoder, and count the edges in polling code. See how fast you can spin it before loosing counts.

What rate can the controller actually read encoders? Edges/second. How many encoders can it support?

We used (before it lost us a regional) a feature in WPILib which let us read an encoder immediately after a hall effect had a positive edge by firing an interrupt. This was awesome since we could use it to figure out when a joint passed the homing position without having a polling delay. Does this work in the new version?

How is the timing jitter? +- ?% Am I going to have to work really hard to get my timing accurate to avoid control loop problems like on the cRIO? I can't count how many control loop problems ended up being timing jitter problems...

[Greg McKaskle]

Tom, the shipping company has graciously volunteered to do that testing.

Austin, I was going to send you a URL to the specs, but what I could find was pretty light. I'll try to answer them here.

USB bus can be overloaded, but so can ethernet. The effects when you do so are different, and you are correct in being skeptical. It will be tested in this situation.

As for other I/O rates -- the roboRIO is a similar architecture to the cRIO and all other NI RIO devices. RIO stands for Reconfigurable I/O, and that means an FPGA is user-programmable to handle the hard-real-time duties like building a decoder from some digital inputs. The I/O speeds of cRIO are determined by the modules, so the roboRIO digital speeds are actually far higher.

I don't have the details on count and speed, but it I believe it is the same number of encoders at about 10x the speed.

As for jitter of the RT code reading the encoder, that is something you control. VxWorks is a realtime OS and you should be able to measure and control jitter, but the base templates and frameworks in FRC avoid many of those features in order to keep the code clean. The RT-linux OS will offer many of the same features, but if you don't use them, you are trading simpler code for jitter. How much jitter are we talking about, and do you know what was causing it?

Greg McKaskle

[Mark McLeod]

The symptoms generated by the new system under low voltage conditions (momentary dips as well as a dead battery) would be of interest. The effect of low voltage brownout on a maxed out USB power output, various communications types, PWM/Relay/Digital outputs, voltage regulators, peripherals like the pneumatics breakouts, in addition to how the system handles staged shutdown. The shutdown response to a short on the different types of power pins (what doesn't get cut), overloading the power draw of the USB ports, that type of thing.

Ability of software to detect and report or act on these power outages.

How the system responds to the overload of time unconstrained loops and other common rookie programming mistakes - grind to a halt or critical operations take priority.

Easy to write a test program that exercises every I/O at once. Identify limits, e.g. max number of encoders supported, and what happens when you try to go past those limits. Test and verify the limits they've told you the new system has.

[Ether]

Originally Posted by Mark McLeod

...

^This. Spoken like a true veteran of realtime embedded programming

[rfolea]

Originally Posted by Greg McKaskle

There was the one FRC situation I know of ... see attachments.

Of course in the grand scheme of things, it doesn't do much good when the cRIO simply gets mashed through the bottom of the robot and crate. The energy has to go somewhere. The cRIO may survive, but you still have to build it a new body.

I hope Rick doesn't mind me bringing back painful memories.
Greg McKaskle

I was wondering if someone was gonna post those ... Back story - during shipping between events a pair of 500lb steel sprockets were dropped right through the shipping crate and onto our robot (you can see them in the picture on the robot after we removed the crate scraps) and the cRIO took the full brunt of the force (last picture). Other than a bent connector and smashed module, the cRIO still functioned fine (in fact I still use it today for demos, tutorials and CSA duties ...). Unreal ...

I dare anyone to try that with the roboRio ... any Alpha Tester volunteers?

[Jon Stratis]

Originally Posted by rfolea

I dare anyone to try that with the roboRio ... any Alpha Tester volunteers?

FIRST has told us that they'll be asking for the control system to be returned at some point... I doubt they'll like it very much if we return a pile of scraps!

[Alan Anderson]

Originally Posted by rfolea

...during shipping between events a pair of 500lb steel sprockets were dropped right through the shipping crate and onto our robot...

I've always meant to ask -- did the drayage company put an "overweight" sticker on [what was left of] your crate?

[rfolea]

Originally Posted by Alan Anderson

I've always meant to ask -- did the drayage company put an "overweight" sticker on [what was left of] your crate?

Actually, that's the funny part ... The Steel Sprockets smashed through the top of the crate - the rest of the crate was intact. It was a tall crate so the average person couldn't see down inside it, so it looked normal from the outside.

The only reason they knew something was wrong was they were missing two sprockets when they unloaded the truck and that the crate seemed to be a LOT heavier than when they put it on the truck!

[Team3266Spencer]

I would be curious to see how well it handles intensive vision processing.

[Tom Line]

Originally Posted by AustinSchuh

I'm curious how the rRio deals with a second high bandwidth USB device being used at the same time as the usb radio. Are both ports connected to the same root hub so that they share the bandwidth? Having written a USB driver for work, I have learned enough to be skeptical of USB.

I'm also curious what the bandwidth is to the various digital inputs and outputs. How fast can you actually read them by polling? I have seen other controllers (Chumby Botball Controller, I'm looking at you) get this very wrong. Hook up an encoder, and count the edges in polling code. See how fast you can spin it before loosing counts.

What rate can the controller actually read encoders? Edges/second. How many encoders can it support?

We used (before it lost us a regional) a feature in WPILib which let us read an encoder immediately after a hall effect had a positive edge by firing an interrupt. This was awesome since we could use it to figure out when a joint passed the homing position without having a polling delay. Does this work in the new version?

How is the timing jitter? +- ?% Am I going to have to work really hard to get my timing accurate to avoid control loop problems like on the cRIO? I can't count how many control loop problems ended up being timing jitter problems...

Austin, can you suggest a test that would determine if there are jitter problems?

I'd guess you're talking about high-speed velocity control, but give me specifics on what caused you trouble in the past.

[AustinSchuh]

Originally Posted by Tom Line

Austin, can you suggest a test that would determine if there are jitter problems?

I'd guess you're talking about high-speed velocity control, but give me specifics on what caused you trouble in the past.

Hi Tom,

Write a loop which sleeps for X msec, or runs code in a notifier at a specified frequency, and then reads the system clock and looks for how long it actually slept. Do this a large number of times and plot it. I would expect to see Gaussian noise of small magnitude if things are working right.

Yes, we had a problem with a high accuracy velocity loop. On the cRIO, I was seeing a 100 hz loop returning anywhere from 5 to 15 ms (I think it was worse, but I'll guess low). You could hear it in the loop. The state space observer made the assumption that the timestep was constant at 10 ms to simplify the math. The loop would throw the motor into full reverse occasionally on our flywheel to deal with the large apparent change in velocity. Our programmer dealt with this by registering a timer interrupt on vxworks so our code would get run every 10 system ticks, but that was a lot more work than it should have been. You would expect that "sleep" on a realtime system would sleep for the correct amount of time if the priority is right. 254 never got it working correctly with Java, and I had to detune the observer to deal with it.

Austin

[Greg McKaskle]

The link below digs into this a bit and shows the jitter of various timing mechanisms in LV. It is not safe to assume that all programming resources on a realtime OS are realtime.

http://team358.org/files/programming...Everything.PDF

Greg McKaskle

[marccenter]

I guess one really good way to test the hardware for potential issues is to beta test the hardware and software during the 2014 season. This is not directly related to the thread, sorry, but it would seem this would be the next logical step.

It would also seem if there are few perceived advantages given to a beta hardware/software test team over the existing cRIO2 system that this should be seriously considered. It also provides an excellent way for other teams to get a chance to see the system while at district and regional events.

[Jon Stratis]

Originally Posted by marccenter

I guess one really good way to test the hardware for potential issues is to beta test the hardware and software during the 2014 season. This is not directly related to the thread, sorry, but it would seem this would be the next logical step.

It would also seem if there are few perceived advantages given to a beta hardware/software test team over the existing cRIO2 system that this should be seriously considered. It also provides an excellent way for other teams to get a chance to see the system while at district and regional events.

That would certainly be a good test for the new control system, but there are a few problems with doing so:
- The new control system provides some significant enhancements, like increased processing power and USB host/device ports
- To the best of my knowledge the new control system has not been tested with the old FMS - there may be compatibility issues running it at an event
- The new control system could introduce bugs/issues that other teams wouldn't encounter, leaving those teams running it at a disadvantage in competition

That said, FIRST is doing a lot to test it. We've been asked to implement the new control system on our 2013 robot, and all the Alpha Test teams will be getting together in NH this fall for a weekend "event-like" testing. I imagine this testing would be similar to what they did for the communications issues last year, except obviously with a different focus. Regardless, we'll be running the control system on the FMS it's meant to be run on in a competition scenario to try and find issues.

We're also expected to bring the new control system with us to events January-April to help expose other teams to it. Since it won't be on our 2014 robot, it will be available for other teams to look at and ask questions about the entire competition, instead of being unavailable when the robot is on the field.