NI releasing/designing new controller for FRC

[apples000]

Originally Posted by apalrd

The only reason it's so darn hard to do stuff now is because of inefficiency in the current system - The WPIlib in LabVIEW is so unoptimized that it's nearly impossible to run code that uses IO calls faster than 20ms task time without saturating the CPU.

The fact that we're currently saturating a 400mhz PowerPC amazes me. I don't like the idea that we should just throw more power at it to deal with it, since there's no reason to need anything near a 400mhz PowerPC.

In my opinion, the new control system should be less powerful than the current cRIO setup. I've looked at the great robots from 2008 and before, and they don't really lack anything that we have today. The CMUcam wasn't as great as the current axis cam, but the only successful implementations of vision that I know of don't use the cRIO. In Java, the libraries are a little better than LV, but we still see high processor utilization when barely running anything. Having a slower and less powerful control system would force teams to come up with solutions that aren't completely inefficient (some of the code I see helping at competitions is amazingly inefficient) which would cause teams to come up with innovative control solutions.

Also, the current system is WAY overkill. The FPGA is a much higher-end model than what is needed, and there is no reason why the sidecar, digital module, analog module, the SSR module, the analog breakout, the pneumatics breakout, and the radio could not be integrated into one device. A good control system would have one enclosure for everything but power distribution. It would be MUCH cheaper than a $2,000 cRIO replacement.

[Ricky Q.]

Originally Posted by coalhot

Also, I think I worded the second part of my comment badly. Here goes: I really want to see what IFI and other companies came up with. Some of them may not ever release what their idea was because they got turned down for this contract: that would be a shame.

IFI / VEX did not submit a bid for the control system as a whole, we opted to focus on the motor controller portion of the RFP instead.

Best,
Ricky

[dtengineering]

Originally Posted by Michael Hill

Except for that whole image processing thing...or all of the libraries that are needed. Arduinos don't have THAT much memory to put robust enough code on.

Think of all the stuff WPILib has in it. None of that would be available just because of memory restrictions.

<Chuckling> Yes... you're correct. It was, actually, impossible to have a meaningful FRC robot before 2009.

While the old IFI controller did have more memory and power than an Arduino Uno, it pales in comparison to what an Arduino Due can do, eh?

While there are a few teams out there that will use every clock cycle that they are given, there are far, FAR more teams out there who are struggling to figure out how to get a limit switch to stop their arm from destroying itself, or how to get their robot to move forward for three seconds and stop in autonomous. "Simple and supported" is likely to benefit more teams than "sophisticated and speedy".

And I'll also suggest that just as we have limits on motor power that force us to design elegant mechanical systems, it may be that meaningful limits on processing power would force teams to design more elegant software systems.

Jason

[apalrd]

Originally Posted by dtengineering

While there are a few teams out there that will use every clock cycle that they are given, there are far, FAR more teams out there who are struggling to figure out how to get a limit switch to stop their arm from destroying itself, or how to get their robot to move forward for three seconds and stop in autonomous. "Simple and supported" is likely to benefit more teams than "sophisticated and speedy".

EXACTLY!

I've noticed over the years that the authors of the WPIlib seem to continuously pile on features with no concern for library cohesiveness or efficiency, while there are still issues (e.g. the execution cost of writing a motor value or especially a relay value) in the core IO access. We really don't need more features, we really need something that works reliably within the constraints of the 400mhz processor. This dosen't even include all the CAN issues, which I'm sure you've all heard me rant about.

I did some testing a few days before kickoff 2013 and found that the DEFAULT CODE from 2012 (without Network Tables) ran at about 40% CPU utilization on the 4-slot cRio (Back in 2011 I measured the default code to be about 65% CPU utilization on the 8-slot), running a single task that runs at something around 25ms iteration time (nowhere near consistent) and does nothing but set two motors to the values of two joysticks. By comparison, I got around 20-25% utilization running an early PalLib in a 10ms RT task with <20us average jitter, reading and writing an entire analog and digital card of IO. The processor is capable of far more than is possible due to pure library inefficiency.

Some numbers for efficiency comparison: Our 2012 code ran at ~80% utilization running a 10ms RT task for gun speed control only and ~22ms non-RT task for everything else, while our 2013 code was able to run in a single 10ms RT task with extremely solid timing. Our 2012 code had a LOT of WPIlib mods to improve efficiency to get it to run at all (mid-build season we hit 100% continuous loading before we even merged in about half of the code), including a Set Motor Simple VI which we released on CD. Our 2013 code never encountered any issues using a totally new library, in fact we were under 50% CPU load for almost all of build season.

I talked to a friend of mine who is a programmer on another local team, and they struggled to get their (relatively simple) code to run under 100% CPU utilization, while getting the arm PID controller to run as fast as possible (they eventually got to 15ms by pushing some other tasks as slow as 100ms). 10hz control should never be considered an acceptable solution on a 400mhz system.

I also worked with several other teams with electronics or software issues during various events, and I was amazed how s l o w the compile/download process STILL is, it's now quite a few minutes. I can do a full build of Chrysler PCM software (1.85 million lines, 1.3million of code) in under 20 minutes on my laptop and flash in under a minute on Nexus or ETK. If it takes 5 minutes to compile a team project with ~15 team VI's and another few minutes to download, on 100mbit Ethernet, we've got a SERIOUS problem.

I see it all happening again with the new Athena controller. I really shouldn't say a lot about it, but IMHO NI/Athena team really don't know what's important to the vast majority of teams, and they keep focusing on the expansion possibilities that <5% of teams will think about using. CTRE gets it though, their solutions are fantastic, simple, efficient, and light.

[Michael Hill]

Originally Posted by dtengineering

<Chuckling> Yes... you're correct. It was, actually, impossible to have a meaningful FRC robot before 2009.

While the old IFI controller did have more memory and power than an Arduino Uno, it pales in comparison to what an Arduino Due can do, eh?

While there are a few teams out there that will use every clock cycle that they are given, there are far, FAR more teams out there who are struggling to figure out how to get a limit switch to stop their arm from destroying itself, or how to get their robot to move forward for three seconds and stop in autonomous. "Simple and supported" is likely to benefit more teams than "sophisticated and speedy".

And I'll also suggest that just as we have limits on motor power that force us to design elegant mechanical systems, it may be that meaningful limits on processing power would force teams to design more elegant software systems.

Jason

In reality, they already do by limiting the motors we can use

[crake]

Originally Posted by dtengineering

"Simple and supported" is likely to benefit more teams than "sophisticated and speedy".

Genius of the AND: simple and supported and sophisticated and speedy.

choosing between seemingly contradictory concepts—focusing on this or that—leads to missed opportunities

[apalrd]

Originally Posted by crake

Genius of the AND: simple and supported and sophisticated and speedy.

choosing between seemingly contradictory concepts—focusing on this or that—leads to missed opportunities

If the current cRio system is any example, choices have to be made to retain the Simple and Supported requirement. As much as we want everything, the cRio system clearly isn't anywhere close.

The VAST majority of teams want to be able to drive their robot and actuate their mechanisms with joysticks or buttons, and possibly do something in autonomous. For those teams, the current control system has a LOT of setup and puzzle pieces to fit together and configure separately, THEN they have to write code to make it do anything. I would estimate that at least half to two thirds of all FRC teams are in this place, maybe adding a limit switch or two. These are the teams that benefit most from any control system improvements.

The next class of teams uses sensors and feedback controls in some way. These teams want to be able to connect their analog potentiometers and quadrature encoders easily, read them easily, and execute their code. The current LV environment makes no attempt to maintain any sort of timing determinism, which makes basic example control loops including calculus terms hard to deal with. These teams spend a lot of time fighting this, and most LV teams in this category will also hit 100% CPU utilization trying to run their feedback controllers at a moderate speed using the 2013 libraries. I've talked to MANY teams and programming leaders who asked for advice on code optimization, trying to get their code to run at all, let alone in a reasonable execution time with reasonable determinism.

There is also the <1% of teams who design custom circuits (other than COTS computing devices) and complain about how hard the cRio is to interface to, because they want higher-speed SPI or LIN or some other protocol which it doesn't support. These are NOT the teams we should be focusing on, because we still haven't met the needs of the 99% (or come anywhere close). In fact, we were closer in 2010 than we are now - Code compile/download times and CPU hogging 'bonus' library features have gone up significantly since the cRio was released, and in many ways the usability has gone DOWN.

Teams REALLY want a controller that just works. They want to be able to hook it up and drive their robot without doing too much electrical and software work, configuring a whole bunch of separate devices using separate tools and instructions, and writing code. Anything else is secondary to this goal.


Speaking of this, why is there no default code for this control system like IFI and Vex provide? It's a HUGE help to Vex teams to be able to just drive and test things without writing code.

[Greg McKaskle]

More details about Athena will be made available in about seven weeks. I'll respond to just a few of the recent rants here, but this isn't the place for the details.

CPU usage:
I'm out of town attending a wedding, so I don't have a cRIO with me, but I do not believe that your measurements of the default system are accurate. On my computer, I have 50,000 log files from numerous teams acquired during the 2013 season. I do not see a rampant CPU usage problem in that data. Most teams are below 50% on their finished robot.

Default Code:
This was an intentional shift. Default code was replaced by default source code for the same reasons that the frame elements aren't preassembled, the electrical prewired, etc. Preassembled components to avoid a challenge will likely speed up one task, but may deny others from contributing.

WPILib:
I'm well aware that you do not like and do not use the higher level components of WPILib. As you found by digging deeper into WPILib, you are not forced to use floating point numbers, math, and fancy stuff. Features like NetworkTables do not take away your ability to do low-level UDP protocols and the majority of teams use them.

Getting it:
Lucky for you, CTRE and NI and WPI and AM are all contributing to Athena.

Greg McKaskle

[apalrd]

CPU Usage:
I am good friends with the programming leaders from three local teams (51, 1718, 2337). All three use LabVIEW and felt severely limited by the compile/download cycle time, especially when making changes between matches, and CPU utilization. I believe all tree hit 100% CPU at some point, all asked me for advice on reducing CPU utilization and were forced to run control loops slower than they wanted, and at least one was unable to run even a single control loop faster than 15ms. To my knowledge, none were using any sort of vision processing on-robot. I do not directly talk to other software team leaders who use LabVIEW, but I've seen similar sentiments on ChiefDelphi over the past two years. The fact that most teams are under 50% shows that most teams aren't doing anything reasonably advanced or using a lot of IO.

Default code:
IFI provided the source for their default code, as well as binaries, and pre-imaged controllers as shipped with it. You could use it as-is or modify it to your needs. Every test chassis we have built using an IFI control system has run default code, and we sometimes modify it to suit the specific chassis (e.g. when we built the DualDrive pre-2011, we implemented a C version of the auto-lift control, but we only modified the lines where relay1 was set to command it from something different). Source-only default code is better than nothing, but the current source-only default code is nowhere near the IFI default code in functionality. With the IFI default code, there was a table that maps all joystick inputs to motors, and all buttons to relays, and one relay and DIO pair were used for the compressor. You could wire it up and test it. The current default source only provides for a tank or arcade drive, even the compressor control isn't enabled by default.

WPIlib:
In 2012, I was able to reduce CPU usage on a 10-PWM robot by ~20% by rewriting a single VI and dependencies (this created Motor Set Simple), out of the hundreds of VI's in the WPIlib only ~10 were touched. Many team programmers personally thanked me at the Championships and via CD that year, as the 20% CPU utilization was significant to their robot. I do not believe that high CPU usage is limited to only a few teams. I think it's quite rampant with teams who control their robot using feedback controls, especially those who don't know how to optimize code to the LabVIEW execution system.

[AllenGregoryIV]

Originally Posted by apalrd

CPU Usage:
Default code:
IFI provided the source for their default code, as well as binaries, and pre-imaged controllers as shipped with it. You could use it as-is or modify it to your needs. Every test chassis we have built using an IFI control system has run default code, and we sometimes modify it to suit the specific chassis (e.g. when we built the DualDrive pre-2011, we implemented a C version of the auto-lift control, but we only modified the lines where relay1 was set to command it from something different). Source-only default code is better than nothing, but the current source-only default code is nowhere near the IFI default code in functionality. With the IFI default code, there was a table that maps all joystick inputs to motors, and all buttons to relays, and one relay and DIO pair were used for the compressor. You could wire it up and test it. The current default source only provides for a tank or arcade drive, even the compressor control isn't enabled by default.

This is seems like something we as a community can help with without NI or WPILIB people getting involved. If a couple people just make default code (LabView, C++ and Java) that is at the level of the old IFI code and then just distribute to teams. It would be easy to have on a thumb drive at competition, to fix the teams that tried very hard to write their own code but failed.

[Greg McKaskle]

by the compile/download cycle time

This is not related to CPU usage and was caused by bugs in the latest LabVIEW release related to the compiler cache. Please understand that I'm not claiming WPILib or LabVIEW are perfect. Internal to NI I called a special meeting with VPs and the President in order to highlight these bugs, the impact they had on teams/customers, and to motivate that they not only get fixed, but that testing is improved to keep them fixed. I continue to use FRC as motivation to push various internal teams to improve areas of the product, but this has nothing to do with cpu usage.

I have two years logs for 1718. Their typical usage for matches in 2013 St Louis was under 30%. At earlier 2013 competitions it is several points lower, and in 2012 at the end of the season it was 40%. I don't have logs for the other teams on my laptop.

CPU usage is an interesting challenge. One loop with more work to do than time to do it in -- and the result is that the CPU will be pegged. Finding that loop can be a challenge, but they seem to have accomplished it and hopefully learned because of it. There are numerous tools to help professionals and FRC teams alike in monitoring and controlling CPU usage, loop rates, etc. I am more than happy to help here or by other means, but to me a CPU usage challenge doesn't mean WPILib is broken. It means that FRC isn't easy.


Defaults:
The decision not to have default binary was made five years ago, and I've been approached only once since, by an alum, and we discussed the tradeoffs related to it. To a large degree, test mode was added as a result of that previous discussion. Features should be added because they will increase the success of students in FRC, and that needs to include a variety of opinions. This forum thread is not the right place to design this ... but perhaps another thread?

In 2012, I was able to ...

And that was a nice accomplishment. If WPILib were perfect, would you have learned more, or less? While WPILib isn't intentionally trying to get in your way, I don't want it to hide real-world programming issues from you either. I'm pretty sure that the changes you made in 2012 were already incorporated in the 2013 code along with a number of other performance improvements -- not all WPILib changes add on top and make it heavier. Other changes in 2013 were test mode and an newer, leaner, interop version of Network Tables. These were intentionally added to aid newer teams.

If you have strong feelings about WPILib, and I know you do, please don't rant in various threads all over CD. Create a thread that is dedicated to it, and we can get into discussions about how much it should do for teams, what it should not do for teams, etc.

Greg McKaskle

[Greg McKaskle]

to fix the teams that tried very hard to write their own code but failed.

This portion of the post jumps out at me, and again, I'd like to discuss it in another thread. I've also needed to "start over" a few times on Thursday at an event, and we do it by opening the template and writing the code together. I assist, but they drive. Starting over is not that common, and typically we just debug and fix their code. We just "finish" it.

It is not great that this happens at the event rather than within the team during the season, but this is no different than helping the teams with weight or wiring issues. Should we default those elements too?

If the template code needs other features, that is where I'd prefer to start. I'd like to hear other thoughts on this ... in another thread.

Greg McKaskle

[apalrd]

I started a new thread for default/template discussion. See Here.

I also started a new thread for library discussions. See Here.

Continuing on,
As for the LV issues, I didn't know that. I assumed it was library bloat, since the time goes up every year and compile/download times seem primarily related to number of files. A Buzz18 build (2013) are a lot shorter than a Buzz17 build (2012) and about the same as a Buzz15 build (2010). Other teams 2013 builds (from what I've seen, fairly small sample size) take 5-10x as long. It's painfully slow to help teams work through their issues between matches when you can't hardly deploy to look at anything. It's good to see it's being fixed, but the compile times have been on an upward trend for a while now.

Edit: Greg, was there ever a fix to the no-app issues (at high CPU load, bootload seemed to be starved and unable to download new code without no-app DIP)?

[Jim Giacchi]

In 2001 as a (High School) Sophomore I decided to teach myself the control system. I took home the new system with a motor and a battery and withing an hour had everything setup using radios, to make the motor spin back and forth. Latter that season I wrote the code for the robot because no one else wanted to and did so by downloading and installing the several megabyte program (I'm pretty sure I may have even copied it on a floppy disk, yes I am that old, I also built my robots using candle light and hot wax burns!!!).

In 2009 - 2010 as a graduated and employed Mechanical Engineer who works everyday on robots I attempted, and I stress attempted to get the control system to work. I failed miserably, giving up in frustration and turning it back over to the electrical advisor telling him to fix it because I just didn't care anymore.

The funny thing is that our robots are the same complexity as what I did in 2001, they are no more complicated, but in order to get them up and running the control system is orders of magnitude more complicated and the compiler takes multiple DVDs (Thats gigabytes, with a big old capital G) God help us when our computer crashed last year halfway through the season, took the students and mentor over a week to get it back up and fully updated.

So basically what I am saying is Please.... Please I beg you and wholeheartedly agree with the others who have posted that it needs to be less complicated and easier to setup. The metric should be, can a student get this running on their own. I understand that they may not be, can the student get this to use a camera, pick out a shape and have the robot do a backflip frisbee throw into the top goal in autonomous, but a student should within an afterschool meeting get a robot to drive from start to finish. Right now the system is not even close to that.

Spending time troubleshooting something this complicated always eats away at the fun I have working on this and I have longed for the days of that beautiful black IFI box, oh how i miss thee (and it's not the IFI part, it's the simplicity)

Thanks for reading my book of a post,
Jim

[apalrd]

Originally Posted by Jim Giacchi

The funny thing is that our robots are the same complexity as what I did in 2001, they are no more complicated, but in order to get them up and running the control system is orders of magnitude more complicated

Interesting analogy.

Our 2001 robot ran two PI controllers, each had 2 potentiometers and a single motor (it actuated a complex cable/spring driven linkage so it switched sensor part way through the travel), plus an auto bridge balance program. All of this ran in 63 bytes of variable space on a BASIC Stamp, with code in PBASIC, in a 26ms task. (authors note: The linkage claws on the 2001 robot are some of the coolest things I have ever seen)

Our 2013 robot ran one PI controller, with one potentiometer, and several state machines. Autonomous has no closed-loop controls, just thresholds and sequences. Granted, we use interpolation tables (which include For loops, not entirely lightweight) and floating-point math, but we're running a 10ms task on a 400mhz PowerPC and using more than half of the bandwidth.

Both robots were world finalists. The 2013 robot still has more feedback controls than most of FRC. FRC does NOT need more processing power.

It takes our 2013 robot a long time to boot up and find the field, and builds take a minute or so. The 2001 control system would have booted and established a radio link in 5s, under 1s on tether (~200ms I've heard).

Speaking of boot times, I've worked on programs which mandate a 250ms boot time from power applied to ready to synchronize and start, and can do a soft reset (module stops executing code and starts from the beginning without powering down) without stalling the engine. These programs run over a million lines of C code on a processor that runs half as fast as the cRio, with control loops at 1khz, on a PowerPC core similar to the cRio.

I've heard VxWorks boots in a few seconds on the cRio, why does it take soo long for user code to come up and init? I know some of the init code is even more inefficient than the runtime code (Encoder4x does a typedef conversion of the three lines A/B/Index 12 times or more at init), but it's still 30s or so before user code even begins to init.