Re: NI releasing/designing new controller for FRC
 

In the WPI Lib PID code in PIDController.cpp (as of version 3487) opens a Notifier object with a default parameter of 50ms. It does like you describe forces the loop to a certain timing, and the reason for this is that this design does not wish to calculate uneven time slices within the PID algorithm. The result is good results from a functionality standpoint, but will have the same issue of context switching especially if one wishes to use a lower period. In c++ the programmer does not have to go in this direction, they can write their own PID http://www.termstech.com/files/Archi...Controller.zip and avoid context switching all together. I had proposed this solution for the WPI library but there was not enough expressed interest in going in this direction. Really when you think of it... a machine can only execute one instruction at a time... when you have tasks/threads... the context switch overhead can really eat up cpu processing, and it doesn't have to be this way. I have spent several years on this issue documented here http://www.termstech.com/articles/PID_Kalman.html

Re: NI releasing/designing new controller for FRC
 

Interesting,

The last time I ran the VI profiler with the full WPIlib the highest time VI's were the Relay Set and Motor Set, and the inefficiencies of them stacked up.

One of the real reasons LV is so inefficient even with seemingly simple code is because of the way it deals with with execution. In C and most other languages a function is an almost logical construct which just segments code, LV does not separate VI's this way.

In LV, each VI is a node in the execution system. LV then manages a smaller set of VxWorks tasks and an execution state of each VI. For this reason, by default each VI has a single instance and any local memory is retained between calls (you can use a VI for data storage by having a get/set input plus a data input/output and a shift register, the WPIlib does this a lot if you look). Any single VI can also only execute once at a given time, so an execution in another thread blocks the same VI from executing in a different thread. When all of the inputs necessary for a VI to execute is ready, the VI will be scheduled into a task to be run at the earliest opportunity, and then the data output will be set and the dependent nodes can execute.

This execution system is significantly more inefficient than a C function call, but virtually nobody realizes this. For this reason, a VI call is considered an expensive entity, not as expensive as another task entirely but not as cheap as a C function call. The ways around it are to set the VI as subroutined (this will not work if any contents are non-subroutined or blocking nodes) which cheapens it to near a C function call, or set the VI as inlined (this does not require non-subroutined subVI's but does require the VI to be re-entrant) which is inlined at compile time (this can reduce compile efficiency if you change a VI which is inlined as it has to rebuild all VI's which include it). Both subroutined and inlined VI's cannot display front panel data or probes in realtime when debugging, but you can still pass data through the connector as usual.

In a lot of ways the LV execution system helps a lot when you want to do multitasking (which is trivial in LV) and the single local variable set helps with data storage in quite a few cases, but if you don't understand it and set the subroutine and inlined properties rigorously for every VI in a project, the inefficiencies of the execution system stack up really fast, especially for a library the size of WPIlib plus a team project with over a hundred VI's. In 2012 I thoroughly went through my WPIlib copy to subroutine and inline VI's where possible, I believe some of this was later integrated to WPIlib.


I personally think it's quite reasonable in RT embedded systems to essentially cheat the OS. Every other RT embedded system I've worked on runs purely statically allocated RAM and uses processor ISR's to deal with tasks, so the OS kernel is a single function (timed ISR) and there are no context switches. There is then no penalty for doing context switches frequently, but we still try to optimize it.

In C++ the PID only runs at 50ms (20hz)? That seems insanely slow! I would expect at least 20ms to be considered a realtime control loop.

Re: NI releasing/designing new controller for FRC
 

Our homebrew PID normalizes for time, so the calculation itself does not need to be inside a timed loop. We place it in a timed loop so that we can attempt to match the speed controller update rate for the best 'idealized' performance. I'm not sure how you'd go about this without a timed loop.

Re: NI releasing/designing new controller for FRC
 

That is very cool in regards to the ISR's dealing with tasks... I'll want to research this a little tomorrow. What I have found (using Intel processor I7). Is to treat threads (aka tasks) as sparingly as possible to only be reserved for timing and I/O tasks. We tried some parallelization techniques, and found unless there is NO convergence (which is usually not the case) we don't come ahead in saving time except for very rare cases which only benefit on some machines. Doing less work that is simplified in one task usually wins. I'm not saying that this is not possible, but to successfully achieve optimal parallelization infringes on making the code less readable and maintainable than what we cared to pursue.


This is what this looks like:

I am not sure why this was chosen as the default parameter, but as I recall from Brad, this was tested by several teams to work well for many general cases. I think I may have an email on that somewhere. One can easily construct this class to use a different rate, but I know some teams may have used the robot builder for this season and they probably could have over-looked this in the auto generated c++ code.

Re: NI releasing/designing new controller for FRC
 

There are a few questions I have on this:
1. How well would this work mechanically if you used a 10ms timed loop?
2. How much improvement would the cpu usage be for this?

Unfortunately, I do not know much about LabView at all, but in c++ you call GetTime() in your main OperatorControl() loop and pass this time as a parameter. For our code the entire loop consists of "void TimeChange(double dTime_s)" call delegated out to various classes including the PID controller class (per rotary system).

Here is our main loop

Re: NI releasing/designing new controller for FRC
 

Performance:
The attached image shows how performance has been tracked over a few years. Times in ms. While you may not feel that anything/enough has changed in WPILib, the attached image shows the relative call timing for various WPILib functions and with different WPILibs that were shipped. The tests have been in place and the results used to verify that the speeds were where we wanted them.

So, the numbers for 2012 sped up. Why? The default for LabVIEW WPILib was to leave debugging on and make the code easier to explore and run subVIs. After discussions and tests, we decided to change a few implementation details and to migrate to a newer built-in error merge node. We decided to turn off debugging on some core functions and to inline a few others. The libraries could be faster, but the goal is to strike a balance. This is not meant to be a maxxed out race car, but a training/learning car. The runtime performance of WPILib seems fine from the logs I've seen and from the performance tests we regularly monitor. Vision can easily max out the PPC or any other CPU, and that is a relatively independent issue. If WPILib were for professional engineers it would be written far differently, perhaps like the sensor classes that our commercial robotics product ships.

LV is inefficient:
LV subVIs offer many features, and when used, they naturlly add overhead. Inefficient implies it isn't useful work. A subVI has a set of Execution Properties that affect how it executes. They are ...

Debugging: Allow for breakpoints, probes, and realtime panel viewing. The code is a bit larger, slower, and much friendlier for mentor/student debugging sessions often without even needing to recompile.

Reentrancy: Select between nonreentrant, preallocated, and shared pool reentrancy. Nonreentrant means that the subVI has a critical section around it to protect state data, a hardware session, etc. It is a parallel language, and this is the default to avoid race conditions. Preallocated is useful when state data needs to be per instance such as with a PID. Shared has a common set of pooled temporaries and the lack of critical section enables parallelism. For computational subVIs that are very small and called often, it is sometimes useful to make them reentrant to lower call time by losing the critical section.

Inlining: The function call overhead is jettisoned entirely and the code is compiled into callers. This trades off call overhead for debugability and potentially code size.

Priority: All but subroutine are implemented by choosing which priority the OS thread has when it executes this VI. Boosting is also used to avoid inversion. The subroutine is special in that it serializes code within the VI, doesn't generate cooperative yields, and has the least amount of debugging features. For high function count calls this is an optimization tradeoff you can make.

System: This is typically used to isolate I/O libraries that use thread-local storage, sleep, or otherwise block their thread.

------------

Execution System Explanation:
As discussed above, VIs can be low overhead or higher overhead. The execution system of LV allows for data flow of wires to trigger and suspend execution. It does it very efficiently, and with simple syntax. Features have a cost, and for example, C++ dynamic dispatching isn't as fast as static dispatching. Debugging code tends to be slower to execute than optimized code. Overly inlined Vis are just as hard to debug as C++ code. The fact that you are aware of these tradeoffs is great, and one of the benefits of projects such as FRC. I do wish that you could accept that WPILib needs to balance runtime performance and development experience. Even within your team, I'd expect that there are benefits to using floats and higher level abstractions. Put another way, is the CPU usage wasted if it is used to clarify your thought process or to
simplify how something is taught?

I'm happy to answer questions or look into performance issues you perceive in WPILib, but it would be great if the feedback was more direct and less of a rant that hops from subject to subject.

Greg McKaskle

Re: NI releasing/designing new controller for FRC
 

It is better understood. It has less to do with high CPU usage and more to do with error handling, but since the current error hook is slow, they go together.

The issue is that the auto-error handling hook that we added which allows us to send errors to the console is not typically used on RT or even on desktop. It is not very optimized and it is difficult to modify. It also turns out that it can confuse the RT protocol that is trying to exit the current app and start the new one.

I have written code that will route errors more efficiently and will not cause issues with RT, but at the moment it is shelved waiting for the LV release to complete. It is expected to make it into the next version of LV, and there is potential for it to go into an FRC patch.

If this doesn't happen, we now know what causes it and will be able to work around it. So in effect, yes. It should get fixed this next year.

Greg McKaskle

Re: NI releasing/designing new controller for FRC
 

Real-world software developers have to make approachability/speed of development vs raw performance tradeoffs every day. Consider implementing a desktop app. Do you use a cross platform environment like Java, gaining the ability to run in many places but losing some fidelity to platform look and feel and less control over the memory footprint of your application? Or do you use C# and lose the ability to move across platforms, but get high productivity and platform fidelity if you're on Windows? Or write in C++, gaining the performance benefits it brings but also bringing lower developer productivity? Depends on the situation.

The FRC control system provides a range of options for control system development that let you trade off approachability and speed of development against raw performance.

Generally, the LV environment is more approachable and easier to get up and running, with some tradeoff in performance. Whether the tradeoff matters varies based on your scenario.

To beat up on the LV implementation because "it's slow" misses the point -- as Greg points out, the LV implementation is intended to support approachability first, while not unnecessarily giving up on performance. But, like every other environment, if you care about speed greatly (e.g., you're going to debate 10 vs 20 vs 50 ms update times), you're going to end up having to learn about the internals of the environment, and may end up replacing parts of it with custom coding that implement a different approachability/speed tradeoff.

Or you may decide that LV isn't for you and you should use Java or C++ because it's better suited to your scenario.

Re: NI releasing/designing new controller for FRC
 

A few things,

First, I don't know of anyone else on a team who debugs the WPIlib. It would make sense to me to heavily optimize it, since every other LabVIEW team I've talked to treats it as a golden black box. They don't see it as something to be modified, they see it as something to use and assume it works as they expect. I acknowledge that it's improving over the years, but I still don't like it.

Second, I come from a realtime software world. We write our code in C or autocode from Simulink/Stateflow, and assume a lot about timing determinism and stuff. It bugs me a lot to see a software environment that should be capable of real-time control algorithms running with such poor timing determinism and everyone accepting this as good or acceptable thing. LabVIEW itself has proven to be a great tool, but there is a lot of overhead in a lot of the FRC specific stuff which bogs it down too much.

Third, the RT software world assumes timing determinism, it's considered a total failure of the operating system and control algorithm software if timing goals cannot be achieved. I'm not saying we should demand such careful and extensive optimization, but if a 56mhz PowerPC can run some RT PID loops at 2ms we should be able to get 5ms out of our 400mhz PowerPC.

FRC is not a desktop app. We do have realtime control system performance to consider. Every year we have re-evaluated switching to C programming (procedural, not OO) for performance, and keep coming back to the debugging tools of LabVIEW vs what the C environment has to offer.

Re: NI releasing/designing new controller for FRC
 

procedural, not OO eh? Sounds like something I use to say ;)

I feel your pain brother,
When I was a kid making games in BASIC for my commodore and the sprite collision detection was so late... I got so frustrated. I had the need for speed. I was 11 at the time... BASIC got me into programming but if I really wanted speed I had to take a leap of faith as no-one was going to make the BASIC go any faster... so I invested in a Vicmon and started 6502 assembly. I got the speed I needed and this decision ended up changing my life to help me find my career where I am today.

If you are like me and you have the need for speed... join the elite 18% of teams who have found it. You can do C programming in c++ and I believe it supports inline assembly as well. You can still make diagrams too... I make them in PowerPoint and then make the code from them afterward.

Don't complain and hope for change... be the change!

Re: NI releasing/designing new controller for FRC
 

This morning I created a new templated FRC project and ran it on my 4 slot cRIO. It still contains network tables, empty teleop loops, etc. I did not run the dashboard.
CPU usage was 20%.

In Periodic tasks, I drop a timed loop, set it to 5ms period, and placed four pot reads, four PIDs, and four motor sets in the loop.
CPU usage was 45%.

I split the code into four timed loops running in parallel, each with one pot PID and motor set.
CPU usage was 50%.

I deleted three of these and set the remaining one to 1ms timing.
CPU usage was 60%.

The only time the CPU usage went to 100% was when I made a mistake and was receiving error messages and when I was charting my time intervals to look at the determinism.

-------------------

If I were to place more than 5ms of work inside of a 5ms loop, I would indeed lose determinism, peg the CPU, etc. In that case, I'd either need to up the loop period or simplify the processing, perhaps by using the lower level layers of WPILib.

Given the table I posted earlier, you can calculate how many motor sets you can make per second, how many pot reads you can make, etc. Faster loops make more calls and that is the major determining factor, not he loop or scheduling overhead.

The timed loop runs at time critical priority, and its CPU load will be a bit higher than normal loops because it causes more context switches and runs on a different scheduler, but this is not a big overhead if you want good determinism. The default framework uses normal loops because of their simplicity, and while not as deterministic, they seem fine for typical use.

Do you see an issue with my measurements?

To summarize, I concede that WPILib is not the ultimate fast library. It is a learning library meant to introduce people to programming and robotics. It also isn't perfect, and if you see an issue with it, please let us know, ideally with some sample code. And if you have outgrown it and choose not to use much of it, that is fine. It was written so that you can choose your level of abstraction, choose your level of challenge, etc.

Greg McKaskle

Re: NI releasing/designing new controller for FRC
 

Am I reading the diagram correctly? Will there be no router needed either? (The diagram in question: http://i.imgur.com/eA3Bvfu.jpg)

Re: NI releasing/designing new controller for FRC
 

It still say's "wireless communications" bridges the robot and the control station. I would assume we are still using wireless from the diagram. The pics of the board don't look like they have wifi on them to me. I would assume a router of some sort will still be needed.

Re: NI releasing/designing new controller for FRC
 

The prototype at worlds did not have on board wireless. I got the impression that coms between the DS & Athena will still be Ethernet. Makes sense to leave wireless part off board. Makes it easier to change. There was a large Qualcom booth at worlds...

But you will still need a router (actually more of a switch & a bridge)

Re: NI releasing/designing new controller for FRC
 

Will we be seeing more of Athena at NI Week?

Re: NI releasing/designing new controller for FRC
 

Per this tweet:
https://twitter.com/nifirstrobotics/...80410395820035
The event will be livestreamed on August 8th @ 8:30 CST (9:30 EST?) at NI.com/FIRST

Re: NI releasing/designing new controller for FRC
 

Thanks - I'll put it on my agenda. Any other C-D folks going to be there in person?

Re: NI releasing/designing new controller for FRC
 

Yup :)

Re: NI releasing/designing new controller for FRC
 

Absolutely.

Re: NI releasing/designing new controller for FRC
 

We have a mentor and two students going to NI Week from our team.

Re: NI releasing/designing new controller for FRC
 

We'll be there as well. Cameras and video cameras will be allowed into the event, so we'll get all the pictures and video we can.

Re: NI releasing/designing new controller for FRC
 

Does anyone know if the release webcast will be archived afterward?

Re: NI releasing/designing new controller for FRC
 

Is it August 8th yet.

Re: NI releasing/designing new controller for FRC
 

Yes - we will have both the keynote announcement and the Q&A session available for viewing after the event. We will try to get both links up on ni.com/first as quickly as possible.

Re: NI releasing/designing new controller for FRC
 

Thanks!