NEW 2009 Control System Released

[Greg McKaskle]

Originally Posted by Kevin Sevcik

... Did it increase the allocated memory by 1 element? 10? Some constant somewhere? Did it double the size? I can get at this info for a rather lot of other programming languages, but I haven't a clue what labview is doing for me behind the scenes.

... Of course this is, again, just so much voodoo. I have also coded up a rather gate-hungry FPGA program.....

The forums for LV often cover these sorts of implementation details, but they of course change over time. Both the system memory manager and the internal array allocator round up to some degree before allocation. The insert into array doesn't do anything beyond this, but the loop boundaries use a bounded doubling scheme. And all memory management does indeed slow loops down, just as in any language. It is also quite easy to preallocate the array and replace elements if you want to remove the memory management and increase the speed.

In other words, LV will not prevent you from writing code that operates inefficiently. The compiler will do what it can to improve things, and as in any language recognizing and measuring what code is doing takes consistency and practice. One benefit to rapid development languages like LV is that while the array insertion code your fellow student wrote wasn't optimal, it sounds like it did work. If a non-CS user is able to get valid results from their computer, that is a huge improvement over what they could often achieve with a language that gives them super low level control. Hopefully when the time comes for optimizing, they are then able to keep the correct results as they experimentally modify their implementation.

If you have further questions about how LV implements its voodoo, both the NI LV forum, and now this forum are a great place for it to take place.

Greg McKaskle

[qnetjoe]

I keep seeing a lot people of keeping talking about how the FPGA is off limits. I really don't think that FIRST would go that far, because without access to the FPGA what is the real benefit of the new controller?

I would take a guess to say that there will be a custom VI that allows us to tailor the use of the FPGA while allowing FIRST to maintain their required supervisory control. I have done things very similar to this with my cRIO's and PXI chassis with RT controllers.

For years with FIRST safety has only been skin deep. Change a few registars from any of the 2004-2007 controllers and your robot can be really dangerous.

Besides you have 20 engineers at FIRST/NI vs a FRC community of 20k with a passion for this stuff. We WILL find a way to program that FGPA.

[Pavan Dave]

Originally Posted by qnetjoe

I keep seeing a lot people of keeping talking about how the FPGA is off limits. I really don't think that FIRST would go that far, because without access to the FPGA what is the real benefit of the new controller?

I would take a guess to say that there will be a custom VI that allows us to tailor the use of the FPGA while allowing FIRST to maintain their required supervisory control. I have done things very similar to this with my cRIO's and PXI chassis with RT controllers.

For years with FIRST safety has only been skin deep. Change a few registars from any of the 2004-2007 controllers and your robot can be really dangerous.

Besides you have 20 engineers at FIRST/NI vs a FRC community of 20k with a passion for this stuff. We WILL find a way to program that FGPA.

They said for the first year it would be off limits. After that its fair game, unless I heard wrong.

[MWagon]

Originally Posted by Kevin Sevcik

.... I managed to implement the forward kinematics for a 3-DoF rotational joint device, X-Y-Z PD loops, and the transverse Jacobian to take those XYZ forces back into joint space, with lookup tables for all the trig functions.

Didn't I warn you not to talk like this in public Kevin?

[Kevin Sevcik]

Originally Posted by qnetjoe

I keep seeing a lot people of keeping talking about how the FPGA is off limits. I really don't think that FIRST would go that far, because without access to the FPGA what is the real benefit of the new controller?

I would take a guess to say that there will be a custom VI that allows us to tailor the use of the FPGA while allowing FIRST to maintain their required supervisory control. I have done things very similar to this with my cRIO's and PXI chassis with RT controllers.

For years with FIRST safety has only been skin deep. Change a few registars from any of the 2004-2007 controllers and your robot can be really dangerous.

Besides you have 20 engineers at FIRST/NI vs a FRC community of 20k with a passion for this stuff. We WILL find a way to program that FGPA.

Actually, I think it's well nigh impossible for you to programmatically override the supervisory control of the master processor in the IFI controller. All the outputs controlling real world devices sent through buffers controlled by the master processor, so it has the final say if a motor's turning or not. And it takes very little data back from the User Processor in a very controlled manner, so I don't think you could even reliably override it with packet overruns or something.

[dcbrown]

Originally Posted by Dave Flowerday

...except only the things that FIRST and/or NI has thought of are committed to hardware. Since the FPGA is not user-programmable, any novel ideas or sensors which would need interrupt support are not possible - I think this is what Alan was getting at above. In my opinion this is a HUGE step backwards from the current control system and a very disappointing revelation.

Dave you quoted my post, but I also wrote the following in the same post:

Originally Posted by dcbrown

if you want to tweak the servicing of the hardware or integrate data differently or add some weird/unusual sensor that isn't already allowed for, you're pretty much stuck as the FGPA is off-limits to teams.

I guess I wasn't clear. Anyway, I think we're in violent agreement. By logical extension, as Kevin mentioned, this also means that the FGPA may have limits in the number of same type supported devices. For example, the FPGA may only be programmed with support for 4 quad encoders plus 2 GTS plus 2 sonar. (I suspect sonar is a bad example and could see a design decision not to support the sonar sensors in FPGA but rather on I2C ala NXT w/the built-in PIC controller within the sensor box doing all the work and then using the possible I2C bus off the digital sidecar to retrieve data as needed - again polling). So if you wanted to use n+1 sensors of a specific type on your robot, where 'n' is the number implemented in the FPGA, then your kinda out of luck. Although the sensor processing with the FPGA has not yet been completed, I would find it difficult that at least design intent isn't known at this point and its too bad that type of information isn't shared.

One brain blip I had in regards to above was that there wouldn't necessarily be just ONE FPGA programmed image to choose from - after all this is field programmable. In that case, much like configuring a work truck you'd choose from one of a number of standard FPGA setups provided. There are only so many digital input lines to choose from after all, so how you dedicate them to what sensor suite may be selectable. That is, one image supports 6 encoders plus other stuff and another supports only 2 encoders plus other things. With multiple FPGA images provided the likelyhood of finding one close enough to match the robot you want to design becomes greater.

Originally Posted by Greg McKaskle (NI Team)

Realtime OSes don't necessarily have the same division of kernel and user modes. Specifically, the RIO peek and poke are allowed to be called from user code.

The early open of the FPGA must not be as a device then or I severely misunderstood the VxWorks documentation that indicated open device descriptors were not shared across the RTP/kernel interface. Of course there is always the possibility that PEEK/POKE are implemented as custom system calls for the cRIO from the RTP into the kernel or the open of the FPGA isn't as a normal device. It probably doesn't ultimately matter to programmers, but as an engineer I kinda like to know how my request is being routed as it effects how I design the processing flow of the code as the route "distance" in software can effect performance.

Originally Posted by Greg McKaskle (NI Team)

To this point, the C/C++ for the cRIO has been for internal consumption. To this point, all cRIO customers have used LabVIEW.

Don't take this the wrong way -- just after 30 years of systems engineering at the s/w-f/w-h/w boundary this statement quietly screams "WATCH OUT!" One way of interpreting this comment is that FIRST will be the beta test customer for using C/C++ to program the cRIO. From a product roll-out risk perspective, things just got a whole lot more interesting.

[qnetjoe]

overwrite your tether and automous registers and IFI controller will function without any supervisory control. You can use the live debug feature inside of MPLAB to find these

[Danny Diaz]

Originally Posted by dcbrown

The early open of the FPGA must not be as a device then or I severely misunderstood the VxWorks documentation that indicated open device descriptors were not shared across the RTP/kernel interface.

No, you read the VxWorks documentation correctly. However, what the VxWorks documentation fails to mention (not that it could if it wanted to) is that NI has not implemented the use of RTP's in the variant of VxWorks that we're using for our cRIO product line. So, all calls you make are kernel calls since there is no concept of "user mode" in what you're using.

Of course, this is something that is subject to change in future versions of the VxWorks port for cRIO, but it's most likely not going to happen in even the next major release.

-Danny

[Dave Flowerday]

Originally Posted by qnetjoe

overwrite your tether and automous registers and IFI controller will function without any supervisory control. You can use the live debug feature inside of MPLAB to find these

This is getting off-topic, but...

Have you actually tried this? I'm guessing not, because that won't work. Please, if you disagree, I dare you to prove me wrong. Post a video or something.

IFI designed a superb system from a safety standpoint - it's basically impossible to override the safety features through anything you can do on the User processor. After many years of studying the details of the IFI design, it is very, very obvious that safety was a top priority in their system. Field reset crews, referees, and match announcers can step on the field with confidence knowing that if the control system has the robots disabled, they will really be disabled. I certainly hope as much care is being put into the safety features of the new system as IFI put into theirs - it's not a good thing when a 130lb machine moves when you think it's turned off.

[Alan Anderson]

Off-topic for the thread, but still worth mentioning...

Originally Posted by Dave Flowerday

IFI designed a superb system from a safety standpoint - it's basically impossible to override the safety features through anything you can do on the User processor.

Yes, they did good keeping programming errors (or evil-intentioned hacks) from causing unsafe operation. On the other hand, it is still possible to have a robot do dangerous things if you violate the wiring rules. One team was surprised recently to find that their pneumatics worked when the robot was disabled, because they had wired the Spike control signals to RC digital outputs instead of the required relay outputs. A Victor could run a motor without regard to the disable input if it were wired to something other than a PWM output (I've used servos in disabled mode, and I think you have too).

While I'm sure the new system is going to be just as safe as the old one, it's going to be just as possible to bypass the safety features with illegal wiring. Fortunately, that sort of thing is not hard to catch with a visual inspection.

[Dave Flowerday]

Originally Posted by Alan Anderson

On the other hand, it is still possible to have a robot do dangerous things if you violate the wiring rules.

True enough. If we have a battery, some wire, a motor, and have no regard for the rules (or sloppy application of those rules), we can make any robot move, no matter what the control system is

Luckily, it is (in my opinion) still much easier to catch wiring errors than coding errors... and that will be more true of the new system (basic rule of software engineering: as system complexity increases, so do the odds of having bugs).

[Jay Lundy]

I too would love full access to the FPGA, but that may be impossible. Here are some ideas that may give us more freedom even with a locked down FPGA.

Since the FPGA is not user-programmable, any novel ideas or sensors which would need interrupt support are not possible

It's definitely possible to send interrupts from the FPGA to the RTOS, at least in Labview. Presumably the FPGA could be hard-programmed to send every digital input to a different IRQ and if the signal processing on the FPGA doesn't work for you, you can do it in the RTOS. The help file on interrupts makes me worried about the interrupt latency using this method: "The Interrupt VI is a shared resource, so multiple uses of it induce additional delay and jitter due to arbitration," but it may be a possible solution. If you want to handle the interrupt in C and write something like Kevin's utilities, my guess is you could register your ISR with some sort of Labview run time library that processes and dispatches the interrupts (I don't think these are real interrupts so you can't just register ISRs with VxWorks). The WPI libraries could make this simple for us.

By logical extension, as Kevin mentioned, this also means that the FGPA may have limits in the number of same type supported devices.

Just because we can't change the physical logic on the FPGA, doesn't mean we can't change the way it behaves. We can still modify the registers that control the logic (right?).

For example, assume you have a black box that performs the signal processing on one encoder input. Now, stick a mux in front of it and feed every digital input into the mux. Then, use the RTOS to fill an array of registers with the inputs you want processed and one final register with the number of entries in the array. Then just start a counter to iterate through the array and feed that into the select pin on the mux to process each input one-by-one. Oh and you'll probably want a demux on the other side so the encoder count sum goes to the right place.

I'm sure there's a more efficient way of doing it than the way I've described, but I was trying to keep it conceptually simple.

[Edit]: Come to think of it... this won't work too well, but I think the idea may be feasible. Can anyone fix it?[/Edit]
[Edit2]: Each pin that can be configured as an encoder will need to have its own state (count sum). The only thing they might be able to share is the combinational logic, but it takes basically no CL to count an encoder, so there's really no point.[/Edit2]

[Kevin Sevcik]

Originally Posted by Jay Lundy

It's definitely possible to send interrupts from the FPGA to the RTOS, at least in Labview. Presumably the FPGA could be hard-programmed to send every digital input to a different IRQ and if the signal processing on the FPGA doesn't work for you, you can do it in the RTOS. The help file on interrupts makes me worried about the interrupt latency using this method: "The Interrupt VI is a shared resource, so multiple uses of it induce additional delay and jitter due to arbitration," but it may be a possible solution.

This is more or less what I was thinking, except I have some refinements. First, it's not at all clear in the FPGA documentation (there I go again) if the arbitration issues arise for every call of the Interrupt VI, or only for calls of the Interrupt VI addressing the same IRQ number. Ex: Flagging IRQ16 someplace and IRQ28 somewhere else might not have arbitration issues complicating the latency. Of course, issues of interrupt priority and concurrent interrupts might still affect things. Which also isn't well documented. If you can have multiple interrupts with different flags with no issues, they your idea would work. Else, I think it'd be smartest to just implement an interrupt-on-change feature across the entire range of digital inputs so a single interrupt tripped when any digital input changed state. You could run the inputs or the edge detection outputs through a simple and relatively cheap bitmask to enable and disable interrupts on a particular IO pin, even. It'd still be a little complicated, as you'd probably need to latch the edge detector and digital IO states when you flagged an interrupt, but I think this would be a workable solution for giving us access to interrupts. I still don't want to have to interface with quadrature encoders using interrupts but I imagine our only other option for general purpose high speed IO of digital inputs would be to set up a DAQmx task to constantly sample the inputs at some acceptably high rate, and then process this backlog of data either in our main loop or a parallel loop running at a rate between the sampling rate and the loop rate. The accuracy and dependability of this would of course depend on our digital IO sampling rate, but a higher sampling rate would give us that much more data to process, when the majority of it is largely useless.

Originally Posted by Jay Lundy

Just because we can't change the physical logic on the FPGA, doesn't mean we can't change the way it behaves. We can still modify the registers that control the logic (right?).

For example, assume you have a black box that performs the signal processing on one encoder input. Now, stick a mux in front of it and feed every digital input into the mux. Then, use the RTOS to fill an array of registers with the inputs you want processed and one final register with the number of entries in the array. Then just start a counter to iterate through the array and feed that into the select pin on the mux to process each input one-by-one. Oh and you'll probably want a demux on the other side so the encoder count sum goes to the right place.

[Edit]: Come to think of it... this won't work too well, but I think the idea may be feasible. Can anyone fix it?[/Edit]

Anyway, you can do this with every type of sensor. Just have one black box that you share between the inputs. Obviously it takes longer this way, but at 40 MHz the FPGA should have plenty of time to spare. If time is a problem then add more black boxes in parallel.

Does anyone see any problems with this?

I was also pondering something like this, but the problem with your second idea is that function blocks on the FPGA represent actual physical silicon. Counting through all the different inputs/etc. in a single cycle loop on the FPGA isn't going to be possible. You could get away with it in a less strictly timed loop, but then you have to worry about jitter and problems if your loop runs over schedule, and FPGA code outside a single cycle loop takes up a good bit more space on the FPGA. If you just said the 2N first inputs on the IO represent N encoders, then I think you could loop through the set of inputs, old states, and count registers for N encoders, processing one encoder per FPGA cycle. This obviously slows down your overall capture rate, but if they get the FPGA running at 20-40MHz as it should, then it shouldn't matter. Then you let the programmer define N to pick how many encoders she wants.

The other solution is to hard code in X encoder counters, and Y GTS counters and Q other devices, spread across the digital input lines as widely as possible, but each dedicated to a particular port, much like the different peripherals on the PIC. Then have various enable/disable lines to select which ones you want to use as GPIO and which to use as peripherals. Maybe you could even have various bits to twiddle on and off to select slightly different functions for the various modules or otherwise modify their operation. You could even have a few registers here and there to further tweak the function of the various peripheral modules. I bet if the NI/WPI guys work hard enough they could come up with the equivalent of, ooohh, three PIC18F2431 chips (only $3.07 per!).

Which is to say that if we wanted a large amount of flexible hardware based IO that we could simply turn on and off with the flick of a switch, a controller based on a local net of microprocessors might have been a better move. The whole point of an FPGA is that you only have to implement what you need and it can be whatever the heck it is you want. So I hope the various people involved in the control system understand our frustration with paying more for a new controller that we don't have yet and will quite likely lock us out of the single most useful feature it has.

[Greg McKaskle]

Originally Posted by dcbrown

One brain blip I had in regards to above was that there wouldn't necessarily be just ONE FPGA programmed image to choose from - after all this is field programmable...

The early open of the FPGA must not be as a device then or I severely misunderstood the VxWorks documentation that indicated open device descriptors were not shared across the RTP/kernel interface...

Don't take this the wrong way -- just after 30 years of systems engineering at the s/w-f/w-h/w boundary this statement quietly screams "WATCH OUT!" One way of interpreting this comment is that FIRST will be the beta test customer for using C/C++ to program the cRIO. From a product roll-out risk perspective, things just got a whole lot more interesting.

There have been many postings looking into the crystal ball of the FPGA, and I'll try to give a bit more technical info on what is possible. But policy decisions as to what is allowed are of course FIRST's call, and independent of what is technically possible.

The FPGA on the cRIO is an open LV target. This means that LV code can be designated to run on the host PC, the PPC, or the FPGA. Of course LV targets the FPGA via VHDL, and therefore the cRIO is ultimately targetable with C-based .out PPC files and bitfile produced from VHDL. To promote a migration path, promote working, stable robots, and ensure safety, the decision was to keep the FPGA closed for at least the first year. After that, depending on how things go, it could be opened to the extent that there are vanilla, chocolate, and rocky road flavors. It could be opened further by providing 09 source and allowing teams to go nuts. Technically, these are all possibilities, and it is a policy decision as to what makes the most sense for the organization and competition. At this point, I'm sure FIRST is listening, but any opinions you may have honestly aren't well founded. After a season, the feedback will carry much more weight.

As for the SW stack required to peek and poke. I'm not entirely sure, as this isn't something I've used much directly. The success of the product is hinging on the performance of said calls, so I'd say that it is probably pretty performant and more importantly, deterministic. Moving forward, I will be happy to get the gory details for those who are curious about how it works. But for now, I think you can trust that the FPGA is not using dev/IO, and is instead a low-level custom driver.

Greg McKaskle

[Kevin Sevcik]

Originally Posted by Greg McKaskle

Technically, these are all possibilities, and it is a policy decision as to what makes the most sense for the organization and competition. At this point, I'm sure FIRST is listening, but any opinions you may have honestly aren't well founded. After a season, the feedback will carry much more weight.

Greg,

Respectfully, I honestly don't think your opinions are well founded either. There are at least a few people in this thread that have experience with the cRIO platform. I personally have experience a PXI-RT/FPGA platform. The project I was discussing earlier was a graduate level project. I managed to use 32-bit Q-Math to cram the entire forward kinematics of a 3-DoF PHaNTOM haptic device on to a 1M gate FPGA, including the interface logic for my quad encoders, logic for virtual walls and the transverse Jacobian to translate my Cartesian forces back into joint space. If you're not familiar with robotics, that's, basically, one-half to one college ruled page of trig dependent non-linear equations. I've also been working with the C-based IFI controller since its inception. So I hope you believe that I might have some clue about this.

So, I think it was a valid opinion that it would be highly annoying to be forced to interface with custom sensors or encoders using fixed rate sampling or interrupts on the cRIO processor. My annoyance is based strictly on my personal dislike of compensating for hardware deficiencies in software AND NI's own literature lauding the flexibility and utility of this wonderful hardware that we're not going to actually be able to use, so I feel it's valid under the circumstances. I'll note that at the last control system change-over, we were given free reign over all the hardware on the PIC. The annoyance there was simply that so much of it was blocked off by buffers and other things necessary for safety.

I will agree that any opinion on how things will turn out in the 2009 season is entirely premature. However I think this is primarily because of the distinct lack of information available to us right now. The fact that the nature and utility of the FPGA code is still undetermined frankly makes me nervous. The fact that the development environments are still under development highly worries me. I do appreciate your willingness to discuss the nitty gritty of the RTOS and such with us, but since we don't know what sort of high speed or interrupt based processing we'll be needing on the RTOS nor what the structure of the code will be, I don't think I can ask any intelligent questions.

I think a lot of this angst and annoyance could have been taken care of if this system was closer to a prototype of the system going out to teams and farther from a demo piece put together by NI to sell us on the system. I know I would have been fine with more man hours spent on these TBD features and less man-hours spent on a pretty holonomic robot with a sponge canon and flashy interface that still on bears a resemblance to the environment and hardware we'll be working with.