NEW 2009 Control System Released

[Kevin Sevcik]

Originally Posted by Jay Lundy

These are all related to the fear that NI's ultimate goal is that someday we will all be programming our robots in LabView because the C/C++ interface is crippled. The NI reps have stated many times that is not the goal and the C/C++ and LabView libraries will be of similar quality. If you want to program in C, nothing is stopping you.

I think our primary concern is that Labview often times makes things seem perfectly transparent, easy, and efficient when they're anything but. It's not terribly straightforward or apparent what it's actually doing with the code you've given it and unless you're paying very close attention to what you're doing, it can easily come back and bite you. I was working with a Real Time VI developed by a fellow MechE student that was being used for data-logging for an experiment. I was having difficultly getting the loop rate as fast as I thought I should be able to when I finally noticed the problem. Each data point was simply added to an array using the Insert into Array function block. The VI was blithely allocating a new chunk of memory, copying over the entire old array, and adding a single new data point to it. Lacking a minor in CompSci and thus previously acquired knowledge of how arrays actually work, I would have just assumed that that was how fast things were supposed to run. More to the point, after discovering this, I attempted to ascertain just how intelligent Labview was about resizing the array it was inserting this element into. Did it increase the allocated memory by 1 element? 10? Some constant somewhere? Did it double the size? I can get at this info for a rather lot of other programming languages, but I haven't a clue what labview is doing for me behind the scenes.

Also, to those hoping to use the FPGA like myself, you won't be subjected to VHDL unless you really want to be or are boycotting Labview. The Labview FPGA module lets you do everything in function blocks just like everything else and it then works our your VHDL code for you. Of course this is, again, just so much voodoo. I have also coded up a rather gate-hungry FPGA program to do some forward kinematics at ludicrous speed. I knew I was cutting things close on the size of my FPGA, but I didn't know how close until I changed the values of some constants I had rounded slightly out of laziness. Imagine my surprise when my design suddenly no longer fit on the FPGA. Of course changing back to my old, slightly rounded values made everything fit just fine again.....

[lynca]

Originally Posted by Al Skierkiewicz

Finally, a delivery date of kickoff is too late (see above). When asked, I stated I need it next week. FIRST, if you are listening, please consider giving some teams prototype systems now so we can break them. Give us the chance to work out problems and solve them before the 2009 season starts.

As a mentor of two different rookie teams, I completely agree with Al's statements, rookies will have a lot of trouble debugging the new control system when things are not working. Also, I hope FIRST distributes KITS to teams that agree to throughly test the new control system and publish their results online before kickoff.

I can understand that giving KITS to all teams will be a major hurdle, and some will feel wronged if their team does not get a KIT. However I will not feel bad if my team does not get a KIT early, but I will be quite worried if ZERO teams receive a KIT early.

[ayeckley]

Originally Posted by yoyodyne

Working with software at the device level is what made the FRC embedded software development experience different and in my opinion more valuable then just more applications level programming experience.

I'll second that! I've never been in love with NI hardware or software products (though I'm grateful for their sponsorship).

I wonder how many teams will sit 2009 out because of the steep learning curves? For teams that have four-year students and tight budgets it might make sense, which is unfortunate. Unless the system comes pre-configured with a very, very, strong set of reference VI's and some unbelieveably great documentation, I can envision a large backlash against FIRST by frustrated and upset teams in March 2009. The IFI reference code was pretty good - NI needs to top that.

I'd have loved to have a year or two to make the transisition. I can't see any technical reason that the two field control systems couldn't have operated in parallel in 2009 (cost issues aside). Better yet, it would have been great to let the market decide which system the customers (e.g. the teams) prefer. One entry fee gets you an IFI system, and a higher entry fee gets you an NI system. That would have made a great introduction to the economic realities of the engineering profession for students!

I also wonder if Kevin Watson will stay involved.

[Billfred]

Originally Posted by Doug Leppard

To me the biggest programming problem of teams is that the hardware is usually not done until shipping day, thus the programmers usually don't get the machine until that day or at competition.

I do hope they will allow a few teams (like 1902) to have the processor ahead of time if they have the commitment to help others. Our fear is that with a new system in competition there will be many problems and even the experienced teams will be scrambling to make it happen and will not be able to help others.

I remember NI making available some DAQ hardware to teams that proposed what they'd do with them. What if some combination of FIRST and NI accepted proposals from teams to devise methods of getting the new hardware exposed to as many as possible before the season? With some bias towards geographic diversity (lest all the Mid-Atlantic teams that host off-seasons would have pretty epic odds), you could probably get a lot of teams up to speed pretty quickly, more with the benefit of YouTube/Vimeo/TBA.

[Kevin Sevcik]

Originally Posted by ayeckley

I'll second that! I've never been in love with NI hardware or software products (though I'm grateful for their sponsorship).

I wonder how many teams will sit 2009 out because of the steep learning curves? For teams that have four-year students and tight budgets it might make sense, which is unfortunate. Unless the system comes pre-configured with a very, very, strong set of reference VI's and some unbelieveably great documentation, I can envision a large backlash against FIRST by frustrated and upset teams in March 2009. The IFI reference code was pretty good - NI needs to top that.

I'd have loved to have a year or two to make the transisition. I can't see any technical reason that the two field control systems couldn't have operated in parallel in 2009 (cost issues aside). Better yet, it would have been great to let the market decide which system the customers (e.g. the teams) prefer. One entry fee gets you an IFI system, and a higher entry fee gets you an NI system. That would have made a great introduction to the economic realities of the engineering profession for students!

I also wonder if Kevin Watson will stay involved.

I'm afraid the financial and technical difficulties inherent in running the two systems in parallel are effectively insurmountable. Given that FIRST is still working out logistics, hardware designs, software support, and communications, I don't think you'd be able to add IFI compatibility on top of it and still have a product by kickoff 2009.....
That said, I wouldn't mind if we didn't have a product by kickoff 2009 and we had to fall back on the IFI system for one more year. To my mind, FIRST should have had a LOT of this stuff worked out already. The demonstrations at Champs should have been of prototype hardware for the initial production run, and we should have been looking at V1.0 of the software environment. Given that they announced a new control system in May 2007, and were certainly working on it before then, many of these issues should have been worked out. At the very least, they should have a solid price estimate for the system. One year into the project, apparently fully committed to the transition and they still don't know what it's going to cost? That's pretty astounding.
I'd be (moderately) happy if the control system was available around registration time this year AND FIRST shipped control systems to teams early along with the software etc. that they ship out months before the competition. We all saw the lovely results of beta testing ~15 fields with brand new scoring software and hardware AT the regionals in 2005. I really don't want to see the results of beta testing 1500 robot controllers AND field controllers during the 2009 build and competition season.

[JesseK]

Originally Posted by Kevin Sevcik

I'd be (moderately) happy if the control system was available around registration time this year AND FIRST shipped control systems to teams early along with the software etc. that they ship out months before the competition. We all saw the lovely results of beta testing ~15 fields with brand new scoring software and hardware AT the regionals in 2005. I really don't want to see the results of beta testing 1500 robot controllers AND field controllers during the 2009 build and competition season.

My single hope out of all of this is that teams will get the control system as soon as they register. That's about all we can practically ask for.

The good news is, if they fix any beta bugs during a competition it will take mere minutes to implement and mere seconds to upload over the wireless to our bots Hopefully FIRST is thinking about this framework and will keep a special upload port ready for such bug fixes.

[Alan Anderson]

Originally Posted by JesseK

...if they fix any beta bugs during a competition it will take mere minutes to implement and mere seconds to upload...

It's possible that such bugs would be in the FPGA portion of the system. Fixing them could take a whole lot longer than you expect. The equivalent of the code/compile/load/test cycle for VHDL is often measured in hours.

[Kevin Sevcik]

Originally Posted by JesseK

The good news is, if they fix any beta bugs during a competition it will take mere minutes to implement and mere seconds to upload over the wireless to our bots Hopefully FIRST is thinking about this framework and will keep a special upload port ready for such bug fixes.

Minutes is probably a little optimistic. For a team with a laptop that just meets LabVIEW requirements, I'd guess 10-15 minutes, since a fix to some base level code is going to invalidate a fair number of things. And the field controller... That's going to be complicated enough that any fixes would probably be done over lunch.

Originally Posted by Alan Anderson

It's possible that such bugs would be in the FPGA portion of the system. Fixing them could take a whole lot longer than you expect. The equivalent of the code/compile/load/test cycle for VHDL is often measured in hours.

I'll note that on a 1M gate NI FPGA, I managed to implement the forward kinematics for a 3-DoF rotational joint device, X-Y-Z PD loops, and the transverse Jacobian to take those XYZ forces back into joint space, with lookup tables for all the trig functions. It only took 30 minutes to compile on my 2.0 GHz dual-core laptop. And if I changed some of my built in constants for joint lengths, it wouldn't fit on the FPGA anymore. So while I don't think it'll take hours to recompile the safety, PWM, and (hopefully) encoder logic on the FPGAs, it's definitely not going to be mere minutes.

[dcbrown]

If I understand correctly, to eliminate software from having to deal with hardware latency issues, the majority of the software that was implemented as an ISR within the PIC is implemented within the FPGA.

So for example, a quad encoder ISR on the PIC that had phase A/B input and tracked a count would be implemented within the FPGA and could have the added value of dealing with all four phase transitions on multiple encoders. Each encoder would have its own dedicated FGPA hardware to process it. The resulting exported data would be a h/w register with the current count.

Or the GTS sensor which had major latency issues a couple years ago due to the small time differential between forward and reverse pulse widths (<50us)? Implement the servicing in hardware and the software latency issue is no longer a problem, a good count can be maintained.

By committing the time critical processing to hardware, this eliminates the need for interrupts to service the hardware events and makes time sliced multitasksing that polls & processes the data viable.

You can then create independent tasks that poll, combine, and process the data for each of the various devices you want to use. With fast task switching time and task priorities of the RT/OS, the different data flow processing within the different task trees appears to be simultaneous. So, you could have one set of independent tasks integrating sensor information in an inertial navigation system to determine position and a different set of tasks "simultaneously" working on motor control, and a third set of tasks operating on operator input for control of a manipulator.

???

Not sure I like it, but I can see how you'd shift your design viewpoint to program such a system. For example, take the polling of each device out of the main interrupt routine of the PIC and create a separate polling task tree for each, then schedule them to run every so often as needed to initiate processing of the data...

The biggest plus is also the biggest minus:

+ the hardware servicing is committed to the FPGA - once the bugs are worked out it will work for all teams the same way.

- the hardware servicing is committed to the FPGA - if you want to tweak the servicing of the hardware or integrate data differently or add some weird/unusual sensor that isn't already allowed for, you're pretty much stuck as the FGPA is off-limits to teams. The whole lower level hardware layer has been abstracted and put off limits as a result.

Not a big minus. (I could whine about this, but it is what it is - learn, adapt, overcome.) I'll miss it, but can still teach how it works at the lowest levels.

Ok, so I'm babbling... Traditionally, interrupts are used for servicing hardware events, for time-based integration of data, and initiating event processing due to some hardware or software event (IR sensor detects wall in front of robot, stop!?!) The first two are very time critical, the last is mostly a convience of the interrupt structure and is less time critical.

The FPGA seems to address the first, and the 2nd *IF* that integration is implemented in the FPGA too, and event processing can be simulated by running independent polling tasks on a fixed schedule. The result is not exactly the same, but is probably close enough to work ok. I'm beginning to wonder if the new processor is fast enough

[Kevin Sevcik]

Originally Posted by dcbrown

So for example, a quad encoder ISR on the PIC that had phase A/B input and tracked a count would be implemented within the FPGA and could have the added value of dealing with all four phase transitions on multiple encoders. Each encoder would have its own dedicated FGPA hardware to process it. The resulting exported data would be a h/w register with the current count.

This is assuming, of course, that all this functionality is actually implemented on the FPGA for us. I'm not entirely certain that's a given. Of course, if it's not on the FPGA and we aren't allowed to implement it ourselves, I'm going to feel pretty darned silly still doing quadrature encoder counting using interrupts. Actually I'm going to feel pretty annoyed even if they implement the encoder logic on the FPGA for us, since it's pretty unlikely ever team is going to need the exact same number of encoders. Seeing as how a mechanum system with two feedback controlled joints would potentially need 3 times the interfaces as a two wheeled kit bot.

[Dave Flowerday]

Originally Posted by dcbrown

By committing the time critical processing to hardware, this eliminates the need for interrupts to service the hardware events and makes time sliced multitasksing that polls & processes the data viable.

...except only the things that FIRST and/or NI has thought of are committed to hardware. Since the FPGA is not user-programmable, any novel ideas or sensors which would need interrupt support are not possible - I think this is what Alan was getting at above. In my opinion this is a HUGE step backwards from the current control system and a very disappointing revelation.

A perfect example occurred this year - the FIRST-supplied IR receiver sucked, so Kevin whipped up some code that allowed you to do it on the RC with a simple receiver plugged into an interrupt port. I don't see how that would be possible in a new system that does not support interrupts. Sure, IR happens to be slow enough that you might be able to poll for it (remember, your polling frequency is going to be limited by your OS tick rate and if multitasking it may not be constant), but it doesn't take much imagination to think up other instances where something like this is not possible. And, if you start polling for enough things, all that nice extra MHz provided by the new processor will be chewed up doing menial tasks which are handled in hardware for free basically by a $5 PIC.

A few years ago myself and a few other mentors from Wildstang along with a few from the Technokats investigated using an optical mouse for tracking position. We did this by bit-banging the non-standard synchronous serial protocol used by the chip inside the mouse on the RC hardware. This is another example of something that might be difficult or impossible under the new system without interrupts.

[Kevin Sevcik]

Dave,

The RTOS-FPGA combo running on the cRIO does support interrupts, but you have to program them into the FPGA for them to work. They also appear to be a shared resource, so there's no precise guarantee when a particular one will be serviced. Basically, the FPGA would probably need to fire an interrupt whenever there was an edge transition on a digital input, and we'd have to use code like Kevin's Encoder 3-6 code supporting the PIC's interrupt on change port. But I'm really hoping we're not forced to go this direction. Given the availability of an SPI port and an extra 10/100 port, I think I'd be tempted to grab a semi-cheap FPGA demo board and have our students implement an encoder counter on it.

[Al Skierkiewicz]

I failed to point out in my early post a positive answer to one of my questions...Yes, there will most definetly be a default program that ships with the control system just like the IFI default software.

Also,
It might make more sense to split hardware and software discussions into two threads at some point in time.

[Doug Leppard]

I can't believe FIRST would release this without interrupt (encoder) support.

My recommendation for FIRST is give us the processor upon a paid registration to allow people to train on it and help find bugs that may be in it.

[Greg McKaskle]

Originally Posted by dcbrown

So NI_RIO_PEEK and NI_RIO_POKE (assumption) would work within kernel applications/tasks, but what about RTPs? Since opens are not traditionally shared across the RTP/kernel interface could RTPs do their own open on the FPGA image or will this type of macro call only work inside the kernel?

Is there a technical architecture overview document for the FPGA and NI I/O modules - essentially similar to VxWorks component API manuals? Trying to find any real tech information on this platform has been particularly frustrating. Especially since this is NOT a new platform. I don't need to know the particulars of how the FPGA/IO interface will be set up for FRC, but it would help immensely to know what a typical engineer recieves in terms of documentation, software templates, and information when they buy an cRIO IO module.

Realtime OSes don't necessarily have the same division of kernel and user modes. Specifically, the RIO peek and poke are allowed to be called from user code. As for documents that would be received if an actual engineering customer. To this point, the C/C++ for the cRIO has been for internal consumption. To this point, all cRIO customers have used LabVIEW.

The best source of technical overview info that I know of is http://zone.ni.com/dzhp/app/main.
Search for cRIO. The first five articles or so seem like they may be useful.

Greg McKaskle