Anyone interested in a Linux-based robot solution?

[Greg McKaskle]

This thread has hit on a number of topics, and I'm playing catchup, so I'll comment on what I can.

Protected Memory Model:
The VxWorks OS version NI adopted for the cRIO only supports the unprotected, flat memory model. Because it would be such a nice feature, we investigated RTP support for FRC, with protected kernel and process layers, but this would involve upgrading to a newer version of VxWorks, redoing the OS image, redoing many of the drivers, etc. Additionally, the NI engineers who had already investigated this found the overhead of kernel transitions would have a big impact on the industrial I/O customer. Keep in mind that as sold to industry, the cRIO is a LV target. LV is a safe language, so running it on a safe OS benefits the NI engineers and the professionals who are writing their own .out extensions in C/C++, but isn't needed by the general user.

Since RTP turned out to be expensive to build, and unlikely to happen soon, I'd like to see the lib ported to a protected OS and run with an emulation layer. I'm currently looking at doing this for LV. This will be library level, not the entire OS, but I think it will be appropriate for finding logic bugs, compile problems, and teaching exercises as long as the feedback mechanisms are good.

As for the availability of cRIOs, and other control systems, NI and its suppliers are doing what they can for the pricing of the existing HW. Since the system will evolve, I'd love to hear the value of different features. I think everyone will agree that the industrial temperature spec has high cost and is of little value to FRC. Feedback on the other aspects of the HW that makes up the control system will be invaluable for FIRST to select or design the next control system.

Finally, there have been vague statements about how WPILib should be improved, but nothing specific enough to be actionable. Let me give my view of what WPILib is and is trying to be, then once again, ask for feedback.

My metaphor for WPILib is that of a big community swimming pool. It needs to have several pretty well defined zones that offer different levels of challenge and safety. It needs to support wading, general swimming activities, and for the daring, diving in head first trying to touch the bottom of the pool.

The framework and examples hopefully act as the wading zone. Lots of hand-holding, lots of prebuilt code that does simple things, allows for safe exploration as well as retreat. Since it exposes source, hopefully it serves to learn the next zone.

The general swimming is the actuator and sensor libraries themselves. You leave the framework entirely, or heavily modify it to add in and use many new elements in the library. Since it is distributed in source, it also supports exploration beneath the surface to see the dependencies, and the notification mechanisms that make up the higher level interfaces.

The deep diving, at least in LV is the NIFPGA interface. This exposes accumulators, triggers, and other raw forms of I/O which the general and wading layers are built from. It should allow for alternate implementations, alternate frameworks, and external sensors and actuators to be added in. At the moment this is as deep as the pool can go since the FPGA implements the safety mechanisms, and it doesn't seem appropriate for teams to modify the safety layer. Ideally, that layer will be opened up too, showing how to do encoders, PWM generation, triggers, integrators, etc on an FPGA.

Obviously, the current implementation is only one potential way of doing things, and honestly, the pool was opened while still under construction, and probably always will have some areas under construction. Again, feedback is super important. All of the zones were built at the same time, not enough documentation exists, and there are many alternate implementations that should be considered.

I've created three feedback threads on the technical/control systems/FRC control systems and on the technical/programming pages. Hopefully that will allow for the feedback to take place without intruding on this thread.

Greg McKaskle

[dcherba]

A type example is the way protocol stacks are built and copied. In the realtime critical environment we utilize large but fixed size circular buffers that have overrun protection and full 4 level handshake on interface that does not require coping blocks of data from thread to thread. The interupt handlers are designed to minimize context switch and often only require a few bytes to be saved and restored reducing the overhead of the time slice needed to work on a packet. This style avoids the mutex issue and what often leads to thread lock problems.

It is amazing when I count the number of thread lock events that occur every 24hours even in robust systems.

While I agree with the robust abilities of many small hardware platforms the idea of mounting the crio on insulated materials to protect it goes against every real world experience I have with building industrial tough computers.

I am just returning to the FIRST community after finishing graduate school and completely changing from the controls field to bioinformatics. I have not started looking too deeply at the crio and the wpilib but trying to use the video camera and seeing the protocol induces watchdog timer errors was disturbing.

I applaud all of you that have helped make the wpilib a reality and like my earlier post think it is great. Some of the crio hardware libraries I looked at really need to be redone in order to be reliable. The camera software relies on some low level routines that are absolute time slice hogs and until they are fixed no amount of low resolution will stop the 50-100millisec slices.

As I get more up to speed I will start posting suggestion and fixes. I wrote my own timer class this year which functions more like timers in the PLC's and it allowed our control program in autonomous to be extremely small and robust. Hope to get that posted soon. It was interesting to build that because many of the gettime functions that are standard on every os did not operate as expected or documented. What should have been a two hour project turned into 3 days of real head scratching and looking at more header files than I knew existed for stdio. That is an example of what really eats your time even when you are very experience let alone a student attempting these types of things for the first time.


Dave

[biojae]

Originally Posted by Greg McKaskle

The deep diving, at least in LV is the NIFPGA interface. This exposes accumulators, triggers, and other raw forms of I/O which the general and wading layers are built from. It should allow for alternate implementations, alternate frameworks, and external sensors and actuators to be added in. At the moment this is as deep as the pool can go since the FPGA implements the safety mechanisms, and it doesn't seem appropriate for teams to modify the safety layer. Ideally, that layer will be opened up too, showing how to do encoders, PWM generation, triggers, integrators, etc on an FPGA.

Is there a way that any of the implementation can be exposed?
Maybe only the things you mentioned (encoders, PWM, triggers, etc.) without comprimising any of the safety layer.
Even a screen shot would be better then the virtual "black box" that it is now.
Some knowledge of how the system works would help diagnose some issues, even if they can't be directly fixed (indirectly through bug reports).

[Greg McKaskle]

If you are pretty convinced there is a bug in something, I suppose we could give out a screenshot to help get to the bottom of it, but again, since this is the safety protocol, we wouldn't want any enterprising team to "optimize" it, resulting in someone getting hurt. So whatever we'd do we'd keep FIRST in the loop.

Trust me, NI and especially Xilinx would like you to have access.
Greg McKaskle

[biojae]

Originally Posted by Greg McKaskle

If you are pretty convinced there is a bug in something, I suppose we could give out a screenshot to help get to the bottom of it, but again, since this is the safety protocol, we wouldn't want any enterprising team to "optimize" it, resulting in someone getting hurt. So whatever we'd do we'd keep FIRST in the loop.

Trust me, NI and especially Xilinx would like you to have access.
Greg McKaskle

I haven't noticed any bugs, at least in what I have experimented with,
but as a non-FIRST project I would like to start working with a FPGA.

I understand the concern for safety, though it shouldn't be in every VI especially in the ones used for input (Accumulator, encoder, etc.).

Could the ones that wouldn't comprimise the safety layer be released somehow, or somewhere(Firstforge)?

[Radical Pi]

Not intending to derail the topic, but I guess it is somewhat related.

This is mainly to Greg: Is the possibility of including the FPGA module of LabVIEW in the FRC edition being considered? Even if FPGA code mods aren't allowed by FRC rules (similar to how the grey jags were made with CAN capabilities that weren't allowed in 09), it still would be put to use by teams doing off-season stuff with the cRIO that require FPGA changes (using different modules comes to mind). Even if it isn't FRC legal, the learning and experimenting with it (something I love to do) is great for the programmers (and great marketing too ). And on the safety problem, what about just giving us the regular code with the safety things stripped out of it?

[Greg McKaskle]

Personally I think the FPGA tools would be a great addition for offseason projects.

Perhaps you want to measure gas exchange in your local rain forest, or connect it to your neighborhood hadron collider, make an engine controller for your motorcycle, or send it up in a rocket to perform flight control, ... just as long as it doesn't involve driving somewhat powerful robots without the appropriate safety mechanisms.

At the moment I'd say your best bet is to write a letter or build a presentation explaining why you'd like the tools and asking for a license. Treat it like a fund raising event. If you need additional ideas, poke around on http://www.ni.com/solutions/. Send the request to either your local sales representative or to someone at NI corporate in the academic department. If you can't find someone, send it to me.

Greg McKaskle

[taichichuan]

Can anyone provide some idea of what types of safety protocols are provided in the FPGA? I don't need the specifics of the code, but if we're trying to simulate the system in software, we need to have an understanding for what safety we're trying to implement. Is it like a current limiting governor, or a sample rate limitation or what? And, did we implement any of these safety interlocks before the cRIO came into the picture? if so, how did we do it before the slick FPGA?

TIA,

Mike

[AustinSchuh]

I'm going to give my best educated guess, which will be pretty close.

Before the cRIO, the IFI system had 2 CPU's in it. One CPU dealt with communications and the safety protocols. Eg: if it didn't see a packet from the driver station within a set amount of time, it would put the robot in disabled mode and disable all the motors. We weren't allowed to change the code on that CPU, and could only change code on the other CPU in the system.

On the cRIO, I'm pretty certain that the FPGA kills the motors if it doesn't see a packet from the Driver Station within enough time, or gets a disabled signal, or gets an e-stop signal. So, it's serving the same purpose as the IFI system's second CPU. It probably also implements the watchdog. This makes it so that if we totally mess up the software as users, if someone tries to kill the robot, the robot will stop.

To sum it up, you want to have some sort of safety system that won't rely on end user code (that can have bugs in it) to stop the robot when it looses comms, gets a disabled signal, or an e-stop signal.

[taichichuan]

Originally Posted by AustinSchuh

I'm going to give my best educated guess, which will be pretty close.

Before the cRIO, the IFI system had 2 CPU's in it. One CPU dealt with communications and the safety protocols. Eg: if it didn't see a packet from the driver station within a set amount of time, it would put the robot in disabled mode and disable all the motors. We weren't allowed to change the code on that CPU, and could only change code on the other CPU in the system.

On the cRIO, I'm pretty certain that the FPGA kills the motors if it doesn't see a packet from the Driver Station within enough time, or gets a disabled signal, or gets an e-stop signal. So, it's serving the same purpose as the IFI system's second CPU. It probably also implements the watchdog. This makes it so that if we totally mess up the software as users, if someone tries to kill the robot, the robot will stop.

To sum it up, you want to have some sort of safety system that won't rely on end user code (that can have bugs in it) to stop the robot when it looses comms, gets a disabled signal, or an e-stop signal.

OK, that can be dealt with easily enough through a uP like the ATMEL ATMega. Also, Xilinx has recently announced some hard-core ARM and PPC440 processors w/ 10000+ FPGA gates on them that could be used for the safety and still run Linux. I was just trying to make sure there wasn't something more sophisticated.

Thanks,

Mike

[jhersh]

Originally Posted by AustinSchuh

To sum it up, you want to have some sort of safety system that won't rely on end user code (that can have bugs in it) to stop the robot when it looses comms, gets a disabled signal, or an e-stop signal.

You got it.

Originally Posted by taichichuan

OK, that can be dealt with easily enough through a uP like the ATMEL ATMega. Also, Xilinx has recently announced some hard-core ARM and PPC440 processors w/ 10000+ FPGA gates on them that could be used for the safety and still run Linux. I was just trying to make sure there wasn't something more sophisticated.

It can certainly be done on a different hardware platform... in fact the new Xilinx parts sound like a pretty similar architecture.

[Greg McKaskle]

Rather than go into the details, not sure that I'd do it very accurately anyway, find a way for the "dangerous" outputs to have a trusted element that can shut them down when things aren't updated or when the field explicitly requests it.

On the cRIO, there are several trusted pieces that pose challenges to each other to make pretty sure they are legit. The FPGA ultimately controls the PWMs, relays, and solenoids, CAN doesn't go through the FPGA, but includes info from the FPGA to generate the keys for the jag heartbeats. If/when this project goes further, ask for more details.

Greg McKaskle