Comments/Complaints on NI control system

[Mike Soukup]

Originally Posted by Chris Hibner

As far as we're concerned, our autonomous is set up so that it can only run once no matter what. Once it starts, you're forced to cycle power to get it to go again. So for us, the STOP button is just as good as the disable since we're going to have to reboot to do another test anyway. I guess if we test using LabVIEW's play button, we can just push the LabVIEW stop and then play again to clear the memory and that is much quicker than a full reboot, but with deployed code it doesn't matter how we end auton.

You better make sure that your drive team completely understands this limitation. At an event I was watching this weekend, the robots were enabled for autonomous, the foghorn sounded .5 second later, then the robots were re-enabled .5 second later. It was probably a mistake by the field operator who fat-fingered the stop button. My first reaction was "didn't that just mess up everyone's autonomous? What if they can only run autonomous once?" If you have the limitation that Chris mentioned, you'd better be sure your drive team raises a stink if that ever happens in your match.

Greg, I'll be happy if someone fixes the bug I reported a month ago. Can anyone else confirm that this is a problem with their setup?

[Bharat Nain]

We've already posted our complaints. At this point, I only wish the system would reboot significantly faster. It takes us an average of 35-45 seconds to reboot and that is extremely costly time when we have to perform a repair by calling a timeout during eliminations.

[Chris Hibner]

Originally Posted by Mike Soukup

You better make sure that your drive team completely understands this limitation. At an event I was watching this weekend, the robots were enabled for autonomous, the foghorn sounded .5 second later, then the robots were re-enabled .5 second later. It was probably a mistake by the field operator who fat-fingered the stop button. My first reaction was "didn't that just mess up everyone's autonomous? What if they can only run autonomous once?" If you have the limitation that Chris mentioned, you'd better be sure your drive team raises a stink if that ever happens in your match.

Greg, I'll be happy if someone fixes the bug I reported a month ago. Can anyone else confirm that this is a problem with their setup?

We've been doing it that way since 2003, so Ken is well aware. We decided a long time ago to keep it that way to avoid the safety issues Dave brought up (and most years we allow autonomous to run into teleop depending on the game so our auton scripts can often last longer than 15 seconds). They know that if a match is foghorned, they need to cycle the power to the robot. The FTAs complain a little, but they should expect that some robot reset is going to be necessary if they blow a match dead.

We have the same issue on our gamepad, but I really don't want the bug fixed at this stage because I don't want to re-write things now that we have our software working with the current limitation.

[Kevin Sevcik]

Ditto the comment on selective memory. I vividly remember the "7.2V" bug biting us at Great Lakes. And the horrible growing pains switching from the even older STAMP controller to the newer PIC controller. And so on and so forth. Every new control system has growing pains and atleast some people pines for the Good Ol' Days. So for a change in this thread, I'd like to point out (some) of the (numerous) positive aspects of the new FRC Control System:

• That you DO have the option of probing things live in Labview/Windriver for debugging. Slight bugginess in a feature not even available on IFI? Score one for the new system. I can tell you I won't miss having to litter my code with printfs.
• Real hardware encoder inputs. No more worrying about high resolution, high speed encoders jamming your processor with interrupts. Plus you can do more than 4 now.
• Real hardware sampled analog inputs. No more worrying about your interrupt routine to sample and average your analog inputs. As a bonus you can sample them at ludicrous speeds as well.
• Expanded speed controller abilities through the Jaguars + CAN bus. Now we have speed controllers that at linear and have smarts approaching the entire IFI controller. You don't even have to spend processor time on PID loops.
• Expanded peripheral counts. No, I haven't needed 20+ speed controllers and servos yet, but isn't it nice that it's available?
• For that matter, no more digging around in PIC datasheets to figure out just how you're supposed to use peripheral XYZ.
• Pre-made extremely useful dashboard with highly useful error messages and robot feedback. No more deciding which user byte is the most useful to you at any particular time. No more punching the user button to find out what your battery voltage is.
• Actual video on said dashboard. No more swapping camera cables back and forth trying to figure out what on earth that silly CMUCam is fixated on this time.
• Increased ease of programming. Knock it all you want, I can have a newbie team competently programming in Labview while you're still teaching your team the intricacies of C syntax. "You mean = and == are different?" vs. "You mean I just wire these little pictures together?"
• No more 8/16 bit integer math! Doubles for everyone! Say goodbye to all those pesky under/overflow bugs. Aren't you happy you no longer have to worry about the order of your multiplication and division and the loss of precision you're incurring? If I never see another typecast in our robot code, it'll be too soon.
• Download-less software testing and debugging (in Labview anyways). Wanna test that fancy new autonomous mode? Just hit the run button. Made your robot explode? No worries, just reboot and everything's back exactly as it was.
• No more backup batteries to worry about. Yes, this is a function of the new PD board, not the cRIO, but you're lumping it all together anyways. Also, new highly compact PD board with WAGO connectors. Take that giant copper maxi breaker blocks.
• Decentralized wiring. Isn't it nice not having to cram all of your wiring into the border around a 4x5 box? That also needs to be highly visible and accessible? Plus, swapping a cRIO out of a bot is loads simpler than swapping an IFI controller out.
• Development software that's actually licensed to be installed as many computers as you'd care to install it on. Yes, I know you didn't care that MPLAB was a single seat license. But now you can continue not caring about licenses and still be perfectly legal.
• The driver station doubles as a development platform. I'm sure you have to dig you way out of a pile of laptops if you open the wrong door, but I've run into many rookie teams that couldn't do a thing with their robot simply because they had no computer. Now one comes in the kit, ready to go for programming.
• Finally, no more analog joysticks and the hacks accompanying them! The fact that I no longer have to hoard '04 and earlier flightsticks? Excellent. The fact that I can now run down to the local CompU Buy Radio City and grab a joystick/steering wheel/game pad off the shelf and have it just work? Awesome. The fact that I never have to trim another joystick? Priceless.

So, I think there's a rather lot of positives to the new control system that help to balance some of the annoyances. Frankly, my largest peeve with the new system is the sheer cost and the fact we don't get a new one on an annual basis.

I find it difficult to really get annoyed with "lengthy" compile/download times. As long as they are under 2-3 minutes, I could really care less. If you're downloading code so often that 2 minutes vs. 30 seconds KILLS you, then you're doing something wrong. And you should probably just be hitting the run button to avoid all that pesky downloading and rebooting, as your code clearly isn't in a finished state.

Similarly the back-to-back matches. There's never a scenario where this is a problem. You always have more than 5 minutes between matches, even in the finals. If you can't organize yourself enough to reset your robot in the allotted time, you have other issues.

Also, to your autonomous example. If you're still in development on your autonomous mode, you're not deploying code, you're running it in RAM. Which is a lot quicker. And if you're really clever, then the important variable in your autonomous code aren't constants. They're linked to front panel controls that you can change at whim without changing the program a bit. Time to change a front panel variable? About 5 seconds, plus restarting your DS in auton. Say about 5 times faster than your IFI controller.

Point being, the new control system gives you loads and loads of new tools and features. More than I can think of off the top of my head. You really have just two options:
1. Learn. Figure out what new features you can use to make your coding easier. Embrace the new system and delve into it as deeply as we all delved into that PIC processor. I'm not sure what the cRIO equivalent of the Counter0 control registers is, but there's gotta be something as esoteric and useful. Go explore and find it.
2. Stagnate. Continue to pine for the Good Ol' Days and resist working with the new system. Grumble about how much better things used to be. Don't bother with all those new-fangled debugging techniques. Declare how the youth of today don't know how easy they have it. Back in the day, they only had integers for math. Five typecasts and three overflow checks from the joystick to the victor it was, if you were lucky.

Oh. And shake your fist too. That always shows 'em.

[Greg McKaskle]

Greg, I'll be happy if someone fixes the bug I reported a month ago. Can anyone else confirm that this is a problem with their setup

We have the same issue on our gamepad, but I really don't want the bug fixed at this stage because I don't want to re-write things now that we have our software working with the current limitation.

Everyone has that issue with the gamepad, and yes it is a bug, but one that wasn't found or reported until well into the season. The decision was made to leave it alone. It is a bug with compatibility with the previous driver station, but not one worth the churn of a mid-season change in functionality.

At the beginning of the season, we were hoping to move to a new joystick API, one which would support higher resolution, more axes, more buttons, and more traditional POV controls. The bug was introduced trying to map the MS HID data to reproduce the behavior of the 09 DS. The amount of mapping required go from named axes to a limited number of numbered axes was a royal pain. Anyway, the '11 API can certainly fix the bug or replace it completely.

As for the state to transition to after Auto, the code can easily be modified to do this, but it would be up to FIRST to decide what to adopt, or to decide if this is a team-settable option.

Greg McKaskle

[apalrd]

@Kevin:
I can see your advantages to the new control system over the IFI processor, but I have several more comments:

1. I use probes all the time. I often crash Labview trying to open probes right after downloading, which is really annoying.
2. The IFI processor had 6 hardware interrupts, enough for 2 1x and 4 2x encoders or less with different decoding. The new processor has 8 hardware counters, but 4x decoding takes two counters. Not a big improvement, but no more ISRs
3. Sampling a gyro from Process_Data_From_Master_uP seems to have enough resolution and speed for everything to work fine.
4. I agree on the Jaguars, although the jaguars are actually more powerful than the entire IFI processor. However, the "safety" features included (shutdowns and lockouts on small errors) sent us to Victors this year.
5. I like the more IO but not the way the IO is handled - especially how the whole system has max. 7 power feeds to a whole bunch of separate boards on a whole bunch of separate modules on a cRio chassis. It could be a little more integrated.
6. I like the Dashboard, but the camera comm seems to cause problems with robot comm so I don't like camera comm.
7. I have several uint8 and uint16's in our robot, and typecasts associated with them. Most of them are involved in kicker set distance, as it prefers doing math in feet.
8. I often do run stuff by pressing the play button. And it takes me around 20 seconds to download over wireless, but using controls saves a bit of time.
9. It looks to me like the new PD board is quite a bit larger than the single MAXI block and ato6 panel on BuzzXIII, and a MAXI panel and ATO panels fit much more nicely into most of our framing. I also hate WAGO connectors because I am always loosing the WAGO tools and every little screwdriver we have, and the ability to get enough space on the side of the PD board to fit the WAGO tool in is often not enough. It does weigh less, which is nice.
10.Since most of our wiring is crammed around the Digital Sidecar, it ends up having twice as much IO in half the space (although it is burried deep in the belly of the robot, not visible remotely)
11. The LabVIEW splash screen dosen't even fit on the Classmate. I tried developing on it. The keyboard is just too small. I like the attempt of FIRST, but not the laptop.
12. Although I do not miss the analog joysticks, I do miss the raw analog IO. The new Cypress board is probably the single thing I hate most about the new system, since it A. runs on 3.3v, and especially B. the Driver Station has many issues finding it when combined with FMS.

As for advantages, the big ones I have that change the most, not including language.
1. Realtime trig without lookup tables
2. Floating-point math
3. Virtual multithreading
4. That's about it.

As for my autonomous example, I have found myself in many situations at competitions where there is a minor bug in autonomous, such as going a little bit too far or kicking a little too high, which is caused by a change in the reaction of something such as the kicker as it is worn in, and I must do a full deploy since it will be used on the field.

@everyone:
Many people probably see this thread as a request for the IFI processor back. That is not my intentions. I am saying that, while the cRio and associated control system has many bugs that the IFI system dosen't, neither one is perfect. It would be nice if only we could have a control system that was as fast, small, light, and reliable as the IFI system but more powerful. We don't really need the power of the 400mhz PowerPC, but having floating point math and LabVIEW really helps alot. If having a large several-pound cRio with many times the necessary power is the solution, then so be it. Just make the rest of the system work as well as it does. Having a dedicated driver station dosen't run Windows, lockout users after every match, and actually finds all of its inputs every time would be a nice start, followed by faster download/bootup times and more reliable radios. And maybe a solution to those ejecting analog modules other than glue.

[Greg McKaskle]

Something that I would like to see improved is safety at the driver station...

There are many cases where the robot auto-disables, but avoiding a double auto never came up during the beta. If you feel strongly, please post or discuss directly with FIRST. Clearly safety is a priority, and this is their call. BTW, I don't get the distinction between suggestion one and two.

As mentioned by another response, the spacebar acts as the Disable button. It is polled like a joystick button and should work no matter the key focus. When attached to a field, field controls take precedence, and F1 serves as a way to redo joystick enumeration and is the only key shortcut. As for the Stop button, it initially acted as a disable, but again, during beta requests were made to simulate what would happen on the field.

Similarly, requesting the FMS Lock have another way out is something that should be made on the FIRST forums. With encrypted radios, I don't really see a way to wirelessly connect to the robot anyway, but again, it is their call. I think I see the issue with the code that can allow for unlocking. I'll notify FIRST and see what they want to do about it.

Greg McKaskle

[Greg McKaskle]

My primary concern with the current system is the long reboot time for the DS...

When this happens, do the joystick LEDs and list respond to the gamepads? Pressing any button on the gamepad should turn the LEDs blue.

As mentioned, F1 was added in an update to reenumerate the joysticks for teams that find they need to swap or plug in a joystick after the auto period. That will close all handles to joysticks, redo the enumeration, and reopen according to the list. Even if this fixes the issue, if at all possible, please help document how it gets in that state.

If anyone else finds a way to provoke joystick loss, please post steps. Everyone wants a robust DS.

Greg McKaskle

[Mike Copioli]

Originally Posted by Greg McKaskle

When this happens, do the joystick LEDs and list respond to the gamepads? Pressing any button on the gamepad should turn the LEDs blue.

I did not notice the LEDs. I will be sure to check if this happens again.

Originally Posted by Greg McKaskle

As mentioned, F1 was added in an update to reenumerate the joysticks for teams that find they need to swap or plug in a joystick after the auto period. That will close all handles to joysticks, redo the enumeration, and reopen according to the list. Even if this fixes the issue, if at all possible, please help document how it gets in that state.

I was not aware that F1 also re-enumerated USB. Good to know. As I am sure you know reproducing this kind of issue deterministically will be very difficult.

.

Originally Posted by Greg McKaskle

Everyone wants a robust DS.

I do not know if this will be possible using a Windows based solutions. Windows is not intended for target specific or mission critical tasks such as a FIRST competition. Know matter how much you slim it down it will still take an excessive amount of time to reboot. At events, teams are given a very limited amount of time to troubleshoot between matches. Every second of every minute counts when you are only provided three minutes between semifinals and finals. Add the reboot time of the DS (up to 2 minutes) with the time the crio takes to boot (40 plus seconds) with the time needed to deploy code and you do not have enough time to make last minute changes to auton or other robot functions that may be critical to an effective strategy. The part that was really frustrating was that fact that no one from NI was at the FLR to provide support, this was something that you did not have to worry about with IFI.

[apalrd]

@Greg:

1A: 30s is death in the pits. The IFI system could be up and running in under a second on tether, 5s on radios. 30s or more is waaaay too much time.

1B: My download times appear much greater than you stated, although I could attribute that to the 5 people standing behind me saying "WHEN CAN WE TEST THE ROBOT!!!! GET IT DOWNLOADED NOWWWWW!!!!! GET IT THERED AND MOVE THE ARM NOWWWWW!!!!" while waiting for my code to download.

1C:The IFI system sent packets every ~26 ms. The RS422 900mhz radios sent exactly the data that was needed in one fairly small packet to and from the robot. Data to the robot was 26 bytes in one packet, data back was 2 26 byte packets (most of the data back was exclusively used by the Dashboard). I do not know the latency on it, but it wasn't much. As for the new radios, we had noticeable control lag when using the camera on the Dashboard, but only on the field. This lag was aprox. 3 seconds. I can't say for sure what the cause is, but I can say it appeared when we added the camera and was fixed after we removed the camera, leading me to believe it was the camera.

IIA: We have to reboot Driverstation after every match to get to loose the FMS lock (another annoying feature) but sometimes we do reboot it. If it goes on to the field shut down, it will not find the Cypress board and we are dead.

IIB: This is why we are looking for a solution to the Cypress board, and this almost cost us 2 matches.

IIC: We come off the field with back-to-back matches and have to change all four bumpers, tether the robot to move the arm, change the battery, and get it ready to play again in 5 minutes. 45 seconds is DEATH.

III: Our radio had a loose terminal during one match (duck tape fixed this) and after loosing comm once it had no possibility of ever gaining it again during that match. We tied that match after playing 18 seconds.

IV: The cRio has a 400mhz processor, and while it has aprox. 30% free space (shown during Deploy) it is still not using nearly the power it has unless doing vision processing. We had a crab-drive last year using this system. We needed to do some trig calculations. Even with the IFI processor back when we built a crab drive in 2005, we had plenty of power to do trig calculations with lookup tables and it worked great. That was an 8-bit PIC with far less power or code space then the cRio, but it performed the same task just as well.

IVB and IVC: See IIC

V. If I place a probe on a VI before the "Robot Code" light on the Classmate comes up, LV will crash on my PC. Guaranteed. Always.


@Dave: I would like to see it reset to Teleop after comm or code loss, but not after disable. I often set controls on my VI's for commonly changed parameters (e.g. distance or kick force) and change them while the code is running. I often run autonomous for many times in a row when debugging it, without ever changing to Teleop or downloading new code.
Spacebar disables the robot without e-stopping it. The e-stop button killing code annoys me too, but I have actually used it before (the Toyota bug) so I wouldn't get rid of it.

On the Stop button acting as an E-Stop: Sometimes if the robot spontaneously acts wierd and needs to be stopped, it is nice to be able to probe everything to see what it is trying to do and find your problem. We have an arm on our robot with 800:1 off a CIM, and it has enough torque to cause damage to many parts of our robot. If it starts destroying something, we want to be able to probe it to see where our problem is. We had this exact issue, and found that the analog module on the cRio actually ejected from the cRio.

@Greg again: f1 re-enumerates JS's? Does it find the Cypress board again?

@Everyone: Did anyone notice how nobody EVER debugged comm or firmware things EVER with the IFI system, and how it just always worked? I do not see the cRio or the entire control system as designed to handle the rigorous environment that is FIRST, nor see it as better than the IFI system. Everything, except vision, that we are doing now could be done just as well on the IFI processor. Vision could be done too, using the CMUcam. Now, we are doing vision and other complex processing using many times more code space on many times more expensive equipment, yet achieve the same result, sometimes worse. I am not against using the cRio as a robot controller, most of the cRio based things except for download and bootup times are quite nice to work with. The whole system is not designed to work together; it is a bunch of off-the-shelf stuff that people are trying to make work together. None of it is designed for competition robotics, and can't handle the demands we place on it like the dedicated robot control system provided by IFI could. This all leads to another problem: Since the control system is not made by any one company, it is hard to lay blame for failure on any company.

[Mark McLeod]

Originally Posted by apalrd

@Everyone: Did anyone notice how nobody EVER debugged comm or firmware things EVER with the IFI system, and how it just always worked?

I never noticed that, seeing as I was involved in three or four IFI emergency beta firmware debugging sessions in the middle of the season...

So soon we forget how many IFI Master Code upgrades we performed over the years. I remember version 15d...
Selective memory is a killer.

No system is perfect. What you should look for is corporate willingness to recognize and react quickly to solving issues when they do arise.

[apalrd]

Originally Posted by Mark McLeod

I never noticed that, seeing as I was involved in three or four IFI emergency beta firmware debugging sessions in the middle of the season...

So soon we forget how many IFI Master Code upgrades we performed over the years. I remember version 15d...
Selective memory is a killer.

No system is perfect. What you should look for is corporate willingness to recognize and react quickly to solving issues when they do arise.

That's not what I was talking about. I am not referring to the need to update firmware, but bugs in the firmware. I have no problem imaging the controller.

Let's look at some fictional robots:

Imagine you have a robot that has a driver joystick and an operator box, using the IFI control system. Suddenly, you plug it in to the Competition controller and BAM - ports 3 and 4 don't work, but ports 1 and 2 do? OH - you just have to power cycle it a few times the right way and all is well. That's what it's like with the Cypress board, except it takes a long time to power cycle since it runs Windows.

Now imagine you have another robot with an IFI controller, and it controls an apparatus that has a potentiometer for position sensing and a motor to move it. Now you drive over a bump, and suddenly all of your analog ports fall off. Now the robot is trying to kill itself because all of its sensors are gone. I have seen analog modules fall out of the cRio while going over a bump, even though they were completely inserted and latched into the cRio.

And lets compare two robots. Both have a slightly loose connection to their radio. This is a likely scenario with the 2009 Gaming adapters, since the power connection does not lock, but unlikely with the IFI system since all of the connections use DB9 connectors which have locking screws. One has the NI control system, one has the IFI control system. Both robots drive over a bump. Both loose connection. The robot with the IFI system will disable itself because it has lost comm, and try to communicate as soon as the radio is functional again. Within 5s of radio power-up, it is up and running again. However, the other robot will also disable itself, but since it dropped too many data packets while connected to the FMS, it will sit dead until somebody reboots it. Sucks for that robot.

And another comparison of robots. This time, both robots encountered an error and need to be rebooted. It doesn't matter what for, they just do. 5s after being booted, the IFI processor will find its OI if the OI is available and have finished initiating radio comm. As Greg measured, it takes the cRio ~30 seconds to boot. The IFI robot has already finished Autonomous mode and scored a few balls since the other robot is too busy booting up to play defense.

And then, the two robots have back-to-back matches. They must tether, boot, move an apparatus to be within the starting box, and change the battery in 5 minutes. The physical process of tethering takes aprox. 30 seconds each, plus boot up times, around 30s to move the apparatus, and a minute to change the battery. The NI-based robot also has to restart Driver Station since it is FMS locked, which takes around 1 min 15 secs. The IFI robot can be ready in 2 min 5 seconds, while it would take the other robot take 4 min 45 secs to do all of this.

Now look at what these two robots can do. Both have swerve drive, with feedback on the front and rear wheels independently. They react almost identically to driver input. There is a solid-color vision target in the game. The IFI processor uses its CMUcam Co-Processor to find it and drive to it. This causes almost no lag on its part. The cRio connects via Ethernet to its camera, downloads a new image, processes it, and drives to the target, while causing the swerve drive controls to freak out from the non-constant loop time (even though vision is running in a separate virtual thread). I have seen vision processing mess up PID calculations. That's exactly why we didn't use the camera in 2009 with our swerve drive.

And finally we will see what happens when the programmer must change a small variable in Autonomous mode. Since the IFI processor is programmed in MPLAB C, the programmer would have to change a variable or function call in the code. They would then have to compile the code, a process that takes around 10 seconds. They would have to plug a serial cable into the robot controller, and download the code. This download process takes around 15 seconds. They would then unplug their serial cable and press the RESET button. In 35 seconds, not including the time it takes to connect the serial cable, they are ready to go. With our other robot that has a cRio, the programmer makes the same function call change. He then builds the code, which using Greg's benchmarking would take around 2 minutes. He then must tether to the robot, which is not included in the time. He then must wait for the robot to boot up, a process which Greg claims takes 30 seconds. Then, he must Deploy his code. I have never ever seen my code download in 18 seconds, so here is my calculation for download time: aprox. 30 seconds to load code + 15 seconds to wait for cRio + 10 seconds to do something unknown before asking me to reboot. Total time: 55 seconds. After another reboot, that brings the grand total minor code change time for this robot with a cRio to 3 minutes 55 seconds. Unacceptable. That's 3 minutes 20 seconds more than the IFI processor, or a bit more than the time it takes to run one match, or a 671% increase.

[Radical Pi]

On the topic of code download and build times, I have seen nothing like that on C++. It probably is beacuse most of the large code is in a precompiled library instead of building every single time you build. I'm don't know much about what can or can't be done with LabVIEW, but if a cache of the compiled code could be kept of the NI libraries, it would greatly speed up builds (From my experiences with the LEGO NXT, it also seems to be in the nature of LabVIEW and variants that programs are unnecisarrily large compared to C equivalents)

@Greg, you said earlier you saw an unexplained delay in camera images showing up. That delay appears to be from the camera taking much longer to boot than the cRIO code, delaying images. This is actually an important time delay for our bot, because in the time between cRIO boot and camera, the console is flooded with errors after camera init is called since it can't connect to the camera, rendering CAN code inoperable due to timing delays messing up the system, as well as a noticable lag in the controls of the robot (probably due to the increased UDP traffic through netconsole)