Comments/Complaints on NI control system [Archive]

apalrd

05-03-2010, 22:46

Comments/Complaints/Annoyances of NI . I am not trying to complain, but it seriously bugs me when there are 10 minutes to get ready for the next match and I have to boot the robot, tether, download new code, reboot it,test it, and change a battery and possibly bumpers. The NI/WPI code dosen't totally suck, but it has many issues that make it clear that the author of the code obviously has never been on a pit crew at a FIRST competition. They have no idea what sort of timing pressure we are under, and I am sure they would have designed their software differently if they did.

I. Timing:
A. Boot-up: The old IFI system could boot and establish radio communication un under 5 seconds. The new system takes aproximately 1.5 minutes to establish communication, begin code execution, and be ready to use.
B. Code: The Old IFI+MPLAB system could build,deploy,and reboot in under a minute. The new system takes aproximately 2-3 minutes to do a full build, plus another 2-3 minutes to download, plus must completely reboot, which is another 1.5 minutes.
C. Network delay: The old IFI system had a simple processor on the OI communicating over 900mhz radios directly to another simple processor, with one level of communication via RS422. The new system has a PC running software, communicating via Ethernet to the FMS, which is communicating wirelessly to the robot, which is translating it back to Ethernet, and then unbundling the giant data packet. This leads to much hetwork delay, as the entire alliance station must share a single Ethernet line.

II. Operator Interface:
A. Boot times: The old OI could boot and establish comm in 5 seconds. The new system must do a full boot, plus run several large applications which take a long time to load. If you have to reboot your operator console on the field, you are seriously screwed.
B. Cypress board and related communication: The cypress board runs on 3.3v which is annoying since everything else runs on 5v. In addition, should the Cypress board loose connection, the Classmate will continue to send old data to the robot, making it impossible to detect the loss of the box. In addition to all of this crap, booting up the Classmate while connected to the FMS will lead to a loss of the Cypress board, which is only fixable with TWO reboots of the classmate, meaning teams faced with this must choose between risking it and possibly not playing because it took them too long to setup, or not having their Cypress board. On Friday of our first district we encountered this scenario twice, and went without the Cypress board.
C. FMS Lockout: On the old IFI system, removing the Competition cable would lead to a robot that would be happy and ready for use. Now, after connecting to the FMS, the Classmate must be logged out and logged back in (to re-launch Driver Station) which takes too long.

III. Radio communication:
A. The IFI system had RS422 radios that communicated on 900mhz on a DEDICATED CHANNEL!! The new system shares bandwidth between teams, and we have found the 802.11 communication to be unreliable during matches, although that could be related to all of the FMS problems at Kettering this year. In addition, on loss of the radio, you are screwed for the rest of your match. If the IFI system were to hang and loose comm, it would reboot in 5 seconds and be happy again.

IV. Robot Processor:
A. The old IFI processor had a PIC which was powerful enough to do complex things without using much program space at all. The new system has a giant heavy cRio with a 400mhz PowerPC which has far more power than we could ever possibly need, like trying to gently tap something with a sledgehammer. It's far too big for its application.
B. Download times: They suck. It takes far too long to download code.
C. Boot time: Too long. The people who wrote the code that runs on boot up obviously have never been on a pit crew and had to reset a robot in only a few minutes.

V. LabVIEW:
A. Probe things while the code is initializing: This causes a LabVIEW crash which is really annoying. Same with downloading before the already-deployed code initializes, it will lock you out and you will have to reboot the robot, which is also really annoying.
B. Build times: Really annoying again. We have encountered many instances where we cannot fix a small code issue because that would require waiting 10 minutes to boot the robot, build, download, and reboot again.

Anything else?

Alan Anderson

06-03-2010, 00:56

If you'd like to condense your list of complaints by removing duplicates, I'd be happy to comment on them one by one. As it is, I'd end up unduly agreeing or disagreeing with you if I addressed them as they currently stand.

Greg McKaskle

07-03-2010, 09:38

As Alan stated, it is difficult to respond to this post point by point, but I'll try instead to get us to the productive stage of understanding what is going on and coming up with solutions.

But first, while it is natural and healthy to make comparisons to the previous system(s), let's use the correct terms. NI donates a few key elements that make up the control system, but there are many additional suppliers and developers involved. It is called the FRC Control system for a reason. Myself and other NI folks are involved in support, but we don't deserve all the credit, good or bad.

My timings are with the default code, camera to dashboard and vision processing on. Classmate was used as the development laptop.

I.A. 1.5 minutes seems long. While tethered, these are the times I see. At 8 seconds, the cRIO TCP stack is up, ping succeeds, and the yellow LED goes off. At 20 seconds, the DS Connection light goes off, meaning that the communications task is sending back Status packets as a reply to the control packets. At 30 seconds, the Robot Code DS light is good, meaning that the Robot Main is running and communicating with the DS. The odd thing I cannot explain is why the vision to the PC will often take another fifteen seconds or so to go live, but at thirty seconds, the robot can be enabled and I can control a solenoid and other I/O.

I.B. Using the Classmate as the laptop, I get a bit over two minutes to build. I really really wish I could speed this up, and traditional LV builds are really quite fast, but with the introduction of libraries, the build times went way up and haven't improved yet. In my measurements, it took about 2:40 to build with the Robot Main VI window open, but 2:10 to build with it closed. Also note that the robot isn't needed for building. If you have known changes, you can build while the robot powers up, while it is coming back from the field, etc. Once the app is built, my deploy takes eighteen seconds. This means that a boot and deploy should take less than a minute, and a reboot and test of the code should take another minute.

I.C. The DS measures the round trip time to send control packets to the robot and get a status return. The robot typically receives the packet about 2 to 4 ms after it is sent. The "giant" data packet sent to the robot is only about 1KB, mostly padded zeroes, as wifi frames have a minimum size that is larger than the packet size. This is the same technology used for Skype and plenty of other near-real-time uses. I'm not sure what speed the 422 ran over the 900MHz modems, but the N speed is likely 1000 times higher bandwidth. If you have actual measurements of latency, please share.

II.A. What is the process you are following? I'd expect that the DS would be in sleep mode which takes about ten seconds to awaken. Yes, rebooting the computer should be avoided when you can.

II.B. This seems to be the biggest issue that you faced. But for what it is worth, the DS doesn't send data if the FirstTouch board is removed. The WPI robot code that wraps the comms does return back the latest known good values. It would be possible to return timestamps or other status data if this is really necessary. I don't really know of interference between FMS and Cypress recognition. This hasn't been reported before, but it is certainly something that we'll test more thoroughly. The indication of whether the board is plugged in is the LED on the DS I/O button and LEDs on the I/O board itself.

II.C. This feature was requested and implemented in beta to provide additional safety. Since the laptops are self-powered, it helps to prevent a team from being able to drive a robot at the field until the FMS allows it. It does mean Exiting and restarting driver again, but that took 45 seconds in my test?

III. I know that the FMS does quite a bit of monitoring and logging. The 5GHz channel being used for N is not commonly used. As for rebooting during a match and being up again, I'd avoid that unless asked directed to do so by a coach or FTA.

IV.A. This system is indeed far more capable than the previous one and yet, it is one of the smallest and slowest systems NI makes. I'm pretty sure your metaphor could be used for every cell phone, laptop, and electronic device at the event. All of them were once done with less computational power, but does that make them sledgehammers?

IV.B. Since my stopwatch isn't marked with "They suck", what numbers are we talking about?

IV.C. Some people have been on pit crews, some haven't. Can you be more specific about what you'd fix first?

V. If you can describe how to reproduce the probe crash, please post it in the LV section. As for times, I know of some things that annoy me, but not in the 10 minute range. Try to measure your times rather than guess, and exaggerations cause confusion.

Greg McKaskle

Chris Hibner

07-03-2010, 11:13

At this point, most of my complaints would be pretty nit-picky.

Overall I think the system is very good, and has potential to be great. Having the power pc on board is a big step up for FIRST. In the real world of embedded programming you have to worry a bit more about optimizing for a smaller processor, but I think it's good that people who are new to programming don't have to worry about stuff like that anymore.

I also think the LabVIEW option is a big step forward for FIRST. We were able to get an 8th grader to do about 75% of our tele-op software this year. I know that wouldn't have been likely with C++ or Java (let me clarify this - if you have a kid who's been programming on his own for years, that's one thing. But to take a middle-schooler with little to no programming experience and have them learn C++ enough to program the robot during the build season - very unlikely.)

Having said all of that, I think there are some BIG improvements that can be made to the LabVIEW hardware interfaces to make them more user friendly and intuitive. After you understand how the current interface works, it's pretty user friendly - but when I first saw it I found it to be very counter-intuitive. When I get some time I'll write some simple VIs to show an example of how I think would make it highly intuitive for first timers.

Dave Scheck

07-03-2010, 12:18

We use the C++ side of the world and have been happy with download times. We've found that if there is running code on the cRIO that flash downloads take a much longer time to download than when there is no code running. It almost seems like whatever is doing the installing on the cRIO has a lower priority than what is already running. Could this be something that would apply to the LabView side too?

Something that I would like to see improved is safety at the driver station. As it currently is, the driver station remembers the teleop/autonomous state between robot resets and between enable/disable cycles. This scares me tremendously for teams that aren't careful about the way they write autonomous. These apply to runs in the pit and practice field, not necessarily the real field. Here are my concerns (the theme is a robot should not move unless expected to):

1. If a team doesn't protect their autonomous code with a flag to tell if it has already run or not, they can accidentally rerun their autonomous if they go from auto-disable -> auto-enable -> auto-disable -> auto-enable. PROPOSED FIX: Always move the mode to teleop when disabling. Yes, if the team wants to run auto again they'll have to change the mode to autonomous, but I think that's a fair thing to do to protect people from a runaway robot.

2. If a team doesn't protect their autonomous code with some sort of lock in procedure (i.e. a switch that if not set at the OI will cause the robot to not move), they can accidentally run autonomous if they go through this sequence auto-disable -> auto-enable -> auto-disable -> robot reset (say, for a download). After the reset, if they want to run teleop and simply click enable, they will actually be in auto-enable since the mode hasn't changed. PROPOSED FIX: The driver station already automatically changes from enabled to disabled across a reset/comm loss. It should also force the state back to teleop.

3. There is no fast way to disable a robot with the Classmate. In years past, we've always had an external disable switch that we could physically have our hand on when testing in close quarters. Now, we have to move the mouse to the disable button and click. This reaction time could mean the difference between safety and tragedy. We have implemented an external software disable switch that neutralizes all of our outputs. Once that is switched, we can worry about moving the mouse to the disabled button. Yeah, I know there's the E-stop button, but I think it's ridiculous that when pressed, the robot needs to be completely rebooted to regain operation. PROPOSED FIX: Change the way the E-stop works. Rather than completely disable the robot, allow the user to re-enable it at the driver station (a simple disable-estop button would work). This button could be made available only when not connected to the FMS to prevent people from messing with it on the field.

There are far too many teams that veteran teams will be helping in the pits that won't have any of the three protections above. For our safety and theirs, please do something about this.

All three of these issues seem like they could be easily handled (at least on the surface) in a much welcomed DS update.

heydowns

07-03-2010, 12:20

II.C. This feature was requested and implemented in beta to provide additional safety. Since the laptops are self-powered, it helps to prevent a team from being able to drive a robot at the field until the FMS allows it. It does mean Exiting and restarting driver again, but that took 45 seconds in my test?

A bit of an aside here, since I don't have comments on the majority of the thread topic...

During our first competition this past weekend, we found this particular feature to be quite intrusive, especially during elimination matches where you can have as little as 6 minutes to remove your robot from the field, reset it to a starting condition, and get it back out there and re-connected to the field.

In our experience 45 seconds probably describes a mean, but we've seen upwards of 2 minutes; the DS software has to exit, Windows logs out the Driver user, user logs back in as Driver, then the DS software has to load again, and finally Cypress recognition and USB enumeration are complete.

Perhaps in the future, field management architecture could be revised to place the burden of safety and field control actually on FMS, rather than relying on the DS.

In the short term, seems like having a way to restart the DS without logging out the Windows account and logging it back in again might help shorten the cycle time somewhat.
Or... perhaps having a way to clear FMS locked when Ethernet physical connection (not robot comms, physical connection) is lost for more than a few seconds?

Chris Hibner

07-03-2010, 12:59

By the way, a team (thanks 397) showed me this weekend that you can enable your robot on a tether even during FMS lockout by using the F1 key. You can then disable with the spacebar. I haven't tried it yet with our robot, but I saw them enable their robot with the big FMS Lockout on the screen. I asked how they did it, and they told me about the F1/spacebar thing.

And Dave: don't you guys have the STOP button? Also, the spacebar works as a fast disable, but you have to train yourself to do it so you don't forget when the important time comes.

heydowns

07-03-2010, 13:14

By the way, a team (thanks 397) showed me this weekend that you can enable your robot on a tether even during FMS lockout by using the F1 key. You can then disable with the spacebar. I haven't tried it yet with our robot, but I saw them enable their robot with the big FMS Lockout on the screen. I asked how they did it, and they told me about the F1/spacebar thing.

That's fantastic - never thought to try the shortcuts there! Thanks!

Mike Copioli

07-03-2010, 14:23

My primary concern with the current system is the long reboot time for the DS.

On a couple of occasions a reboot of the DS was necessary during our build. At FLR not being able to reboot or power cycle the DS in a reasonable time caused us to sit dead in the water during Teleop for two practice matches. Auton executed but during teleop the robot did not respond to either gamepad. At the time we assumed the issue was either FMS or a problem with our robot code or possibly CAN. The symptoms were as follows:

- Auton executed as expected (the robot moved and obtained a ball)
- The FMS performed its transition to teleop. The robot did not respond to input from either Gamepad during the entire teleop period but all non-gamepad dependent systems performed (compressor tasks).
- Both Gamepads showed up in the dashboard view.
- All Jaguar LEDS were solid orange during the teleop period.
- The 2CAN status LED indicated the Plugin has loaded properly and that Comm with the crio has been established.
- The robot was rebooted between practice matches but the DS was not. The matches were back to back.

After all practice matches were finished, the FTA (Liz) allowed us and several other teams that were having different issues to connect to the FMS and pinpoint the issue. We regressed our robot code three versions from what was on the robot during the failed practice matches. We performed four tests with the last test using the code that was loaded during the failed matches. All four tests produced the same results.

-The Robot moved in auton and in teleop without issue.

This result led me to conclude that the problem was most likely an issue with USB. We were not able to reproduce this issue after several attempts. The next day (Fri) while on the practice field, the symptoms appeared again. Good com, Good CAN, solid orange LED on the Jaguars, both gamepads showed in the dashboard view and auton executed as expected. We rebooted the DS, re-enabled, and both auton and teleop executed as expected. The DS was not hibernated between matches I know this because we disabled hibernation after experiencing Ethernet issues when waking. What we did as a workaround was as follows.

Before each match the robot was tethered and a full system check was performed as part of our normal pre-match check list. Instead of powering down the DS, we kept it in the EXACT state as when the system check was performed.

I do not know if anyone else has experienced this EXACT issue if they have I would like to know. It would be more suitable for FIRST if when this happens during a match a timely power cycle or re-enumeration of the USB ports was possible. This is one of the many issues that presents when using a DS that requires a bulky OS to operate. The current DS provides some nice features such as camera feedback and a customizable dashboard. I would trade all of these features for robustness and expeditious and deterministic behavior. Camera feedback is nice if you are driving your robot through a building from a remote location or trying to defuse a bomb but I feel it has little value in a FIRST competition. As I tell my drivers, if you are looking at the dashboard during a match you are looking in the wrong place.

Greg McKaskle

07-03-2010, 14:39

Something that I would like to see improved is safety at the driver station...

There are many cases where the robot auto-disables, but avoiding a double auto never came up during the beta. If you feel strongly, please post or discuss directly with FIRST. Clearly safety is a priority, and this is their call. BTW, I don't get the distinction between suggestion one and two.

As mentioned by another response, the spacebar acts as the Disable button. It is polled like a joystick button and should work no matter the key focus. When attached to a field, field controls take precedence, and F1 serves as a way to redo joystick enumeration and is the only key shortcut. As for the Stop button, it initially acted as a disable, but again, during beta requests were made to simulate what would happen on the field.

Similarly, requesting the FMS Lock have another way out is something that should be made on the FIRST forums. With encrypted radios, I don't really see a way to wirelessly connect to the robot anyway, but again, it is their call. I think I see the issue with the code that can allow for unlocking. I'll notify FIRST and see what they want to do about it.

Greg McKaskle

Greg McKaskle

07-03-2010, 14:49

My primary concern with the current system is the long reboot time for the DS...

When this happens, do the joystick LEDs and list respond to the gamepads? Pressing any button on the gamepad should turn the LEDs blue.

As mentioned, F1 was added in an update to reenumerate the joysticks for teams that find they need to swap or plug in a joystick after the auto period. That will close all handles to joysticks, redo the enumeration, and reopen according to the list. Even if this fixes the issue, if at all possible, please help document how it gets in that state.

If anyone else finds a way to provoke joystick loss, please post steps. Everyone wants a robust DS.

Greg McKaskle

Mike Copioli

07-03-2010, 16:31

When this happens, do the joystick LEDs and list respond to the gamepads? Pressing any button on the gamepad should turn the LEDs blue.

I did not notice the LEDs. I will be sure to check if this happens again.

As mentioned, F1 was added in an update to reenumerate the joysticks for teams that find they need to swap or plug in a joystick after the auto period. That will close all handles to joysticks, redo the enumeration, and reopen according to the list. Even if this fixes the issue, if at all possible, please help document how it gets in that state.

I was not aware that F1 also re-enumerated USB. Good to know. As I am sure you know reproducing this kind of issue deterministically will be very difficult.

. Everyone wants a robust DS.

I do not know if this will be possible using a Windows based solutions. Windows is not intended for target specific or mission critical tasks such as a FIRST competition. Know matter how much you slim it down it will still take an excessive amount of time to reboot. At events, teams are given a very limited amount of time to troubleshoot between matches. Every second of every minute counts when you are only provided three minutes between semifinals and finals. Add the reboot time of the DS (up to 2 minutes) with the time the crio takes to boot (40 plus seconds) with the time needed to deploy code and you do not have enough time to make last minute changes to auton or other robot functions that may be critical to an effective strategy. The part that was really frustrating was that fact that no one from NI was at the FLR to provide support, this was something that you did not have to worry about with IFI.

Dave Scheck

07-03-2010, 16:43

And Dave: don't you guys have the STOP button?Sure we do, and it is a perfectly valid way to stop the robot. My problem with it is if you press the STOP button you have to power cycle the robot to get it running again. If the STOP button mimicked the old disable switch I'd be perfectly happy.
Also, the spacebar works as a fast disable, but you have to train yourself to do it so you don't forget when the important time comes.I was unaware of that. That's good to know.

BTW, I don't get the distinction between suggestion one and two.Suggestion 1 was coming from a use case where I run autonomous, then disable, then reenable without changing to teleop. Our team protects against running accidental autonomous, but there are a lot of teams that don't.

Suggestion 2 was coming from the initialization standpoint. If the last operation from the DS was to run autonomous, it is very easy to forget to change the state back to teleop. My point here was to default to teleop-disabled on startup (or initial link connection)

We have taken preventative measures on our end to try to make our robot as safe as possible with the way things currently work. I'm more worried about the teams that don't. It's far too easy to enable without looking at the run mode.

Dave Flowerday

07-03-2010, 16:56

Suggestion 2 was coming from the initialization standpoint. If the last operation from the DS was to run autonomous, it is very easy to forget to change the state back to teleop. My point here was to default to teleop-disabled on startup (or initial link connection)
You could always use my Virtual DS for testing... it's done that from the start. :rolleyes:

Mr. Lim

07-03-2010, 16:58

(Details of USB input device problem)

Mike,

1310 experienced a similar USB input device problem twice during the build season. We were never able to deterministically repeat it, however we saw all the same symptoms.

We use one Logitech Dual Action Gamepad and one Logitech Attack 3 Joystick (w/ modified internals) to control our robot.

On both occurrences of the problem, the devices appeared in the list of USB input devices in the DS. I do not recall if pressing buttons on them caused the LEDs to turn blue. However we printf'ed the getRawAxes on both joysticks and received nothing.

Simple replugging of the joysticks did not solve the problem.

Ultimately, it required us to quit the DS app, unplug the entire supplied USB hub, plug in one of our controllers directly to the Classmate, verify the OS re-install the device driver, unplug the device, restart the DS app, plug the hub back in, and reorder our input devices - the enumeration doesn't always put them back in the order that we need them in.

Right now, I'm a little weary of the Targus USB hub that was supplied. I'd like to know if you're using it as well? Beyond that, I'm a bit stumped as to the cause of the problem.

Mike Copioli

07-03-2010, 18:14

The first time the event occured we were using the hub. The second time the devices were plugged directly into the classmate.

Chris Hibner

07-03-2010, 21:34

Sure we do, and it is a perfectly valid way to stop the robot. My problem with it is if you press the STOP button you have to power cycle the robot to get it running again. If the STOP button mimicked the old disable switch I'd be perfectly happy.

You know me - I'm just giving you a hard time, Dave.

As far as we're concerned, our autonomous is set up so that it can only run once no matter what. Once it starts, you're forced to cycle power to get it to go again. So for us, the STOP button is just as good as the disable since we're going to have to reboot to do another test anyway. I guess if we test using LabVIEW's play button, we can just push the LabVIEW stop and then play again to clear the memory and that is much quicker than a full reboot, but with deployed code it doesn't matter how we end auton.

apalrd

09-03-2010, 09:58

@Greg:

1A: 30s is death in the pits. The IFI system could be up and running in under a second on tether, 5s on radios. 30s or more is waaaay too much time.

1B: My download times appear much greater than you stated, although I could attribute that to the 5 people standing behind me saying "WHEN CAN WE TEST THE ROBOT!!!! GET IT DOWNLOADED NOWWWWW!!!!! GET IT THERED AND MOVE THE ARM NOWWWWW!!!!" while waiting for my code to download.

1C:The IFI system sent packets every ~26 ms. The RS422 900mhz radios sent exactly the data that was needed in one fairly small packet to and from the robot. Data to the robot was 26 bytes in one packet, data back was 2 26 byte packets (most of the data back was exclusively used by the Dashboard). I do not know the latency on it, but it wasn't much. As for the new radios, we had noticeable control lag when using the camera on the Dashboard, but only on the field. This lag was aprox. 3 seconds. I can't say for sure what the cause is, but I can say it appeared when we added the camera and was fixed after we removed the camera, leading me to believe it was the camera.

IIA: We have to reboot Driverstation after every match to get to loose the FMS lock (another annoying feature) but sometimes we do reboot it. If it goes on to the field shut down, it will not find the Cypress board and we are dead.

IIB: This is why we are looking for a solution to the Cypress board, and this almost cost us 2 matches.

IIC: We come off the field with back-to-back matches and have to change all four bumpers, tether the robot to move the arm, change the battery, and get it ready to play again in 5 minutes. 45 seconds is DEATH.

III: Our radio had a loose terminal during one match (duck tape fixed this) and after loosing comm once it had no possibility of ever gaining it again during that match. We tied that match after playing 18 seconds.

IV: The cRio has a 400mhz processor, and while it has aprox. 30% free space (shown during Deploy) it is still not using nearly the power it has unless doing vision processing. We had a crab-drive last year using this system. We needed to do some trig calculations. Even with the IFI processor back when we built a crab drive in 2005, we had plenty of power to do trig calculations with lookup tables and it worked great. That was an 8-bit PIC with far less power or code space then the cRio, but it performed the same task just as well.

IVB and IVC: See IIC

V. If I place a probe on a VI before the "Robot Code" light on the Classmate comes up, LV will crash on my PC. Guaranteed. Always.

@Dave: I would like to see it reset to Teleop after comm or code loss, but not after disable. I often set controls on my VI's for commonly changed parameters (e.g. distance or kick force) and change them while the code is running. I often run autonomous for many times in a row when debugging it, without ever changing to Teleop or downloading new code.
Spacebar disables the robot without e-stopping it. The e-stop button killing code annoys me too, but I have actually used it before (the Toyota bug) so I wouldn't get rid of it.

On the Stop button acting as an E-Stop: Sometimes if the robot spontaneously acts wierd and needs to be stopped, it is nice to be able to probe everything to see what it is trying to do and find your problem. We have an arm on our robot with 800:1 off a CIM, and it has enough torque to cause damage to many parts of our robot. If it starts destroying something, we want to be able to probe it to see where our problem is. We had this exact issue, and found that the analog module on the cRio actually ejected from the cRio.

@Greg again: f1 re-enumerates JS's? Does it find the Cypress board again?

@Everyone: Did anyone notice how nobody EVER debugged comm or firmware things EVER with the IFI system, and how it just always worked? I do not see the cRio or the entire control system as designed to handle the rigorous environment that is FIRST, nor see it as better than the IFI system. Everything, except vision, that we are doing now could be done just as well on the IFI processor. Vision could be done too, using the CMUcam. Now, we are doing vision and other complex processing using many times more code space on many times more expensive equipment, yet achieve the same result, sometimes worse. I am not against using the cRio as a robot controller, most of the cRio based things except for download and bootup times are quite nice to work with. The whole system is not designed to work together; it is a bunch of off-the-shelf stuff that people are trying to make work together. None of it is designed for competition robotics, and can't handle the demands we place on it like the dedicated robot control system provided by IFI could. This all leads to another problem: Since the control system is not made by any one company, it is hard to lay blame for failure on any company.

Mark McLeod

09-03-2010, 12:45

@Everyone: Did anyone notice how nobody EVER debugged comm or firmware things EVER with the IFI system, and how it just always worked?
I never noticed that, seeing as I was involved in three or four IFI emergency beta firmware debugging sessions in the middle of the season...

So soon we forget how many IFI Master Code upgrades we performed over the years. I remember version 15d...
Selective memory is a killer.

No system is perfect. What you should look for is corporate willingness to recognize and react quickly to solving issues when they do arise.

Rich Kressly

09-03-2010, 13:05

You could always use my Virtual DS for testing... it's done that from the start. :rolleyes:

Dave, this is Dave. Thought you two should meet...

(sorry, I just HAD to)

Mike Soukup

09-03-2010, 14:11

As far as we're concerned, our autonomous is set up so that it can only run once no matter what. Once it starts, you're forced to cycle power to get it to go again. So for us, the STOP button is just as good as the disable since we're going to have to reboot to do another test anyway. I guess if we test using LabVIEW's play button, we can just push the LabVIEW stop and then play again to clear the memory and that is much quicker than a full reboot, but with deployed code it doesn't matter how we end auton.
You better make sure that your drive team completely understands this limitation. At an event I was watching this weekend, the robots were enabled for autonomous, the foghorn sounded .5 second later, then the robots were re-enabled .5 second later. It was probably a mistake by the field operator who fat-fingered the stop button. My first reaction was "didn't that just mess up everyone's autonomous? What if they can only run autonomous once?" If you have the limitation that Chris mentioned, you'd better be sure your drive team raises a stink if that ever happens in your match.

Greg, I'll be happy if someone fixes the bug I reported a month ago (http://forums.usfirst.org/showthread.php?t=14505). Can anyone else confirm that this is a problem with their setup?

Bharat Nain

09-03-2010, 14:38

We've already posted our complaints. At this point, I only wish the system would reboot significantly faster. It takes us an average of 35-45 seconds to reboot and that is extremely costly time when we have to perform a repair by calling a timeout during eliminations.

Chris Hibner

09-03-2010, 15:11

You better make sure that your drive team completely understands this limitation. At an event I was watching this weekend, the robots were enabled for autonomous, the foghorn sounded .5 second later, then the robots were re-enabled .5 second later. It was probably a mistake by the field operator who fat-fingered the stop button. My first reaction was "didn't that just mess up everyone's autonomous? What if they can only run autonomous once?" If you have the limitation that Chris mentioned, you'd better be sure your drive team raises a stink if that ever happens in your match.

Greg, I'll be happy if someone fixes the bug I reported a month ago (http://forums.usfirst.org/showthread.php?t=14505). Can anyone else confirm that this is a problem with their setup?

We've been doing it that way since 2003, so Ken is well aware. We decided a long time ago to keep it that way to avoid the safety issues Dave brought up (and most years we allow autonomous to run into teleop depending on the game so our auton scripts can often last longer than 15 seconds). They know that if a match is foghorned, they need to cycle the power to the robot. The FTAs complain a little, but they should expect that some robot reset is going to be necessary if they blow a match dead.

We have the same issue on our gamepad, but I really don't want the bug fixed at this stage because I don't want to re-write things now that we have our software working with the current limitation.

apalrd

09-03-2010, 16:38

I never noticed that, seeing as I was involved in three or four IFI emergency beta firmware debugging sessions in the middle of the season...

So soon we forget how many IFI Master Code upgrades we performed over the years. I remember version 15d...
Selective memory is a killer.

No system is perfect. What you should look for is corporate willingness to recognize and react quickly to solving issues when they do arise.

That's not what I was talking about. I am not referring to the need to update firmware, but bugs in the firmware. I have no problem imaging the controller.

Let's look at some fictional robots:

Imagine you have a robot that has a driver joystick and an operator box, using the IFI control system. Suddenly, you plug it in to the Competition controller and BAM - ports 3 and 4 don't work, but ports 1 and 2 do? OH - you just have to power cycle it a few times the right way and all is well. That's what it's like with the Cypress board, except it takes a long time to power cycle since it runs Windows.

Now imagine you have another robot with an IFI controller, and it controls an apparatus that has a potentiometer for position sensing and a motor to move it. Now you drive over a bump, and suddenly all of your analog ports fall off. Now the robot is trying to kill itself because all of its sensors are gone. I have seen analog modules fall out of the cRio while going over a bump, even though they were completely inserted and latched into the cRio.

And lets compare two robots. Both have a slightly loose connection to their radio. This is a likely scenario with the 2009 Gaming adapters, since the power connection does not lock, but unlikely with the IFI system since all of the connections use DB9 connectors which have locking screws. One has the NI control system, one has the IFI control system. Both robots drive over a bump. Both loose connection. The robot with the IFI system will disable itself because it has lost comm, and try to communicate as soon as the radio is functional again. Within 5s of radio power-up, it is up and running again. However, the other robot will also disable itself, but since it dropped too many data packets while connected to the FMS, it will sit dead until somebody reboots it. Sucks for that robot.

And another comparison of robots. This time, both robots encountered an error and need to be rebooted. It doesn't matter what for, they just do. 5s after being booted, the IFI processor will find its OI if the OI is available and have finished initiating radio comm. As Greg measured, it takes the cRio ~30 seconds to boot. The IFI robot has already finished Autonomous mode and scored a few balls since the other robot is too busy booting up to play defense.

And then, the two robots have back-to-back matches. They must tether, boot, move an apparatus to be within the starting box, and change the battery in 5 minutes. The physical process of tethering takes aprox. 30 seconds each, plus boot up times, around 30s to move the apparatus, and a minute to change the battery. The NI-based robot also has to restart Driver Station since it is FMS locked, which takes around 1 min 15 secs. The IFI robot can be ready in 2 min 5 seconds, while it would take the other robot take 4 min 45 secs to do all of this.

Now look at what these two robots can do. Both have swerve drive, with feedback on the front and rear wheels independently. They react almost identically to driver input. There is a solid-color vision target in the game. The IFI processor uses its CMUcam Co-Processor to find it and drive to it. This causes almost no lag on its part. The cRio connects via Ethernet to its camera, downloads a new image, processes it, and drives to the target, while causing the swerve drive controls to freak out from the non-constant loop time (even though vision is running in a separate virtual thread). I have seen vision processing mess up PID calculations. That's exactly why we didn't use the camera in 2009 with our swerve drive.

And finally we will see what happens when the programmer must change a small variable in Autonomous mode. Since the IFI processor is programmed in MPLAB C, the programmer would have to change a variable or function call in the code. They would then have to compile the code, a process that takes around 10 seconds. They would have to plug a serial cable into the robot controller, and download the code. This download process takes around 15 seconds. They would then unplug their serial cable and press the RESET button. In 35 seconds, not including the time it takes to connect the serial cable, they are ready to go. With our other robot that has a cRio, the programmer makes the same function call change. He then builds the code, which using Greg's benchmarking would take around 2 minutes. He then must tether to the robot, which is not included in the time. He then must wait for the robot to boot up, a process which Greg claims takes 30 seconds. Then, he must Deploy his code. I have never ever seen my code download in 18 seconds, so here is my calculation for download time: aprox. 30 seconds to load code + 15 seconds to wait for cRio + 10 seconds to do something unknown before asking me to reboot. Total time: 55 seconds. After another reboot, that brings the grand total minor code change time for this robot with a cRio to 3 minutes 55 seconds. Unacceptable. That's 3 minutes 20 seconds more than the IFI processor, or a bit more than the time it takes to run one match, or a 671% increase.

Radical Pi

09-03-2010, 19:22

On the topic of code download and build times, I have seen nothing like that on C++. It probably is beacuse most of the large code is in a precompiled library instead of building every single time you build. I'm don't know much about what can or can't be done with LabVIEW, but if a cache of the compiled code could be kept of the NI libraries, it would greatly speed up builds (From my experiences with the LEGO NXT, it also seems to be in the nature of LabVIEW and variants that programs are unnecisarrily large compared to C equivalents)

@Greg, you said earlier you saw an unexplained delay in camera images showing up. That delay appears to be from the camera taking much longer to boot than the cRIO code, delaying images. This is actually an important time delay for our bot, because in the time between cRIO boot and camera, the console is flooded with errors after camera init is called since it can't connect to the camera, rendering CAN code inoperable due to timing delays messing up the system, as well as a noticable lag in the controls of the robot (probably due to the increased UDP traffic through netconsole)

Kevin Sevcik

09-03-2010, 23:28

Ditto the comment on selective memory. I vividly remember the "7.2V" bug biting us at Great Lakes. And the horrible growing pains switching from the even older STAMP controller to the newer PIC controller. And so on and so forth. Every new control system has growing pains and atleast some people pines for the Good Ol' Days. So for a change in this thread, I'd like to point out (some) of the (numerous) positive aspects of the new FRC Control System:

That you DO have the option of probing things live in Labview/Windriver for debugging. Slight bugginess in a feature not even available on IFI? Score one for the new system. I can tell you I won't miss having to litter my code with printfs.
Real hardware encoder inputs. No more worrying about high resolution, high speed encoders jamming your processor with interrupts. Plus you can do more than 4 now.
Real hardware sampled analog inputs. No more worrying about your interrupt routine to sample and average your analog inputs. As a bonus you can sample them at ludicrous speeds as well.
Expanded speed controller abilities through the Jaguars + CAN bus. Now we have speed controllers that at linear and have smarts approaching the entire IFI controller. You don't even have to spend processor time on PID loops.
Expanded peripheral counts. No, I haven't needed 20+ speed controllers and servos yet, but isn't it nice that it's available?
For that matter, no more digging around in PIC datasheets to figure out just how you're supposed to use peripheral XYZ.
Pre-made extremely useful dashboard with highly useful error messages and robot feedback. No more deciding which user byte is the most useful to you at any particular time. No more punching the user button to find out what your battery voltage is.
Actual video on said dashboard. No more swapping camera cables back and forth trying to figure out what on earth that silly CMUCam is fixated on this time.
Increased ease of programming. Knock it all you want, I can have a newbie team competently programming in Labview while you're still teaching your team the intricacies of C syntax. "You mean = and == are different?" vs. "You mean I just wire these little pictures together?"
No more 8/16 bit integer math! Doubles for everyone! Say goodbye to all those pesky under/overflow bugs. Aren't you happy you no longer have to worry about the order of your multiplication and division and the loss of precision you're incurring? If I never see another typecast in our robot code, it'll be too soon.
Download-less software testing and debugging (in Labview anyways). Wanna test that fancy new autonomous mode? Just hit the run button. Made your robot explode? No worries, just reboot and everything's back exactly as it was.
No more backup batteries to worry about. Yes, this is a function of the new PD board, not the cRIO, but you're lumping it all together anyways. Also, new highly compact PD board with WAGO connectors. Take that giant copper maxi breaker blocks.
Decentralized wiring. Isn't it nice not having to cram all of your wiring into the border around a 4x5 box? That also needs to be highly visible and accessible? Plus, swapping a cRIO out of a bot is loads simpler than swapping an IFI controller out.
Development software that's actually licensed to be installed as many computers as you'd care to install it on. Yes, I know you didn't care that MPLAB was a single seat license. But now you can continue not caring about licenses and still be perfectly legal.
The driver station doubles as a development platform. I'm sure you have to dig you way out of a pile of laptops if you open the wrong door, but I've run into many rookie teams that couldn't do a thing with their robot simply because they had no computer. Now one comes in the kit, ready to go for programming.
Finally, no more analog joysticks and the hacks accompanying them! The fact that I no longer have to hoard '04 and earlier flightsticks? Excellent. The fact that I can now run down to the local CompU Buy Radio City and grab a joystick/steering wheel/game pad off the shelf and have it just work? Awesome. The fact that I never have to trim another joystick? Priceless.

So, I think there's a rather lot of positives to the new control system that help to balance some of the annoyances. Frankly, my largest peeve with the new system is the sheer cost and the fact we don't get a new one on an annual basis.

I find it difficult to really get annoyed with "lengthy" compile/download times. As long as they are under 2-3 minutes, I could really care less. If you're downloading code so often that 2 minutes vs. 30 seconds KILLS you, then you're doing something wrong. And you should probably just be hitting the run button to avoid all that pesky downloading and rebooting, as your code clearly isn't in a finished state.

Similarly the back-to-back matches. There's never a scenario where this is a problem. You always have more than 5 minutes between matches, even in the finals. If you can't organize yourself enough to reset your robot in the allotted time, you have other issues.

Also, to your autonomous example. If you're still in development on your autonomous mode, you're not deploying code, you're running it in RAM. Which is a lot quicker. And if you're really clever, then the important variable in your autonomous code aren't constants. They're linked to front panel controls that you can change at whim without changing the program a bit. Time to change a front panel variable? About 5 seconds, plus restarting your DS in auton. Say about 5 times faster than your IFI controller.

Point being, the new control system gives you loads and loads of new tools and features. More than I can think of off the top of my head. You really have just two options:
1. Learn. Figure out what new features you can use to make your coding easier. Embrace the new system and delve into it as deeply as we all delved into that PIC processor. I'm not sure what the cRIO equivalent of the Counter0 control registers is, but there's gotta be something as esoteric and useful. Go explore and find it.
2. Stagnate. Continue to pine for the Good Ol' Days and resist working with the new system. Grumble about how much better things used to be. Don't bother with all those new-fangled debugging techniques. Declare how the youth of today don't know how easy they have it. Back in the day, they only had integers for math. Five typecasts and three overflow checks from the joystick to the victor it was, if you were lucky.

Oh. And shake your fist too. That always shows 'em.

Greg McKaskle

10-03-2010, 00:08

Greg, I'll be happy if someone fixes the bug I reported a month ago. Can anyone else confirm that this is a problem with their setup

We have the same issue on our gamepad, but I really don't want the bug fixed at this stage because I don't want to re-write things now that we have our software working with the current limitation.

Everyone has that issue with the gamepad, and yes it is a bug, but one that wasn't found or reported until well into the season. The decision was made to leave it alone. It is a bug with compatibility with the previous driver station, but not one worth the churn of a mid-season change in functionality.

At the beginning of the season, we were hoping to move to a new joystick API, one which would support higher resolution, more axes, more buttons, and more traditional POV controls. The bug was introduced trying to map the MS HID data to reproduce the behavior of the 09 DS. The amount of mapping required go from named axes to a limited number of numbered axes was a royal pain. Anyway, the '11 API can certainly fix the bug or replace it completely.

As for the state to transition to after Auto, the code can easily be modified to do this, but it would be up to FIRST to decide what to adopt, or to decide if this is a team-settable option.

Greg McKaskle

apalrd

10-03-2010, 00:20

@Kevin:
I can see your advantages to the new control system over the IFI processor, but I have several more comments:

1. I use probes all the time. I often crash Labview trying to open probes right after downloading, which is really annoying.
2. The IFI processor had 6 hardware interrupts, enough for 2 1x and 4 2x encoders or less with different decoding. The new processor has 8 hardware counters, but 4x decoding takes two counters. Not a big improvement, but no more ISRs
3. Sampling a gyro from Process_Data_From_Master_uP seems to have enough resolution and speed for everything to work fine.
4. I agree on the Jaguars, although the jaguars are actually more powerful than the entire IFI processor. However, the "safety" features included (shutdowns and lockouts on small errors) sent us to Victors this year.
5. I like the more IO but not the way the IO is handled - especially how the whole system has max. 7 power feeds to a whole bunch of separate boards on a whole bunch of separate modules on a cRio chassis. It could be a little more integrated.
6. I like the Dashboard, but the camera comm seems to cause problems with robot comm so I don't like camera comm.
7. I have several uint8 and uint16's in our robot, and typecasts associated with them. Most of them are involved in kicker set distance, as it prefers doing math in feet.
8. I often do run stuff by pressing the play button. And it takes me around 20 seconds to download over wireless, but using controls saves a bit of time.
9. It looks to me like the new PD board is quite a bit larger than the single MAXI block and ato6 panel on BuzzXIII, and a MAXI panel and ATO panels fit much more nicely into most of our framing. I also hate WAGO connectors because I am always loosing the WAGO tools and every little screwdriver we have, and the ability to get enough space on the side of the PD board to fit the WAGO tool in is often not enough. It does weigh less, which is nice.
10.Since most of our wiring is crammed around the Digital Sidecar, it ends up having twice as much IO in half the space (although it is burried deep in the belly of the robot, not visible remotely)
11. The LabVIEW splash screen dosen't even fit on the Classmate. I tried developing on it. The keyboard is just too small. I like the attempt of FIRST, but not the laptop.
12. Although I do not miss the analog joysticks, I do miss the raw analog IO. The new Cypress board is probably the single thing I hate most about the new system, since it A. runs on 3.3v, and especially B. the Driver Station has many issues finding it when combined with FMS.

As for advantages, the big ones I have that change the most, not including language.
1. Realtime trig without lookup tables
2. Floating-point math
3. Virtual multithreading
4. That's about it.

As for my autonomous example, I have found myself in many situations at competitions where there is a minor bug in autonomous, such as going a little bit too far or kicking a little too high, which is caused by a change in the reaction of something such as the kicker as it is worn in, and I must do a full deploy since it will be used on the field.

@everyone:
Many people probably see this thread as a request for the IFI processor back. That is not my intentions. I am saying that, while the cRio and associated control system has many bugs that the IFI system dosen't, neither one is perfect. It would be nice if only we could have a control system that was as fast, small, light, and reliable as the IFI system but more powerful. We don't really need the power of the 400mhz PowerPC, but having floating point math and LabVIEW really helps alot. If having a large several-pound cRio with many times the necessary power is the solution, then so be it. Just make the rest of the system work as well as it does. Having a dedicated driver station dosen't run Windows, lockout users after every match, and actually finds all of its inputs every time would be a nice start, followed by faster download/bootup times and more reliable radios. And maybe a solution to those ejecting analog modules other than glue.

Joe Ross

10-03-2010, 00:58

@Kevin:
2. The IFI processor had 6 hardware interrupts, enough for 2 1x and 4 2x encoders or less with different decoding. The new processor has 8 hardware counters, but 4x decoding takes two counters. Not a big improvement, but no more ISRs

The FPGA has four 4x encoder modules, and 8 counters. You don't use up 2 counters to run an encoder in 4x.

And maybe a solution to those ejecting analog modules other than glue.

What part ejected, the NI module or the analog breakout? Was the analog breakout screwed in?

Greg McKaskle

10-03-2010, 01:10

My text is marked with ***'s.

1A: 30s is death in the pits. The IFI system could be up and running in under a second on tether, 5s on radios. 30s or more is waaaay too much time.
*** The original post said it was 1.5 min. I measured it at 0.5. I'm sure a 1 second boot would be nice, but it cannot happen. I suspect it could be a few seconds shorter, perhaps as much as ten seconds shorter, but at this point the breakdown in my post was as much info as I have. To modify the image, leave out libraries, or modify the communications task are major undertakings. The other way to look at this is that this affects all teams the same. If the weight limit is changed or the field size changed, it may not be a good compare to prior years, but it is fair because all teams are affected equally. More on this in the micro line below.

1B: My download times appear much greater than you stated, ...
*** If they are different, I'll be happy to help you determine why. My test was with default code and used deploy, not Run As Startup.

1C:The IFI system sent packets every ~26 ms. The RS422 900mhz radios sent exactly the data that was needed ...
*** Personally, I don't think the current protocol sends enough info. The FTC protocol similarly sends about 20 bytes of data, but that has much more to do with transmission speed limits than it being exactly what was needed. I won't claim to be a wifi wizard, but once you start sending framed data, I believe that it is equivalent to send one byte as to send one full frame. I do not think the network speed or the control packet being sent is responsible for the lag you are seeing.

... but only on the field. This lag was aprox. 3 seconds. I can't say for sure what the cause is, but I can say it appeared when we added the camera and was fixed after we removed the camera, leading me to believe it was the camera.
*** I can't debug it with this description. With the right camera settings, I can cause video lag. I can roughly pin this on a memory manager issue, and in another thread I gave compression numbers to avoid the issue. Similarly, we found some DS UI code that could cause lag on the DS consumption of the video. The latest DS update resolved the issue. If you are having joystick lag, it would be good to run some performance tools to get some data on what your app is doing.

IIA: We have to reboot Driverstation after every match to get to loose the FMS lock (another annoying feature)
*** As explained in the other post, FIRST sees the exit/open DS as necessary for safety of field staff and team members on the field. I can't explain the exact benefit, but I think we can all agree that safety trumps convenience. But of course we really want both.

but sometimes we do reboot it. If it goes on to the field shut down, it will not find the Cypress board and we are dead.
*** This is being discussed in another thread, but is not something we've been able to reproduce. If you want to answer questions on the other thread, they may help to pinpoint the issue.

IIC: We come off the field .... 45 seconds is DEATH.
*** Actually, death is what could happen when not enough attention is paid to the safety issues -- death is still probably still a stretch of course. I believe the best approach is to state the case to FIRST. It is their call.

IV: The cRio has a 400mhz processor, ...
*** I would guess the experiences are different in many respects. If/when NASA sends astronauts to the moon again, I highly doubt they'll use 1MHz 2KB computers for the guidance system. Times and capabilities change. The upside includes things like floating point. The downside includes things like boot time.

V. If I place a probe on a VI before the "Robot Code" light on the Classmate comes up, LV will crash on my PC. Guaranteed. Always.
*** Where are you placing the probe, and when are you probing? I often have a numerous probes set before I even boot the cRIO. I suspect that you've found a timing issue with the RT protocol in LV, but it would certainly be helpful to have more information to narrow it down.

@Dave: I would like to see it reset to Teleop after comm or code loss, but not after disable. I often set controls on my VI's for commonly changed parameters (e.g. distance or kick force) and change them while the code is running. I often run autonomous for many times in a row when debugging it, without ever changing to Teleop or downloading new code.
Spacebar disables the robot without e-stopping it. The e-stop button killing code annoys me too, but I have actually used it before (the Toyota bug) so I wouldn't get rid of it.

@Greg again: f1 re-enumerates JS's? Does it find the Cypress board again?
*** The DS doesn't explicitly find the Cypress board in the first place, so no. F1 explicitly closes, enumerates, and reopens joystick HID handles. The DS would do this continuously, but it takes around 100ms to enumerate the USB bus devices. For this reason, it does this once a second when disabled, but only when requested during teleOp.

Greg McKaskle

Greg McKaskle

10-03-2010, 01:24

@Greg, you said earlier you saw an unexplained delay in camera images showing up. That delay appears to be from the camera taking much longer to boot than the cRIO code

It may not have been clear, but those times and that issue were done only by pressing the reset button on the cRIO. The camera was powered the entire time. I have not had time to look at the odd delay in the dashboard vision yet, so still a mystery.

Greg McKaskle

Greg McKaskle

10-03-2010, 02:10

That's not what I was talking about...

Consider that the boot times of the cRIO are very similar to last year. The build times for LV at least are smaller, and often significantly less. A new language was added and a new DS was added. With change comes the potential for unforeseen issues.

At this point it seems like the dropout of USB devices may be related to connecting the devices through the hub in a way that draws more power than USB can supply. I thought the documentation recommended that the Cypress FT board be plugged directly into one USB port and the hub and other devices into the other. Can anyone verify if that is in the documentation? Whatever the cause, this is the top issue being investigated.

There are also reports of the Cypress board not working after a suspend or when booted on the field. This is probably the next highest priority issue. Data on the driver being used as well as the steps to reproduce would be very helpful as I personally have not been able to reproduce this.

If other issues arise, they will be compared and investigated as well. Unless a performance issue has a dumb-simple solution, it really isn't feasible to change it at this point in the season. I'd love to be able to ride in on a white horse and explain how you can now boot in negative 5 seconds, deploy code without having to turn the controller on, etc. That isn't going to happen no matter how many times you repeat your points.

Knowing what you'd like to see improved in the system is very useful. Thanks. It would also be useful to use the correct name of the system -- once again, it isn't the NI control system.

Greg McKaskle

Greg McKaskle

10-03-2010, 08:15

The cRio connects via Ethernet to its camera, downloads a new image, processes it, and drives to the target, while causing the swerve drive controls to freak out from the non-constant loop time (even though vision is running in a separate virtual thread). I have seen vision processing mess up PID calculations. That's exactly why we didn't use the camera in 2009 with our swerve drive.

I hadn't heard many complaints about this. The framework for 2009 was kept simple and not designed specifically for a swerve robot. The "virtual threads" are real vxWorks threads BTW, and by default everything runs at the same priority. My suggestion to try and improve this would be to up the priority on the PID or lower it on the vision. VxWorks and LV RT are certainly used in realtime applications just as demanding as keeping wheels pointed in a particular direction. Also, learning to use the tools to learn where the processing is going, what latencies are being seen, and where memory is going, is IMO a good problem.

The thing about vision is that it produces so much data that it can bring almost any processor to its knees. Some simple things can be done with little code, but people have very high expectation -- "My two year old sees it, why doesn't the computer?"

If you want to post a good description of your lag, perhaps I can help you find the issue and solve it for real rather than talk about fictional ones. PM me if you want to try and do this interactively or want to provide code.

Greg McKaskle

Chris Hibner

10-03-2010, 08:29

As for my autonomous example, I have found myself in many situations at competitions where there is a minor bug in autonomous, such as going a little bit too far or kicking a little too high, which is caused by a change in the reaction of something such as the kicker as it is worn in, and I must do a full deploy since it will be used on the field.

I can understand how this can be frustrating. However, you can solve this issue by taking advantage of one of the other big benefits for the cRIO: a real file system with plenty of file space.

To get around the problem you described, all of our autonomous parameters are stored in files on the cRIO. Upon boot-up the file system checks all of the files and loads the default. The drivers can cycle through all of the files from the driver station while in Disabled mode and the last selected file is what is used for the match. In order to add/change/delete files, we use FileZilla to FTP the files. It takes only a few seconds to do.

I would be more than happy to e-mail you our VIs that do all of this, if you want.

apalrd

10-03-2010, 16:47

At this point it seems like the dropout of USB devices may be related to connecting the devices through the hub in a way that draws more power than USB can supply. I thought the documentation recommended that the Cypress FT board be plugged directly into one USB port and the hub and other devices into the other...

There are also reports of the Cypress board not working after a suspend or when booted on the field. This is probably the next highest priority issue. Data on the driver being used as well as the steps to reproduce would be very helpful as I personally have not been able to reproduce this.

During competition matches, we have two USB devides (Cypress board and Logitech gamepad) connected to the KOP USB hub which recieves power from both USB ports. At home and on the practice field, we also have the Stop button plugged into the hub. We are not using the Cypress board to drive any outputs, so all of its power consumption is from powering the chip. We had no problems at all when at home or on the practice field. We found that if the Classmate is sleeping (but not off) when we bring it to the field, and it has found the Cypress board, and connect the Classmate to the FMS at any time, it is fine. If the Classmate is actually off, and we connect to the FMS before it has finished opening Driverstation, it will not find the Cypress board but it will find our gamepad. Rebooting once without the FMS does not always fix this, but rebooting twice always does. As a side note, if this ever happens to us, even if we re-launch Driverstation without rebooting, we will still not find the Cypress board. Everything points to a dead Cypress board, but rebooting twice will fix the problem. We had many teams come to us at Kettering and ask is if we had a spare Cypress board, since theirs magically died while on the field. As I have said in previous posts, what would you think if you had an IFI control system and some of the ports magically didn't work, you would call IFI support right then and there and tell them their products were not reliable enough for competition use and need more beta testing. The same problem is happening with the Cypress board, and I hope a fix is released soon. I cannot see you not being able to reproduce the problem, many teams at Kettering saw it while on the field, including us twice.

As for the analog module, the NI analog input module actually flew out of the cRio, with the analog breakout board firmly screwed in. The cRio is located deep in our chassis, and we didn't see it until the robot was attempting suicide. We probed all of the sensors with a multimeter, and all of them returned valid voltages aprox. where they should be. Everything pointed to a software problem or a dead analog module, until we saw that the analog module had actually come out of the cRio. We glued in all of our modules, but are hoping a more permanent solution is released. Would it be possible to bolt the modules in? Or make a FIRST analog input module that is powered by the cRio, has the 3-pin headers, and secures in a better way then the sprung clips?

As for timing, I will measure download and bootup times on our practice robot next time I am working with it. I am using "Run As Startup", not "Deploy". Your 30s time sounds about right if I were to click the run button in Robot Main.vi.

And for the lag, we noticed an aprox. 3s delay between when the driver sent a command and when the robot acted on it at Kettering. We had recently put the camera on, so we removed it and the delay went away in our next match. Everything points to not enough bandwidth for the camera.

As for the FMS lock, I am aware that FIRST wants it but please make it easier to reset it. Logging off and on in Windows is not a quick thing, and doing that after every match is really annoying. In past years this has not been an issue, as the OI was rebooted every time it was tethered or plugged into the Competition. Power cycling took a few seconds, a few more with that blue OI from last year, but was done all the time. Now that the classmate is never rebooted, issues that are/have been previously fixed with power cycles are now much more tedious and annoying processes, and often unnecessary. Would it be possible to restart Driverstation without looging off and back on again?

@Chris: I have a labview scripting system I really like, but before writing that I actually considered a Python interpreter to allow me to write the autonomous routines separately. Jim Zondag immediately said it was unnecessary and all development would be done in one language, and the project was canceled. I am still looking to do that for next year, but test it in the off-season.

Schnabel

10-03-2010, 17:18

Things I like about the new control system, from someone who hasn't played with one in over a year and knows very little about the technical side of it.

No backup battery! Remember back in the "good ol days" what would happen if you had a weak main battery and tried to communicate with the robot? Yeah exactly, nothing, then a reboot.

Industry Standard. I think it's cool that we have switched to a device that is being used elsewhere to produce products we use everyday. It's not exactly the same, but in the future we will have to deal with long start up times and many of the problems we deal with here. The difference is that lost time in industry means lost money.

WIFI! Who doesn't love WIFI? You can say that we are trying to run too much through WIFI, but last I checked (Where I live that is) I can run multiple videos through my wireless N router, plus have everyone connect to it to surf, view, and download a ton of content. The bottle neck for us though isn't the WIFI itself, but more the internet bandwith. I never have a problem transfering files over WIFI though, even with 56 computers, iPods, XBOX's, PS3's, ect. connected to it.

USB! When's the last time you saw a joystick connector on a computer?

Room for improvement! Yes the CRIO is way to powerful for us. Will it be in 5 years though? If you can't answer that, think of this, your computer is faster than you need to run nicely today, will it be in 2 years?

Issues!!! Last time I checked, the best way to solve issues was to try to find a fix, not complain about them until someone else tried to find a solution. This is great experience to trying to solve the issues ourselves.

Yes I do feel that the IFI system had it's advantages, but this new system does too, so why don't we just try to adapt and accept that this is the system we will be using now, not the old system.