Comments/Complaints on NI control system

[Greg McKaskle]

@Greg, you said earlier you saw an unexplained delay in camera images showing up. That delay appears to be from the camera taking much longer to boot than the cRIO code

It may not have been clear, but those times and that issue were done only by pressing the reset button on the cRIO. The camera was powered the entire time. I have not had time to look at the odd delay in the dashboard vision yet, so still a mystery.

Greg McKaskle

[Greg McKaskle]

That's not what I was talking about...

Consider that the boot times of the cRIO are very similar to last year. The build times for LV at least are smaller, and often significantly less. A new language was added and a new DS was added. With change comes the potential for unforeseen issues.

At this point it seems like the dropout of USB devices may be related to connecting the devices through the hub in a way that draws more power than USB can supply. I thought the documentation recommended that the Cypress FT board be plugged directly into one USB port and the hub and other devices into the other. Can anyone verify if that is in the documentation? Whatever the cause, this is the top issue being investigated.

There are also reports of the Cypress board not working after a suspend or when booted on the field. This is probably the next highest priority issue. Data on the driver being used as well as the steps to reproduce would be very helpful as I personally have not been able to reproduce this.

If other issues arise, they will be compared and investigated as well. Unless a performance issue has a dumb-simple solution, it really isn't feasible to change it at this point in the season. I'd love to be able to ride in on a white horse and explain how you can now boot in negative 5 seconds, deploy code without having to turn the controller on, etc. That isn't going to happen no matter how many times you repeat your points.

Knowing what you'd like to see improved in the system is very useful. Thanks. It would also be useful to use the correct name of the system -- once again, it isn't the NI control system.

Greg McKaskle

[Greg McKaskle]

The cRio connects via Ethernet to its camera, downloads a new image, processes it, and drives to the target, while causing the swerve drive controls to freak out from the non-constant loop time (even though vision is running in a separate virtual thread). I have seen vision processing mess up PID calculations. That's exactly why we didn't use the camera in 2009 with our swerve drive.

I hadn't heard many complaints about this. The framework for 2009 was kept simple and not designed specifically for a swerve robot. The "virtual threads" are real vxWorks threads BTW, and by default everything runs at the same priority. My suggestion to try and improve this would be to up the priority on the PID or lower it on the vision. VxWorks and LV RT are certainly used in realtime applications just as demanding as keeping wheels pointed in a particular direction. Also, learning to use the tools to learn where the processing is going, what latencies are being seen, and where memory is going, is IMO a good problem.

The thing about vision is that it produces so much data that it can bring almost any processor to its knees. Some simple things can be done with little code, but people have very high expectation -- "My two year old sees it, why doesn't the computer?"

If you want to post a good description of your lag, perhaps I can help you find the issue and solve it for real rather than talk about fictional ones. PM me if you want to try and do this interactively or want to provide code.

Greg McKaskle

[Chris Hibner]

Originally Posted by apalrd

As for my autonomous example, I have found myself in many situations at competitions where there is a minor bug in autonomous, such as going a little bit too far or kicking a little too high, which is caused by a change in the reaction of something such as the kicker as it is worn in, and I must do a full deploy since it will be used on the field.

I can understand how this can be frustrating. However, you can solve this issue by taking advantage of one of the other big benefits for the cRIO: a real file system with plenty of file space.

To get around the problem you described, all of our autonomous parameters are stored in files on the cRIO. Upon boot-up the file system checks all of the files and loads the default. The drivers can cycle through all of the files from the driver station while in Disabled mode and the last selected file is what is used for the match. In order to add/change/delete files, we use FileZilla to FTP the files. It takes only a few seconds to do.

I would be more than happy to e-mail you our VIs that do all of this, if you want.

[apalrd]

Originally Posted by Greg McKaskle

At this point it seems like the dropout of USB devices may be related to connecting the devices through the hub in a way that draws more power than USB can supply. I thought the documentation recommended that the Cypress FT board be plugged directly into one USB port and the hub and other devices into the other...

There are also reports of the Cypress board not working after a suspend or when booted on the field. This is probably the next highest priority issue. Data on the driver being used as well as the steps to reproduce would be very helpful as I personally have not been able to reproduce this.

During competition matches, we have two USB devides (Cypress board and Logitech gamepad) connected to the KOP USB hub which recieves power from both USB ports. At home and on the practice field, we also have the Stop button plugged into the hub. We are not using the Cypress board to drive any outputs, so all of its power consumption is from powering the chip. We had no problems at all when at home or on the practice field. We found that if the Classmate is sleeping (but not off) when we bring it to the field, and it has found the Cypress board, and connect the Classmate to the FMS at any time, it is fine. If the Classmate is actually off, and we connect to the FMS before it has finished opening Driverstation, it will not find the Cypress board but it will find our gamepad. Rebooting once without the FMS does not always fix this, but rebooting twice always does. As a side note, if this ever happens to us, even if we re-launch Driverstation without rebooting, we will still not find the Cypress board. Everything points to a dead Cypress board, but rebooting twice will fix the problem. We had many teams come to us at Kettering and ask is if we had a spare Cypress board, since theirs magically died while on the field. As I have said in previous posts, what would you think if you had an IFI control system and some of the ports magically didn't work, you would call IFI support right then and there and tell them their products were not reliable enough for competition use and need more beta testing. The same problem is happening with the Cypress board, and I hope a fix is released soon. I cannot see you not being able to reproduce the problem, many teams at Kettering saw it while on the field, including us twice.


As for the analog module, the NI analog input module actually flew out of the cRio, with the analog breakout board firmly screwed in. The cRio is located deep in our chassis, and we didn't see it until the robot was attempting suicide. We probed all of the sensors with a multimeter, and all of them returned valid voltages aprox. where they should be. Everything pointed to a software problem or a dead analog module, until we saw that the analog module had actually come out of the cRio. We glued in all of our modules, but are hoping a more permanent solution is released. Would it be possible to bolt the modules in? Or make a FIRST analog input module that is powered by the cRio, has the 3-pin headers, and secures in a better way then the sprung clips?


As for timing, I will measure download and bootup times on our practice robot next time I am working with it. I am using "Run As Startup", not "Deploy". Your 30s time sounds about right if I were to click the run button in Robot Main.vi.


And for the lag, we noticed an aprox. 3s delay between when the driver sent a command and when the robot acted on it at Kettering. We had recently put the camera on, so we removed it and the delay went away in our next match. Everything points to not enough bandwidth for the camera.

As for the FMS lock, I am aware that FIRST wants it but please make it easier to reset it. Logging off and on in Windows is not a quick thing, and doing that after every match is really annoying. In past years this has not been an issue, as the OI was rebooted every time it was tethered or plugged into the Competition. Power cycling took a few seconds, a few more with that blue OI from last year, but was done all the time. Now that the classmate is never rebooted, issues that are/have been previously fixed with power cycles are now much more tedious and annoying processes, and often unnecessary. Would it be possible to restart Driverstation without looging off and back on again?

@Chris: I have a labview scripting system I really like, but before writing that I actually considered a Python interpreter to allow me to write the autonomous routines separately. Jim Zondag immediately said it was unnecessary and all development would be done in one language, and the project was canceled. I am still looking to do that for next year, but test it in the off-season.

[Schnabel]

Things I like about the new control system, from someone who hasn't played with one in over a year and knows very little about the technical side of it.

No backup battery! Remember back in the "good ol days" what would happen if you had a weak main battery and tried to communicate with the robot? Yeah exactly, nothing, then a reboot.

Industry Standard. I think it's cool that we have switched to a device that is being used elsewhere to produce products we use everyday. It's not exactly the same, but in the future we will have to deal with long start up times and many of the problems we deal with here. The difference is that lost time in industry means lost money.

WIFI! Who doesn't love WIFI? You can say that we are trying to run too much through WIFI, but last I checked (Where I live that is) I can run multiple videos through my wireless N router, plus have everyone connect to it to surf, view, and download a ton of content. The bottle neck for us though isn't the WIFI itself, but more the internet bandwith. I never have a problem transfering files over WIFI though, even with 56 computers, iPods, XBOX's, PS3's, ect. connected to it.

USB! When's the last time you saw a joystick connector on a computer?

Room for improvement! Yes the CRIO is way to powerful for us. Will it be in 5 years though? If you can't answer that, think of this, your computer is faster than you need to run nicely today, will it be in 2 years?

Issues!!! Last time I checked, the best way to solve issues was to try to find a fix, not complain about them until someone else tried to find a solution. This is great experience to trying to solve the issues ourselves.

Yes I do feel that the IFI system had it's advantages, but this new system does too, so why don't we just try to adapt and accept that this is the system we will be using now, not the old system.

[apalrd]

Originally Posted by Schnabel

Issues!!! Last time I checked, the best way to solve issues was to try to find a fix, not complain about them until someone else tried to find a solution. This is great experience to trying to solve the issues ourselves.

As most of my problems are on the Classmate end, I have exactly no control over it. It has bugs, but I have no source code to modify and fix it. The fixes I have take too long to implement (e.g. rebooting twice when loosing Cypress IO), or are way more difficult than they should be (e.g. tearing apart a USB joystick that dosen't have an auto-centering axis and soldering all of my IO to that) As for the NI controller, other than the long times I like it very much. Most of the problems with it (ejecting modules) can be fixed with duck tape and such, and I really like the ability to probe things while they are running. The cRio is not the majority of the problem, it is the rest of the system implemented to make it work. Many of my complaints are on the Classmate end.

@Greg: That thing I mentioned about probes, it isn't a problem to have a probe open while the code loads, but it is a problem to place a probe while waiting for code. It seems to be a bug in LabVIEW RT, not the FRC specific software.