Bizarre cRIO Server Errors (FDIO)

[Joe Ross]

Could you be running into this problem? http://www.chiefdelphi.com/forums/sh...d.php?t=126102

[bvisness]

Originally Posted by Joe Ross

Could you be running into this problem? http://www.chiefdelphi.com/forums/sh...d.php?t=126102

A team alumnus of ours referred us to that same bug report yesterday. We tried reproducing the error by temporarily disconnecting the robot, but weren't able to trigger the issue. It seems to strike at random, usually when I'm not paying attention. We're going to try cutting down on our NetworkTables usage and see if that helps.

Also, we'll be switching to an 8-slot soon. I'm always somewhat optimistic that the 8-slot instantly solves all these mysterious problems.

[virtuald]

Originally Posted by bvisness

A team alumnus of ours referred us to that same bug report yesterday. We tried reproducing the error by temporarily disconnecting the robot, but weren't able to trigger the issue. It seems to strike at random, usually when I'm not paying attention. We're going to try cutting down on our NetworkTables usage and see if that helps.

I'm pretty confident that the problem you're having is the problem I reported in the bug report. In particular, the fact that the robot came back after 8 minutes, indicates that it was hanging waiting for the network write to timeout.

Reproducing the error is easy if you replicate the exact circumstances in which the bug occurs. Two things have to happen:

• Your robot has to write to NetworkTables via PutNumber, etc for it to freeze. If no write attempt occurs, no freeze occurs.
• A NetworkTables client (like SmartDashboard) must be actively receiving data from the robot, and then you must disconnect the client *without* sending a reset back to the robot. You cannot close the program as that will cause a reset to occur. Instead, you must disconnect the wireless/ethernet cable. Sometimes after reconnecting the cable, the robot will start working again.

I've found that some wireless cards can induce this behavior more easily than others. For example, our driver station laptop causes this to happen very easily, whereas I find it difficult to randomly get this to happen on my work laptop. I theorize that wireless interference can cause the connection to drop randomly, but vxWorks doesn't always pick up on it and it hangs.

The freeze is caused by the send buffer filling on vxworks for the network connection. So, if you send data less often, you will be less likely to run into this problem -- but the potential for a freeze is still there.

[bvisness]

Thanks for the tips. Oddly, I was never able to reproduce the problem when I wanted to. I tried disconnecting the ethernet cable from our driver station for varying amounts of time but it seemed to have no effect. Then it would just be sitting there, perfectly tethered in, no one touching the robot or pc, and it would freeze.

I could try to make sure that a PutNumber is constantly trying to write, but I think our robot already does that - we have sensors actively writing to the dashboard whenever teleop is running.

[Aren Siekmeier]

The last few days we have been seeing problems again, seeing the RobotDrive timeouts. They typically (and somewhat consistently) begin about a minute into the robot being enabled in teleop (regardless of whether auto ran), and we are required to reboot the cRio because it is completely unresponsive. The SmartDashboard doesn't get any updates from switch states and the robot does not respond to any inputs. NetConsole shows a new RobotDrive safety timeout (the same as the one posted in the first post) roughly every 80ms. This makes sense for an output safety timeout, and suggests that some threads are still in operation while some other ones, particularly the ones that update outputs and the SmartDashboard, are hanging. We have not been seeing any more of the FDIO errors, only RobotDrive timeouts.

We have had other things to test and can still do so by rebooting the cRio, but we haven't a great idea where to start debugging this problem. We are in the process of rewriting our most basic functionality in other frameworks (both LabVIEW and a C++ IterativeRobot Project) to get away from the Command Based model. It seems likely that our set of commands (all of which seem pretty standard) might be finding some edge case that causes the hang. Does anyone have any more insight into this?

If it is network table writes as has been suggested, why do these timeout? Do we really need to be guaranteeing packet delivery on our SmartDashboard updates? It seems something like UDP would be fine in this case, and would avoid timeout issues.

[Aren Siekmeier]

So I actually read all of that other thread now... it sounds like we are probably seeing the same bug with Network Tables. It looks like the bug is being worked through and a patch is in the works. In the meantime, I think we will also try establishing a separate thread dedicated to the SmartDashboard updates, so that the output threads can still keep up.

[virtuald]

Originally Posted by compwiztobe

It looks like the bug is being worked through and a patch is in the works.

There is a complete working patch that should solve the errors you're seeing. It's attached to the bug report. If you're having this much trouble with it, I'd recommend either (a) dropping NetworkTables completely and roll your own UDP solution or (b) compile/patch your own WPILib.

If you want, I can compile the WPILib binary tomorrow morning and send it your way.

[bvisness]

I'm fine installing the patch as it is now, but I'm not sure what the process for that is. Would I have to download the source and compile it myself for that to work? If so, I would appreciate if you compiled the binary for us...

[virtuald]

Steps:

• Unzip your WPILib source code. It's located at c:\WindRiver\Workspace\WPILib\WPILib...zip
• Download and apply the patch to the WPILib source code
• Build WPILib in WindRiver and use your custom WPILib

I'd also recommend changing the constant at RobotBase.cpp line 22 to indicate that you've changed the code. It currently reads "C++ 2014 Update 0". You can view this string in your driver station diagnostics, so if you change it then you can know you're running your version of WPILib.

I'd give better instructions, but I don't have my Windows computer home with me.

[virtuald]

Note that the full patch is attached to the issue, and the patch in the gist link has been deleted since it's incomplete.

[virtuald]

Sorry for the delay, today was busier than expected. I've posted a compiled binary to http://firstforge.wpi.edu/sf/go/artf1719 . Unfortunately, I haven't been able to verify it on a cRio as I don't have access to one at the moment. However, the original WPILib binary is 13.0 MB, and this one is 13.1 MB, so I expect that it should work.

Let me know if this helps your issue!

[Aren Siekmeier]

Originally Posted by virtuald

Sorry for the delay, today was busier than expected. I've posted a compiled binary to http://firstforge.wpi.edu/sf/go/artf1719 . Unfortunately, I haven't been able to verify it on a cRio as I don't have access to one at the moment. However, the original WPILib binary is 13.0 MB, and this one is 13.1 MB, so I expect that it should work.

Let me know if this helps your issue!

We applied this patch, and rebuilt and redeployed our code. It appeared to improve the problem at first in that it gave the same FDIO errors as we saw previously, about as frequently as we had been getting the timeout errors, and we did not appear to be hanging. However (and we don't really understand what happened here), after a very short time, it started timing out immediately when enabled, giving the same error at line 117 of MotorSafetyHelper.cpp. One interesting thing to note is that when disabled all these errors stopped, so it seems to still be making it through the control loop in IterativeRobot.cpp (StartCompetition()). We have most of these error messages logged, so as soon as someone has access to those text files we will post those.

We have ported all of our code to Java and LabView, and neither of these implementations seem to show the same problem, though we hope to run the bot into the ground a little more over the next few days to tease any more errors out of it. Some C++ code that doesn't use Commands or the SmartDashboard/NetworkTables is in the works so testing that may also give some insight. We will probably stick with the Java implementation for the rest of the season.