RoboRio Loses Communication Until Reboot

We use our RoboRio on our practice bot and this week we encountered a problem where sometimes the RoboRio will disconnect from the DS and won’t go back online unless we reboot it. We tried reimaging the RoboRio but that didn’t help.
We then replaced our RoboRio on the practice bot with a spare one, but that didn’t help either.

In order to help you help us more I took the first RoboRio (the one that is no longer on the bot) aside and connected it only to a 12V power supply and a laptop (through USB or Ethernet). I managed to recreate the problem (which was very hard as it happens randomly) and gather some info.
The first thing I see when the problem occurs is this (laptop is connected to RoboRio through USB):

I can see the USB is recognized but no other signs of communication.
All other adapters are disabled:

I cannot ping the RoboRio:

I try to disconnect the USB and reconnect it, but now it is not recognized anymore:

The adapter is gone:

Windows complain that the USB device is not recognized:

The RoboRio thinks everything is good:

Even when not connected at all:

Ethernet connection doesn’t work either:

Can’t even ping:

Back to USB, I reboot the laptop and it doesn’t help.
The RoboRio Imaging tool can’t find the RoboRio.
I took the logs out of the RoboRio (from /var/local/natinst/log/). I couldn’t find anything in them, but maybe you can:
errlog.txt (201 Bytes)
FRC_UserProgram.log (78 Bytes)
kern.log (6.2 KB)
kern.log.1.gz (849 Bytes)
lvlog01-29-20-18-55-22.txt (2.1 KB)
lvlog01-29-20-19-40-10.txt (2.1 KB)
lvlog01-29-20-19-57-06.txt (4.7 KB)
lvlog01-29-20-19-57-11.txt (2.1 KB)
lvlog01-29-20-19-59-25.txt (2.7 KB)
lvlog01-30-20-14-50-29.txt (2.9 KB)
lvlog01-30-20-15-09-33.txt (2.7 KB)
lvlog02-03-20-16-33-40.txt (2.5 KB)
lvlog02-07-20-06-24-44.txt (2.5 KB)
lvlog02-07-20-06-26-52.txt (4.3 KB)
lvlog02-07-20-06-43-40.txt (4.2 KB)
lvlog03-15-19-19-19-45.txt (417 Bytes)
lvrt_19.0_lvuser_cur.txt (4.3 KB)
lvrt_19.0_lvuser_log.txt (2.5 KB)
nirio_emb_can_log.txt (1.1 KB)

Any help or direction is greatly appreciated! Thank you all!

1 Like

Aside from the fact that it looks like the RIO has been beaten within an inch of its life (what is going on with that ethernet port?!), have you tried factory resetting it? If you contact NI, they can send you the software to bring it back. I would also get a can of air and make sure that there are no metal shavings or bits of schmutz in any of the ports, although if something shorted, you might be SoL.

If it shows up in the Roborio imaging tool and sometimes works over USB, there’s not really a need for the recovery process. That’s really only useful for failed firmware updates.

A reimage is worth trying, though I’m not sure it’ll help. I agree, that Rio has seem MUCH better days. I also recommend trying another USB cable to confirm it’s not some kind of intermittent break in the cable.

True, but if the imaging tool can’t see the RIO then it might be kinda hard to re-image. Also, if the connection were to drop in the middle of a re-image, you would have a $435 door stop. The firmware is still at 6.0.0f, right?

I think the value of that particular Rio has deprecated quite a bit, so it’s more of a $50 doorstop at this point.

Joking aside, you can usually recover from a failed reimage fairly easily. It’s a failed firmware flash that gets complicated (and requires the NI recovery.cfg process). I agree that a bad USB cable or port would certainly make reimaging difficult. Swapping out the cable with a known-good one will help reduce the likely cause.

@Omer_Huly, can you confirm what firmware version is on the Rio?

The RoboRio’s firmware version is indeed 6.0.0f.
We successfully reimaged the RoboRio several times, but the problem still occurs at, seems like, random.
Also, after rebooting the RoboRio, it works just fine until the problem happens again.

Have you opened up the roboRIO case and cleaned out metal dust and shavings?
Including inspecting underneath pin plastic guides and inside each comm port?

Actually, we didn’t. I’ll give it a try the next time I’ll be in the shop.
But, the same problem occurs with both of our RoboRios and I find it hard to believe that they will both have the same metal dust and shavings inside which cause the same issue.