NavX2 disconnecting reconnecting intermittently (not browning out)

Technical
, ,
1

We have been using navx through rio MXP for a couple seasons now and have run into this new issue this season where it would spam disconnecting / reconnecting / reset detected. This happens randomly (every 10 ish seconds) only when we are enabled. I am confident it is not a power issue because although the bot is enabled, we aren’t really driving it at the moment, and it happens when the bot is enabled but completely idle as well. I see that there are similar issues already on CD, but are all related to browning out which we aren’t having. There are also no wires run over the navx so I dont think EMI is an issue.

Specifics

  • Rio2 is newly imaged for 2023
  • using 2023 WPI - KauaiLabs_navX_FRC 2023.0.3
  • connected via MXP expansion port on rio
  • CPU utilization is not high

When initializing the navX to use SPI (protocol we used last year with no issue), it would spam disconnecting / reconnecting / reset detected randomly every 10 ish seconds.

When initializing the navX to use I2C, console messages go in the order of:

  • navx disconnected
  • navx reconnected
  • *and then robot comms die (super weird but this happens consistently)

This is mostly just a drivebase at the moment and the rio just has power, ethernet (via rev RPM), an empty canbus loop that is terminated to the REV PDH, USB to canivore with a non empty can loop, and of course the navx2 connected via MXP port on rio2 - properly screwed in.

This was tried with 2 navX2’s. One was brand new, the other was from our 2022 bot which had none of these issues.

I checked the physical pins and they all seem unobstructed. I also shook the robot and carefully tried to wiggle the navx while enabled and found that the disconnections were not related to these physical disturbances.

The root of the issue might be either electrical, firmware, or somehow software? (is there any way code could cause this??) . However, electrical seems fine as it is just the MXP port and we aren’t browning at all or anything and firmware is all 2023… So, I am kinda stuck on this issue.

Any help, questions, ideas, critiques would be greatly appreciated in solving this problem.

Thanks in advance!

3 Likes
2

Shouldn’t you initialize the NavX using the SerialPort.Port.kMXP argument if you are using it through the RoboRIO MXP expansion port?

1 Like
3

Sorry for not being clear, but yes, we do pass in the serial port as well as the serial protocol.
Here is our init:

try
{
navx_ = new AHRS(frc::I2C::Port::kMXP);
}
catch (const exception &e)
{
cout << e.what() << endl;
}

also tried SPI instead of I2C as outlined in original post.

Thanks for the response.

4

When using I2C, you say the DS Comms light goes red. Does it come back or do you need to power-cycle the rio first? If you need to power-cycle it, that sounds a lot like this known issue (which officially only affects the onboard port, but anecdotally seems to happen with MXP I2C as well in a few rare cases): Known Issues — FIRST Robotics Competition documentation

SPI and Serial should work just fine though.

Just for clarity, when you say you tested a 2022 model, are you saying it had the same behavior or that it worked correctly?

1 Like
5

the 2022 model worked correctly, without problem. We used the navx2 and rio2 all season for RAPID REACT via MXP with serial protocol of SPI.

I am aware of the issues with I2C but I thought to test it just to gather more data… Ideally we wouldn’t want to use I2C. The I2C known issue says a workaround is to use the MXP I2C (which we are), but yeah i’m guessing we are running into I2C lockups because we did have to power-cycle when the comms went red.

Thanks for the response, let me know if that clarifies.

6

TROUBLESHOOTING UPDATE

When replacing with a navx1 through the same(and only) rio MXP and initializing via SPI, it works just fine…but we want to use navx2 for the improved yaw values.

Also, a blank code file with just the navx init doesnt change our outcome from the original post.

7

Check for metal shavings in between the pins of the navx.

8

combinations of 2 different rio2s, 2 navx2s, 1 navx1 - all visually inspected all have the same intermittent disconnect reconnect issue.

With an isolated system of just a power supply, radio, navx not on robot- same issue happens

Navx1 works fine on both rio2s.

9

Are you saying one NavX doesn’t work right but these 2 other ones - a new one and one from last year - do work right now with the same connection?

I had that situation and your screen name says it all. I had three doing the same thing - one failed randomly and fairly quickly and often and 2 worked fine. I threw mine away.

If you are saying all your NavXs fail on that port then you need to test the port - loopback or some other device you know works.

1 Like
10

I’m pretty sure we’re having the same issues. We have a RoboRIO 2 with a NavX2 plugged directly into the MXP port. We have the latest NavX2 library. It seems to take a long time to connect sometimes, but we also experienced it just randomly disconnecting tonight.

This isn’t a new NavX2. It was our backup from last year, and was used a few times (with a RoboRIO 1 and 2022 code) to make sure it worked.

This is concerning.

1 Like
11

hmmm,

yeah im still not sure what to do at the moment…

For everyone, here is the collection of all the combinations we have tried. Combining our 3 rio 2s and 4 navXs. And all parts were visually inspected, in fact one of the rios and navx2s are brand new.

Using NavxUI, I verified that all the navxs are writing good data. And when the navx isnt in its disconnect, reconnect cycle, we do see good data on smart dashboard from it.

12

they fail on all the ports when using 2023 image and navx2, here is a spreadsheet of all our data.

1 Like
13

Can you verify the actual firmware version you’re running in the NavX2? I’m pretty sure we’re not on the latest firmware though. But if you’re on the latest, then I don’t suspect that’s the issue. From your data it points towards a difference either in the 2023 Rio2 image, or the new 2023 Kauai library.

1 Like
14

Good presentation and clarity of experimentation. I agree with @Programming4907 that you should add the firmware level to your spreadsheet to document that important check.

At some point soon you have to decide if finding blame and then resolution can be completed before your first competition. Speculating what has happened is interesting to a point but then you have to move on.

My team discovered the day before our first competition last season that the NavX micro on USB didn’t play nice with CANivore on USB (couldn’t risk the I2C ports). We ran away from the NavX and used team family and friends for a Pony Express to CTRE to procure a Pigeon 2. We had no time and no choice and ended up loving our Pigeon that only took a few minutes to change the code to work.

15

Updated the spreadsheet with navx firmware versions. I only updated 1 navx2 to the latest firmware so I would have a range of versions to test with. All versions are supposed to work right?

2 Likes
16

Our navx2 also randomly decides to go into a disconnect/reconnect loop, out of nowhere at times.

image
image750×714 127 KB

It hasn’t caused any issues for us so far though,just a strange thing to see on the riolog

2 Likes
17

Our solution was to keep the navx in the mxp but just communicate via usb serial protocol as it is just as fast. Kind of a workaround, but it works.

18

Last night we had a situation where our 2022 robot (with a RoboRIO 1) which had been upgraded to 2023 WPILib, etc., had the same error of the NavX2 suddenly disconnecting and reconnecting repeatedly.

So I think that points to a 2023 image problem, not a RoboRIO 2 problem.

We’re trying the serial port option (using one of the USB ports on the RIO).

19

How do teams report stuff like this?

20

Probably by putting an issue on the WPILib GitHub repo, or trying to ping one of the maintainers of WPILib (@Peter_Johnson @Joe_Ross @Thad_House). As this can be either a WPILib issue of an issue with the RoboRIO, they would coordinate with NI to come up with a fix/workaround to this.