ANNOUNCING: navX MXP Robotics Navigation Sensor

[Thad House]

We competed with our robot this weekend, and we had more of the navX issues. Probably in 5 or 6 matches, the navX would have the issue where it always returned 0. We were not able to actually figure out if it was the bus issue or an actual issue with the gyro, but have you guys been able to look more into the issue with the busses?

[Dunngeon]

Originally Posted by Thad House

We competed with our robot this weekend, and we had more of the navX issues. Probably in 5 or 6 matches, the navX would have the issue where it always returned 0. We were not able to actually figure out if it was the bus issue or an actual issue with the gyro, but have you guys been able to look more into the issue with the busses?

Was this the same issue that caused your issues in the semifinal match?

You guys are on labview right?

[Thad House]

Originally Posted by Dunngeon

Was this the same issue that caused your issues in the semifinal match?

You guys are on labview right?

The issue in the semi final match was caused by either a bad Ethernet cable or something with the field. The gyro issue wasn't super noticeable, as long as we ran the auto we ran all through eliminations. But we still need to get it figured out, because not being able to rotate at times in auto is a bad thing.

[Shahil_FRC]

Originally Posted by Thad House

We competed with our robot this weekend, and we had more of the navX issues. Probably in 5 or 6 matches, the navX would have the issue where it always returned 0. We were not able to actually figure out if it was the bus issue or an actual issue with the gyro, but have you guys been able to look more into the issue with the busses?

We had a similar issue this past weekend in our QF2 match in autonomous. We believe there was a spi bus or gyro error which led to a value of 0 being returned continuously. This issue caused our robot to enter an uncontrollable spin and the FTA disabling our robot for the remainder of the round. I feel bad for our alliance partners because this issue did not let us move forward to the SF. We could have returned to open loop control upon start of teleop.....

[slibert]

Originally Posted by Thad House

We competed with our robot this weekend, and we had more of the navX issues. Probably in 5 or 6 matches, the navX would have the issue where it always returned 0. We were not able to actually figure out if it was the bus issue or an actual issue with the gyro, but have you guys been able to look more into the issue with the busses?

Here's the issues we've seen so far:

(a) [LabView] an I2C bus hang when communicating w/the navX MXP using I2C (after a simulated stage 2 brownout) in a benchtop test configuration and,
(b) [Java] several I2C bus hangs on our competition robot when using the LIDAR Lite on the MXP I2C bus (in this case, we are currently suspicious of a noise issue corrupting a transmission on the I2C bus, but that hasn't been proven yet).

To get data again from these sensors in both cases required a Roborio reboot (restarting the roborio code did not resolve this issue). We've verified that a Roborio reboot does not cause the navX MXP to be rebooted (the MXP 5VDC rail stays powered across the reboot). This, combined w/the fact the issue occurs w/both LabView, and Java points to a Roborio-side problem, and perhaps at a level lower than the Roborio-side libraries. [All testing was performed w/the latest RoboRio firmware/software versions.]

Our next steps are to refine our test setup to find the simplest reproducible case that will hang the I2C bus (to identify the range of the of time during which a Stage 2 brownout can cause the I2C/SPI bus to hang, and try to get some more info on the bus line states and errors received on the RoboRio side) - and also to reproduce the SPI bus hang on the navX MXP you have mentioned. Once we've got that we'll package up the info and send it along to National Instruments.

Until a resolution is found, these are the recommendations:

- We haven't seen the MXP TTL UART communication path to the navX MXP get hung by the disconnect, so using the MXP TTL UART this is recommended for teams experiencing I2C/SPI bus hangs.

- As noted before, if keeping the navX MXP powered in the face of a Roborio Stage 2 brownout is needed, use the USB interface 5VDC/GND leads connected to a 5VDC, 500mA output from the VRM.

One question for you: have you any evidence of brownouts in your Driver Station Logs during the period of time when the I2C bus was hung? Or do you think another condition may be triggering this case?

[Thad House]

Originally Posted by slibert

Here's the issues we've seen so far:

(a) [LabView] an I2C bus hang when communicating w/the navX MXP using I2C (after a simulated stage 2 brownout) in a benchtop test configuration and,
(b) [Java] several I2C bus hangs on our competition robot when using the LIDAR Lite on the MXP I2C bus (in this case, we are currently suspicious of a noise issue corrupting a transmission on the I2C bus, but that hasn't been proven yet).

To get data again from these sensors in both cases required a Roborio reboot (restarting the roborio code did not resolve this issue). We've verified that a Roborio reboot does not cause the navX MXP to be rebooted (the MXP 5VDC rail stays powered across the reboot). This, combined w/the fact the issue occurs w/both LabView, and Java points to a Roborio-side problem, and perhaps at a level lower than the Roborio-side libraries. [All testing was performed w/the latest RoboRio firmware/software versions.]

Our next steps are to refine our test setup to find the simplest reproducible case that will hang the I2C bus (to identify the range of the of time during which a Stage 2 brownout can cause the I2C/SPI bus to hang, and try to get some more info on the bus line states and errors received on the RoboRio side) - and also to reproduce the SPI bus hang on the navX MXP you have mentioned. Once we've got that we'll package up the info and send it along to National Instruments.

Until a resolution is found, these are the recommendations:

- We haven't seen the MXP TTL UART communication path to the navX MXP get hung by the disconnect, so using the MXP TTL UART this is recommended for teams experiencing I2C/SPI bus hangs.

- As noted before, if keeping the navX MXP powered in the face of a Roborio Stage 2 brownout is needed, use the USB interface 5VDC/GND leads connected to a 5VDC, 500mA output from the VRM.

One question for you: have you any evidence of brownouts in your Driver Station Logs during the period of time when the I2C bus was hung? Or do you think another condition may be triggering this case?

It would not report data on bootup. And there was no way to check if it was actually reporting data until the field connected, and by that point we had to be off the field. We did have the USB cord plugged into the VRM, so I highly doubt our issues were caused by a brownout. Everytime we tested in the pits it came up correctly. Just on the field it would only come up about 70% of the time.

[slibert]

Originally Posted by Thad House

It would not report data on bootup. And there was no way to check if it was actually reporting data until the field connected, and by that point we had to be off the field. We did have the USB cord plugged into the VRM, so I highly doubt our issues were caused by a brownout. Everytime we tested in the pits it came up correctly. Just on the field it would only come up about 70% of the time.

Can you please post the drive station logs for the games of interest, and any other data which might provide some insight as to what factors may be different on the field vs. in the pits?

The logs should point out any power-related issues, and may provide indications as to other factors that are coming into play in the on-field setting.

[rich2202]

We are going to try and incorporate the navX this weekend onto our practice bot. We are going to start with the sample code.

In order to avoid the I2C problem, how do we set it up for UART communications?

Thanks.

[slibert]

Originally Posted by rich2202

We are going to try and incorporate the navX this weekend onto our practice bot. We are going to start with the sample code.

In order to avoid the I2C problem, how do we set it up for UART communications?

Thanks.

The navX MXP can communicate via TTL UART, SPI and I2C; selecting the communication path is a software setting. No hardware changes are necessary as long as the "plug-n-play" installation method is used.

For C++ / Java, currently only the TTL UART is supported in software; simply review the respective page on the navX MXP wiki for pointers on getting started.

For LabView, in addition to I2C, both SPi and TTL UART are supported. We have done pretty extensive testing as have others on SPI, and it is the fastest way to communicate with the navX MXP. Our testing on SPI recently has focused on verifying brownout recovery, and for SPI that it is very robust, too. Our current understanding: if you have connected the navX MXP USB to the VRM, the SPI bus is reliable even if there is a RoboRio Stage 2 (or even Stage 3) brownout - and other teams are using this interface w/out problem. If the USB/VRM backup power source is not present, the navX MXP will get reset when a Roborio Stage 2 brownout occurs, but our testing has verified that the RoboRio/navX MXP SPI interface recovers in this case, too.

However, for those that experience trouble w/SPI, the Labview library also has a Serial Example, as well.

[marshall]

Originally Posted by slibert

For LabView, in addition to I2C, both SPi and TTL UART are supported. We have done pretty extensive testing as have others on SPI, and it is the fastest way to communicate with the navX MXP. Our testing on SPI recently has focused on verifying brownout recovery, and for SPI that it is very robust, too. Our current understanding: if you have connected the navX MXP USB to the VRM, the SPI bus is reliable even if there is a RoboRio Stage 2 (or even Stage 3) brownout - and other teams are using this interface w/out problem. If the USB/VRM backup power source is not present, the navX MXP will get reset when a Roborio Stage 2 brownout occurs, but our testing has verified that the RoboRio/navX MXP SPI interface recovers in this case, too.

However, for those that experience trouble w/SPI, the Labview library also has a Serial Example, as well.

Let us know if you have trouble with the LabView library. We're here to help.

[rich2202]

Originally Posted by slibert

For C++ / Java, currently only the TTL UART is supported in software; simply review the respective page on the navX MXP wiki for pointers on getting started.

Since we use C++, I presume the default settings (TTL UART) is what we want.

[slibert]

Originally Posted by Shahil_FRC

We had a similar issue this past weekend in our QF2 match in autonomous. We believe there was a spi bus or gyro error which led to a value of 0 being returned continuously. This issue caused our robot to enter an uncontrollable spin and the FTA disabling our robot for the remainder of the round. I feel bad for our alliance partners because this issue did not let us move forward to the SF. We could have returned to open loop control upon start of teleop.....

Been thinking about what could be causing this issue.

Over the last few days extensive brownout testing w/SPI has been going on; no matter what scenario we throw at it, the LabView SPI library and the navX MXP keep recovering.

However, there was an update to the navX MXP firmware back on Jan. 20 this year that fixed an issue in the navX MXP where it would hang if it encountered a UART error (which could be caused by electrical noise). Since the UART on the navX MXP is always active (as long as the UART dip switch is enabled), and if you didn't have this firmware fix, the navX MXP could lock up even if SPI is being used.

The latest version of the firmware is v. 1.0.482. You can display the firmware version number using the "Magnetometer Calibration Tool".

If you do have a version earlier than v. 1.0.482, directions for upgrading the navX MXP firmware are here.

[Thad House]

Originally Posted by slibert

Been thinking about what could be causing this issue.

Over the last few days extensive brownout testing w/SPI has been going on; no matter what scenario we throw at it, the LabView SPI library and the navX MXP keep recovering.

However, there was an update to the navX MXP firmware back on Jan. 20 this year that fixed an issue in the navX MXP where it would hang if it encountered a UART error (which could be caused by electrical noise). Since the UART on the navX MXP is always active (as long as the UART dip switch is enabled), and if you didn't have this firmware fix, the navX MXP could lock up even if SPI is being used.

The latest version of the firmware is v. 1.0.482. You can display the firmware version number using the "Magnetometer Calibration Tool".

If you do have a version earlier than v. 1.0.482, directions for upgrading the navX MXP firmware are here.

I know ours is 1.0.482, because I2C and SPI didnt work at all in the version that was preloaded, and looking at the flash drive we used to update it was 1.0.482. So at least on our end I don't think thats the issue. But I will double check.

And just to be clear, our issues over the past weekend were all seen on SPI. If it matters, we have the Update rate set to 60hz, The read is getting called once in the teleop loop, which is running every 20ms, and once in periodic tasks in a 20ms loop.

[slibert]

Originally Posted by Thad House

I know ours is 1.0.482, because I2C and SPI didnt work at all in the version that was preloaded, and looking at the flash drive we used to update it was 1.0.482. So at least on our end I don't think thats the issue. But I will double check.

And just to be clear, our issues over the past weekend were all seen on SPI. If it matters, we have the Update rate set to 60hz, The read is getting called once in the teleop loop, which is running every 20ms, and once in periodic tasks in a 20ms loop.

Thanks for clarifying, the details help; we've restarted the SPI longevity and brownout tests at 60Hz. They've been running for an hour now with no interruption, but it's a 24 hour test cycle.

Since you haven't sent any logs, I trust you've reviewed this and ruled out any other systemic issues including brownouts, Roborio CPU usage, loss of communication w/driver station, etc..

One other thing to check would be to look for any shorting between any of the pins on the navX MXP SPI connector header. These pins are electrically connected to the same MXP SPI bus pins used to communicate between the RoboRio and the navX MXP. If anything shorts the CS, SCK, MISO or MOSI pins (e.g., "swarf", or bent pins), that could cause bus communication failures. One other team encountered an issue where the Digital I/O pins on the navX MXP pin were accidentally shorted together (the pins were bent), so it's feasible this could be going on in your case.

[Thad House]

Originally Posted by slibert

Thanks for clarifying, the details help; we've restarted the SPI longevity and brownout tests at 60Hz. They've been running for an hour now with no interruption, but it's a 24 hour test cycle.

Since you haven't sent any logs, I trust you've reviewed this and ruled out any other systemic issues including brownouts, Roborio CPU usage, loss of communication w/driver station, etc..

One other thing to check would be to look for any shorting between any of the pins on the navX MXP SPI connector header. These pins are electrically connected to the same MXP SPI bus pins used to communicate between the RoboRio and the navX MXP. If anything shorts the CS, SCK, MISO or MOSI pins (e.g., "swarf", or bent pins), that could cause bus communication failures. One other team encountered an issue where the Digital I/O pins on the navX MXP pin were accidentally shorted together (the pins were bent), so it's feasible this could be going on in your case.

I dont have the logs yet. They are still sitting in the trailer. Hopefully I'll have them tomorrow.