Reliability of Onboard I2C port for 2022

The known issues page mentions that using the onboard I2C port could cause the roborio to hang.

Is this issue something new this year or has then always been present? Our team has been using the onboard I2C port for a BNO055 gyro for the past 4 years and haven’t experienced that specific problem. Should we stop using the onboard I2C port? If we don’t experience this issue while testing, does that mean it’s unlikely that it will occur during a match?

2 Likes

Teams started reporting issues in 2020, presumably related to a lot more teams using the onboard i2c port with the Rev Color Sensor. For example, see this thread: Robot Keeps Losing Com and Code

You can try to use some stress test code to determine your roboRIO’s susceptibility. Note that code has many more i2c calls then a program normally would in order to induce the problem faster.

Important to mention that your roboRIO’s susceptibility can change with each roboRIO image! It’s a nasty bugger

There’s a possibility we experienced this today. I updated our 2020/2021 bot with new 2022 software, and we got no comms a few times today. I believe we were still connected to the radio, but I’ll have to double check next time this issues comes up. After losing comms, I think it randomly reconnected like a minute later.

Might be the same issue, might be something else. I think we’ll try to stop using the onboard I2C port now just to be safe.

If it reconnected, it’s not the same issue. Seems more likely to be a loose wire and a component rebooted. You can use the driver station log viewer to determine what component rebooted: Driver Station Log File Viewer — FIRST Robotics Competition documentation

To paraphrase others over the past year - “mired in the weeds with a nasty bugger is a very uncomfortable situation.” We had not experienced I2C related lockups before the 2022 roboRIO/WPILib releases. Now there is no end of trouble - some of which is not reproducible and some that is. I’ll ignore the inexplicable, not reproducible stuff here.

Running a moderately complicated robot program with only a LIDAR-Lite v2 on I2C onboard bus always fails. The failure is the robot program stops doing most but not all of its work of looking at sensors and DIO and displaying values and running a motor. The length of time to failure varies from a few seconds to several minutes and is correlated with how much CPU time is being used and maybe how much other digital I/O is being used. For example, with lots of stuff running the failure is almost immediate. When we turn off the navX on the USB port (or some other stuff) the CPU utilization goes down and the failure happens after several minutes.

Run the same moderately complicated robot program with the LIDAR-Lite v2 on the MXP and everything runs fine indefinitely.

We notice that running on the I2C onboard bus requires more CPU and is slower (lower acquisition cycle rate) than using the MXP I2C bus.

The stress test referred to others runs without failure for us on both I2C onboard and MXP. The not reproducible failures included the roboRIO becoming a brick as described in various posts. Other weird not reproducible failures happened with various RevRobotics color sensors.

We won’t be using the onboard I2C has we have in past years for the navX, LIDAR-lite, or color sensor. MXP works almost reliably (it did experience a few not reproducible errors).

How does one connect I2c to the mxp port (without a navx)? We’re also using a multiplexer if that matters. Would the REV more board or similar be necessary?

2 Likes

Here’s one solution using the same mux

4 Likes

For reference here’s the MXP port users guide:

It contains a diagram that includes the power, GND, SDA and SCL signals.

2 Likes

MXP Update - same program that fails in a few seconds to minutes with onboard I2C bus does eventually fail the same way using the MXP bus but it did take hours instead of seconds to do so.

3 Likes

Is this with WPILib 3.1 and the 4.0 image? Are you running a slightly modified version of the stress test above?

Will_Toth modestly presents “one solution.” I dare say that’s the best solution for beautifully done and very low cost.

If you can’t wait for a RevRobotics protoboard that is shown, then a solid solution is find an old floppy disk drive ribbon cable in the junk box.

Flimsier solution is plugin smaller connectors and hot glue them down. They won’t come out during driving but can be removed if you aren’t sloppy and don’t get glue in the connector.

1 Like

We are running the latest releases until yesterday when 3.1 / 4.0 was released. Now we are behind again and get to retest after updating; something to keep us busy Saturday.

We ran the posted stress test unchanged. It runs fine although around the same time we did get complete lockups of the roboRIO with our own code that couldn’t be reproduced on demand. For example, our code bricked almost every time one day using the MXP so I thought the known issues which said onboard might have been a typo so I reported that. We turned off the roboRIO for the night and turned it on the next day and it ran the same code fine never to brick again.

The modest “stress test” that we are running is just a little bit of several different functions sampler of WPILib and hardware I/O for a learning tool for the students. We’ve used it for years with slight variations and this year it fails.

We’re trying to figure out how to get output from the robot code so we have a better idea what’s failing. We have a few ideas we haven’t tried yet - maybe Saturday after the updates.

1 Like

Appreciate it!! This will be very helpful.

I’ve added an example Java project with our implementation for reading multiple color sensors in a thread. https://github.com/FRC3005/I2CMux-Example

This uses ports 1 and 2 on the mux chip matching the configuration in that picture.

3 Likes

So… running this (I2C Mux on the MXP port) on two roboRIOs on my desk worked just fine. One roboRIO 1 and one roboRIO 2. But running this on a third roboRIO 2 on a practice bot and the whole thing falls on its head. The symptom is each I2C call takes a long time.

Haven’t been able to debug the issue much, since I can’t actually recreate it in a more controlled environment. My best guess is maybe on the practice bot there is a bit more noise on the bus or something, and its not handled so well?

At this point we’re just going to drop I2C altogether. It’s not worth the wasted time trying to debug.

2 Likes

This thread has been immensely valuable in highlighting I2C port issues. Thank you to all who contributed their experience.

3 Likes

My students want to use color sensors on the balls while they’re inside the robot. I am mostly a MechE. Help?

Starting from, “I shouldn’t use onboard or MXP I2C based on these bugs”, and “I only know about Arduino variants”, are any of these architectures a good idea? Or is there a better one?:

Arduino Due for 2x I2C input lines https://docs.arduino.cc/hardware/due
It seems like a lot of board footprint when I could multiplex, since I dont need the information arriving simultaneously, and I could borrow Will’s circuit.

Arduino Uno for 1x I2C input, multiplexed
Good news: Plenty of IO options, I2C availabilitiy guaranteed on arduino.cc, Will’s already designed and tested a multiplex circuit I can “borrow”
Bad news: Still really bulky for two sensor readings?

Arduino Nano claims to have I2C but doesn’t list it on the official pinout drawing, but Random Blogpost says use SDA>A4, SCL>A5? ??? I can’t find this documented at arduino.cc though
If that’s real, this seems like both the cheapest and smallest way, again assuming I multiplex the input with Will’s circuit.
There may be communication to RIO issues here that I’m not thinking about, but there are at least 4 DIO available I could use to transmit boolean information.

NEXT PROBLEM, Information to the roboRIO?

Is there a “good way” to communicate to the RIO with Arduino, and forward entire readings?

Fortunately we have a bunch of DIO available on our RIO, so I’m thinking something dumb like just have the Arduino sketch raise and lower four DIO flags (“Red@Sensor 1”, “Blue@Sensor 1”, “Red@Sensor 2”, “Blue@Sensor 2”), instead of actually connecting and forwarding whole readings?
I’d have to get the tuning right and then be comfortable Never Touching The Black Box Again, rather than being able to see live readings in code.
Has anyone else used the Rev Color Sensor in various lighting conditions before? My nightmare is we get it tuned in the lab and then lighting differences wreck our performance on field, and then I cant see that because I’ve hidden all the real sensor code in an Arduino sketch giving me booleans instead of readings on the RIO…

This is a more advanced option, but if you have one, a Raspberry Pi also can be set up with ~6 I2C busses, so it could be an option to read the sensors with I2C and send it via NetworkTables to the RoboRIO.

2 Likes

Hey @Will_Toth , any chance Rev could make the I2C address configurable?
That could let teams move color sensors off the default address and get multiple sensors on the bus… (doesn’t do anything about the RIO’s issue of course)

Ooooh. I like this, because it would make it look just like vision targeting to my new programmers. I don’t like this, because I don’t know how to make NetworkTables entities populate outside of the WPILib garden/prebuilt images. There will be a lot of learning on this path that i don’t know if I have time for.

Could I save a bunch of time here by starting with the WPI Vision image? That way there’s already a piece of code that’s trying to run NT at startup, and I can just go into it and add reading the I2C values to specific NT keys? It looks like there’s a library to read I2C in the Java core SDK if i can figure out how to use it… I2CDevice (Device I/O API 1.1)

On the Robot side, I would want an EntryListener to elegantly trigger processing associated with a balls arrival in front of a sensor? Some filtering/binning of the color values would go there, and then actions triggered when conditions were met.