Our team was running Phoenix framework 5.17.6 and we updated to 5.18.1. Talon SRX firmware was at 20.0 and we updated to 20.1.
We’re running the ADIS 2020.r1 release.
When we updated the CTRE driver, it broke the ADIS16470 gyro driver. There is an incompatibility between these two 3rd party drivers. We are calling calibrate on the gyro just before autonomous begins. Everything worked fine until we updated the CTRE driver. After the update, all of the Talons started throwing messages indicating that they are not updated frequently enough. This is the WPILIB safety feature. We’re enabling safety on the SRX’s with a timeout of 100ms. Nothing worked on our bot, not even the Spark Max’s.
We removed the calibrate call on the gyro (it still does a calibrate when the class is instantiated). This got rid of the Talon failures. However, the gyro now only returns 0 for any axis angle.
This issue was first noticed after powering up the bot. We found that if we reboot from the driver’s station after powering up, both the gyro and the Talon’s work properly… no safety errors. This worked with or without the call to calibrate. Rebooting seems to heal all problems… however, any time the robot is powered down and restarted the failures occur again.
We reverted the Phoenix framework and Talon SRX Java library to version 5.17.6 and the problems went away.
Is it ok to run the Talon SRX firmware at 20.1 and the Java library at 5.17.6. We’ll also revert the SRX firmware if necessary.
I won’t say “impossible”, but it’s extremely unlikely these libraries are interacting with each other. To my knowledge they don’t even touch the same hardware resources in the roboRIO.
If you were seeing watchdog errors, it’s more likely that something else changed in your process of updating Phoenix that is causing your code to run more slowly.
In any case this is an extra-ordinary claim and needs extraordinary evidence, so at a minimum we need to be able to reproduce the issue.
Can you please provide your code along with a specific set of steps to reproduce the issue?
If your code is not publicly available on github, you can e-mail it directly to our support: CTR Electronics
We keep all team code confidential unless directed otherwise by the team.
In general e-mailing us is the fastest way to get support, we don’t constantly monitor Chief Delphi.
Yes, in fact 20.1 is the appropriate version as it was released with Phoenix 5.17.6.
This sounds like the gyro-calibrate blocks for a bit, which causes the motor safety to timeout, causing neutral blips the the motor controllers. You could turn off the motor safeties, but I think you need to first get a handle of how the gyro-calibrate works.
The gyro-biasing may not take the same amount of time each time, and the Phoenix update was likely coincidence.
Rebooting from the DS means you are not power cycling the gyro, and perhaps that impacts how long the calibrate takes. I’m not that familiar with the ADI inertial tech. On PigeonIMU, its all done in firmware specifically so that it doesn’t impact the application run-time.
Experiment
Remove gyro calibrate and confirm normal functionality with the different softwares/firmware.
Perform gyro calibrate in disable init (once) and don’t run auton until it is finished. Which means you need to determine how to check this.
Confirm expected functionality. Or if there is misbehavior, zoom in.
Then finally integrate the calibrate back into auton in a way where you are not calling drive routines if the gyro is still calibrating - which is undesirable anyway. Thought I’m not sure this is necessary, I would think you only need to calibrate once per application boot-up.
Other thoughts
I would recommend calibrating the sensor on power boot so that your drive-team can check its disposition before committing to a thumbs-up on match-start. Otherwise if there is a sensor problem, it can’t be detected until its too late.
Additionally, the Pigeon User’s guide mentions in a few places that you must poll the health of the sensor so that if there is damage/wiring-issue, you don’t spin in a circle (write your software to not use gyro if you know its bad). This is good advice for any inertial sensor.
Thanks for your comments. For now, our solution is to use SRX driver version 5.17.6. Everything works fine with this version. When we install 5.18.1, the gyro breaks. We then install 5.17.6, the gyro works again.
I do want to find the root cause and at some point update to the latest revision of all drivers. Given where we’re at in build season and demands on time, I can’t spend too much more time on this now. However, I’ll try to generate a minimal set of code that reproduces the issue over the next few days if I can manage it.
The problem is not necessarily with the SRX driver. There could be a bug in the ADIS driver that the new SRX release exposed. It could be an issue with when the Talon libraries initialize vs. when the gyro initializes and calibrates. Again, rebooting the roborio from the driver’s station fixes the problem so there is some difference between a power up and a reboot… could be a timing issue with how the gyro chip powers up.
In response to Experiment suggestions,
We have tried removing the gyro code completely and running SRX ver 5.18.1 and everything seems to work ok… except of course, we have no gyro for our controls.
The ADIS driver performs a calibrate when its class is instantiated. This is performed in the chip now. However, the main thread is forced to sleep for 4 seconds during this process initially. There are also calls make to init SPI bus that have 10 ms sleeps. There is no option to not perform the calibrate when the class loads. It just runs on power up by default. I’d expect to get timeouts during the 4 second calibration time but the talons timeout forever, even after calibrate should be complete.
We’ve been through many scenarios where we try to start the gyro at different times, e.g. before and after Talons are added with no success. Nothing seems to change the outcome.
Our team is at Palmetto this weekend and the problem with the ADIS chip started happening every time that we power up our robot. The gyro might start functioning and then just stop (actually it failed every time we powered up our robot and rebooting the rio never fixed the problem). We found that errors were being thrown to the console indicating that a buffer had to be at least 4005 or 4080 , etc. after rebooting. The exact buffer requirement varied a bit on startup.
This was probably being caused because the startup time with the field system or when tethered is different that what we experienced when connecting wirelessly and the chip was sending more data than the buffer could handle during startup.
This is also why we probably started seeing the problem when we first added the CTRE phoenix libraries. They delayed startup a bit and enough data was being generated by the ADIS chip to overflow the buffer.
We had a look at the ADIS16470 driver and found the following in the acquire thread:
*/
private void acquire() {
// Set data packet length
final int dataset_len = 19; // 18 data points + timestamp
// Set up buffers and variables
int[] buffer = new int[4000];
A receive buffer of 4000 is created by the driver and the required size is bigger than 4000 under certain startup conditions.
We changed the buffer size to 10000 in the driver and the problem went away. The gyro now works on power up or any time we reboot our roboRio.
The problem seems to reside solely in the ADIS16470 driver. The problems we have witnessed were just exposed by loading the Phoenix library and also to a more severe extent when we disable wireless communication to the bot and tether with the competition configuration loaded in the radios.
In case anyone runs into the issue reported by @Brhea, we’ve pushed updated to the 470 (and 448) Java and C++ libraries to prevent start-up or buffer overflow conditions. For the ADIS16470, the latest version of the driver that addresses these issues is 2020.4. The latest driver should be pushed to your VSCode instance if you used the online vendor library installation method, otherwise be sure to update your libraries using the release available on GitHub.