My team has been working on a command-based Python project for our 2018 robot this offseason. Our project has four subsystems, about 20 commands, five sensors, and five PID Controllers. Recently, we realized that our code is actually not running fast enough on the RIO, dipping below 50 Hz to ~45 Hz. We were initially getting even slower runtimes (~15 Hz), but were able to reduce sensor calls and Smart Dashboard updates. Is this normal for a Python project of this size? What about other languages? Are there any specific practices we can do to reduce our runtime? Thanks in advance!
I gotta say, the thread title initially confused me; “runtime” usually refers to a some software that supports the execution of a program.
A 45Hz control loop is probably not a very large performance loss (although ~15Hz certainly is!).
My team also had some pretty severe performance issues during last build season (although it only occurred to the team to investigate performance being the source of our problems after Dustin packaged up the loop timing code I had originally written to time TimedRobot).
We had 11 Talon SRXs on our robot last season, and managed to trace down the cause to some inefficient code in the robotpy-ctre WPI_TalonSRX.set() method (which we since fixed during the season, luckily).
I suspect most performance issues that RobotPy teams face are due to how threading works in CPython (not going to go into the technical details here, but read about the GIL if you’re interested). Reading from a sensor in itself shouldn’t be expensive.
Now, as it turns out, the WPILib PIDController of seasons past runs in a thread. NetworkTables also uses threading. So in your specific scenario, your robot code would be using at least 7 threads (counting the main thread). Python on the roboRIO is probably having quite a fun time switching between your different threads, especially if your PIDControllers have a low period…
You could try utilising the WPILib PIDController currently aimed to land for the 2020 season. I’ve backported that here: https://pypi.org/project/wpilib-controller/.
If you’re up for getting a bit dirty, you might consider profiling your robot code using a profiler such as py-spy: https://github.com/benfred/py-spy. Without more concrete data, almost everything in this thread is basically conjecture.
I experienced the exact same issue. Running my code through cProfile showed that the major culprits were a) PDP status calls, b) Talon SRX read calls, and c) LiveWindow calls. I believe © has been fixed. I haven’t had the time to investigate more thoroughly though.
https://gitter.im/robotpy/robotpy-wpilib?at=5afbaf15e1cf621dba221eac
This should also be fixed in the 2019 season. (Currently in the process of updating RobotPy for the 2019 changes.)
I’d echo everything that auscompgeek said.
The PDP in particular causes lots of delays (in all languages, not just RobotPy), but that should be solved for 2019.
One thing to look at is the CPU usage on the RoboRIO. If it’s at 100%, you’re definitely going to run into problems.
One thing that isn’t obvious is that running the sample cscore processing code (that doesn’t do anything!) at 30fps will cause the CPU usage to go through the roof. Don’t do that. However, doing processing at a lower frame rate is fine – but doing image processing on the RoboRIO always requires a fair bit of optimization, especially in Python.
Running CameraServer without processing is very efficient and shouldn’t cause more than 5% overhead, but this is true in all languages.
The loop timer is really useful for measuring execution time. Getting some profile data would be great… I think the answer is that historically, teams have been able to do some fairly complex things with RobotPy. However, it’s totally possible that there’s just something weird we’re doing somewhere that would cause things to be really slow. If you can isolate those things via profiling or whatever, that would be extremely useful.
I don’t really like using the WPILib PIDController in python (because of the extra thread), but try to use TalonSRX instead.
Thank you for your response and sorry for my delay. During the offseason, we changed our rate to 30 iterations per second, and after reducing CAN calls were able to achieve this speed. This year with the 2019 RobotPy, we are once again running into some issues with our runtime (currently around 18 fps, but not fully optimized yet). A few questions:
- Are the PDP status calls taking less time in 2019 RobotPy?
- Would using the Limelight and pushing the stream to Shuffleboard slow down runtime?
- Is the cscore processing code automatically enabled? Do we have to disable it somehow? We are using the limelight for vision anyways.
- Are there any alternatives to WPILib PIDController? We are using Grayhill encoders on DT and need to do PID on them + NavX for motion profiling. Would using the 2020 version reduce runtime?
- Is there anyway to test these issues away from the robot? I tried using the loop timer, but it gives flat 50 fps on my computer (makes sense with a much much stronger processor). Would you recommend py-spy?
- Is it still possible to reduce the set period in Robotpy 2019? We are comfortable running at around 30 fps if necessary.
Sorry for all these questions, and thanks in advance!
- Haven’t tried it yet. It was a HAL issue, and I believe the underlying issues in the HAL were resolved.
- Pushing the stream? What do you mean?
- No, you have to enable it by calling
wpilib.CameraServer.start() - Defer answer to @auscompgeek
- Not that I know of
- The set period?
I think LiveWindow is enabled by default? You might try disabling it… I’ve had a strong suspicion that excessive networktables usage causes issues, and LiveWindow does this indirectly. I haven’t had the opportunity to dig into it. Profiling data would be awesome and help us figure out what’s going on.
Thanks for your response @virtuald. I’ll try and test some of these out at our meeting tonight. By pushing stream, I meant having the Limelight stream show on Shuffleboard. Does the setPeriod() function from last year still work (some members on my team were unable to get it working)?
We’re really not sure which function you’re referring to.
No clue. Hasn’t been battle tested yet. But it does allow for an intuitive way of avoiding having to run it in another thread.
The only two things involved there are the Limelight and Shuffleboard. The roboRIO is not involved there, aside from letting everything know where the stream comes from.
Thanks for the clarification. Today we ran our code and slowly removed different aspects to test what was contributing to the longer period. We found that the PDP/CAN calls don’t take that long, and neither does the Limelight. The majority of the time goes into the scheduler for each subsystem in our command based project. We have a handful of mechanisms, and combined this reduces our frequency from 50 to about 20/sec. Has anyone else run into issues specifically with the Command Robot or know how to speed up the command scheduling process?
Hm. Well, the scheduler code copies what Java does, but it’s certainly possible that some inefficiency has crept its way in without anyone realizing it. I don’t particularly like the command style of doing things, so I’ve not dug in here significantly.
In theory, profiler output even during simulation should be useful for finding hotspots and for guiding optimizations. By eliminating the hotspots, should work better. There are a variety of micro optimizations that could be applied to the code too… but best to figure out where particularly the time is being spent before going down this route.
@Strategos have you learned anything more about the scheduler runtime, or have profiling data related to it?
Currently we are focusing on finishing the robot mechanically for bag and tag. We have temporarily set our period to 50ms to avoid overrunning our loops. After bag and tag, I’ll look into a profiler to locate the hotspots. I’ll keep you all posted!
If it makes you feel any better, we’re using Java and we’re doing the same thing. We’ll probably reduce our SmartDashboard and Shuffleboard usage when we aren’t debugging code and we’ll also try to cut down on TalonSRX reads like said above.