Does anyone know what was causing all the field issues at Houston? Most of the fields experienced some sort of issue that resulted in significant lag and dropped packets. A couple of fields seemed to have it worse than others. The issues did not seem to get resolved. We had a number of replays of matches on both Thursday and Friday. Even on Saturday, there were reports of people still experiencing issues. The issues did seem to get better as the week went on, but never really went away.
I heard from several teams that the issue was Wi-fi related but didn’t really seem to be caused by personal hot spots or cell phone wi-fi usage. Some teams were speculating that the issue was caused by the Wi-fi at the convention center, but if that was the case, it seemed like the FTAs should have been able to fix that sooner and more completely.
I hope someone figured it out and is able to prevent it from happening next year. Several teams broke their robots when they were not able to respond to commands in a timely way. We broke the external casing on 2 NEO motors (in two separate matches) due to the robot running into the cargo ship. We also broke the cargo ship in one match when the robot got stuck underneath it and we needed to yank ourselves back out. One of our alliance parners broke the chain on their manipulator when it over-traveled due to the lag. And several other teams were complaining of damage to their robots as well.
While I am not complaining that this created any sort of unfair conditions (everyone suffered from the same lag), it is not fun to compete under these conditions. I am really hoping that this can be prevented next year. If anyone knows the real story, I would love to hear it.
I don’t know the cause, but I do know that down on Carver, we had only 1 replay (to my knowledge), and the fact that Carver was furthest away from the pits and only had 1 neighboring field, it makes me think that generally wireless interference was the culprit, it was just too noisy on most fields.
Even though carver officially only had 1 replay, the field issues were continuous throughout. Looking at our driver station logs from any of our matches the packet loss was insane.
Our drivers also complained several times about lag interfering with the robot operation, including (but not limited to) robot continuing to turn for half a second even after letting go of the stick, our limelight network tables not communicating for a brief second and causing our drive train pid to go haywire, and the driver camera stuttering frequently.
I’m also aware of several other teams on carver that had similar issues. We showed the FTAs the logs several times, just to make them aware of it (and they did acknowledge the insane packet loss), but we personally never pursued a match replay since none of our match outcomes were affected by it.
On the other hand, Newton, which was right next to Carver, probably had the most replays. I think we had some teams playing 3 replays. Seeing the latency graphs teams were bringing (including ours) to the FTA was mind-boggling.
Big thanks to the multiple FTAs who kept the field going.
Yes, Carver did have some issues, but not as bad as Roebling:
Wi-Fi interference was most definitely one of the culprits, but I believe that the constant pinging of the FMS APs by devices with Wi-Fi turned ON (therefore searching for networks) was the biggest issue here. The FMS APs had to constantly reject SSID requests from thousands of devices each match from my understanding.
As for turing, Wi-Fi signals didn’t have a wall to bounce off of over there, so those signals drifted off into the pits, therefore making Turing the least-affected field.
All in all, hotspots aren’t as much of an issue as client Wi-Fi being turned ON is. I assume most hotspots broadcast in the 2.4Ghz spectrum, which shouldn’t be as much of an issue.
Newton this year had the most issues of any field I have seen. I recall the MC saying that we had 12 replays on Friday alone. My team had to play three replay matches back to back to back (the last one was a replay of a replay), and my driver had issues with lag every match. Eventually the FTA requested us to turn down our camera stream even further to 2 Mbps (we had been running at 3 Mbps before that), yet we still had lag issues.
These lag issues were the worst on Friday, but was noticeable on Thursday and Saturday as well.
I would have assumed the issue would not have happened at MMP, as the field was far from the stands and there weren’t any other fields playing at the same time, but teams have reported issues with controller latency every match.