My team has a vision system comprised of a webcam and Jetson TK1 which is connected to our main radio. In theory, the RIO acts as a UDP server and receives data from the Jetson. This works well when we’re practicing: the latency is under a millisecond and our processing acquires the target well. However, at competition, we have realized that the Jetson doesn’t establish communication with the RoboRIO until a fair ways into the match! Thankfully, we don’t have any critical systems using it right now.
In my testing, it seems that the Jetson boots and loads my code within 20 seconds. Although I don’t have a way to confirm it, it seems that the Jetson itself is up with plenty of time to spare. However, no packets are received on the RoboRIO’s end until at least 1:20 have elapsed since power-on, according to my quick testing. As far as I can tell, the radio loads and is available for connection before this time.
The Jetson is configured with a static address of “10.TE.AM.50”. The RIO is set to the static address of “10.TE.AM.2” based on the advice of the FTA from one of our events.
The simplest explanation that I can come up with is that the Jetson is attempting to connect to the RIO before our code is loaded, and as a result it caches a failed attempt and retries later. If it has a timeout of one minute, that would explain the timings I am seeing (20 seconds to boot followed by one minute of timeout). However, I’m not entirely sure how I can validate this suspicion, nor how it could be resolved. Have others encountered this same issue? Is there a likely cause that I should be investigating, or might I be mis-understanding the symptoms?
I can do a TLDR of it, there’s a configuration on the 2017 radio that results in being unable to communicate between the two ports. Switching to the 2016 radio seems to have resolved this issue for 900. Unfortunately because the FMS is a black box this is literally untestable in your lab and the documentation on ports and protocols used in the FMS also seems to be out of date.
Double check that your Rio is running static and the gate way is set correctly. Also, what are your subnet masks set to? They should be 255.0.0.0 on all devices (don’t ask me why, but this is the official FIRST recommendation likely because of the competition radio config). Our subnet masks were previously set to 255.255.255.0 and switching them to 255.0.0.0 fixed a lot of our problems.
Normally people around here are asking me to shut up.
Use the 2016 radio. I can’t tell you why the 2017 radios aren’t as good but I’ve seen numerous teams with issues that magically get resolved by switching. It might just be my general distrust of the 2017 radios at this point along with other changes they are making but until the 2017 radios have been endurance tested in a controlled way, I don’t trust them.
Additionally, as Ryan said, switch to using /8 subnet masks. Again, not documented, but worth doing. I’ve been told that /24 works (By people at FRC HQ no less) and has been tested but, quite frankly, I don’t believe it and I know the /8 works for us so far this year.
As Andrew said, treat the field network like a black box because that might as well be what it is.
The gateway should be left blank. Your robot is not connecting to the Internet, and that can actually cause interesting problems if your packets get routed out to the AP and back again.
The netmask should be as open as possible, but the only critical one is the Driver Station PC, which must be 255.0.0.0.
The driver station 10.TE.AM.x must be able to talk to FMS at 10.0.100.5, and 10. is the only common element, hence 255.0.0.0
There is more going on there with broadcast domains being whacky. I suspect we’ll see a simplified field network before we see docs explaining how this one works but I have some interesting theories that I’d love to have confirmed or debunked.
SDN has shown me that people can make WEIRD network configs when enabled to do it and FMS definitely isn’t using anything like that but it is still one of the more oddball setups I’ve ever encountered.
One thing to note is that that the ports on the radio not always talking to each other is a radio firmware issue. The fms network setup has nothing to do with it, and it is possible to see the issue in AP mode as well.
In terms of difference when the fms is connected vs not, there are actually only 2 difference between fms and no fms. The first is the control packet to the Rio has the fms flag set. And the second is that said control packet is sent over a different port in fms mode (this is the reason the firewall needs to be off on most machines). Other than that, there really isn’t anything too funky going on. There are some mdns issues, but statics 99.9% of the time solve those issues.
I’ll edit this post in a bit with an overview of how the DS connects to the fms. But do note that the fms actually never, and can’t, talk to the robot directly. It only ever talks to the DS.
So here’s the thing, there is more going on with the FMS than just those two changes. There is something happening with firewalls, VLANs, and bridging.
Let’s be clear, FMS =/= Field Network and the two terms are often used interchangeably (including by me frequently). The issue I have is that the field network is a black box. What happens in the switch in the big case as well as the switches at the ends of the field is a complete unknown without those configurations being published. This means serious integration testing can’t really be done. Teams should be enabled to accurately recreate this environment and they aren’t. It’s very frustrating.
This is my understanding as of 2016. Some changes were made in 2017, like a new Field AP, but the specifics of the system are still somewhat unknown.
So, based on my experience up to 2016, the AP and managed switches in the “big case” and field ends are set up each match with a VLAN for each team. Thus the need to be at the correct Driver Station.
Trying to completely replicate the Field environment at you shop or school is very difficult. It requires a bit of specific hardware and an “FMS like” system . The closest option I can think of is using Cheesy Arena and the AP and Switch that is described in CA’s documentation. BUT, with the changes to the robot radio in 2017, even that hardware and software don’t work properly any more.
So far, the closest I have been able to come to replicating a field environment is to flash a Linksys WRT1900ACS (the same router FMS is using) with OpenWRT and configuring the two radios with SSIDs and bridging them to the lan1 ports. We then configure the OM5P-AC using “FMS Lite” mode from the Team Radio Kiosk to connect to the router’s 5Ghz radio.
This does not include the manages switched, nor the VLANs. What we are able to do is control multiple robots, wirelessly connect the DS’s and programming laptops, and stream live video at 20FPS all simultaneously. We can further replicate a field environment by connecting the DS’s to the lan1 ports on the back of the router if desired.
Wow, thanks for all the tips! This is the list of recommended changes that I’ve gathered from the conversation so far; is there anything I am missing/misunderstanding?
Use a 2016 radio
Switch the static address to be between .5 to .19 to avoid the DHCP range
Set the subnet mask to 255.0.0.0. This goes against what the FTA told us, but it sounds like you guys are confident in it.
Blank the gateway. Again, this isn’t what the FTA I talked to said, and I don’t believe that 900 does this (I’ve been using you guys as a sanity check for my networking config). Having said that, I frequently see the gateway set to 10.TE.AM.1, which is also not what the FTA said, so at this point I am inclined to believe that I don’t need to change it.
I’ll be able to try this today and see how it goes. We’re going to be heading out for competition tomorrow, so let’s hope it works first try!
I wish you luck. Keep us posted on how it goes and we can try to help as best we can.
The gateway thing is an odd one. You shouldn’t need it and I’m not sure why it would cause problems setting it if the field network is configured the way everyone claims it is*. We set ours because we sometimes like to have the DS talk to the stuff on our robot through the radio.
*Not being encouraged to set the gateway when it isn’t being used should be a big 'ol clue that the network isn’t configured the way everyone claims it is.
My understanding (and as CSA the technique I used this year when helping teams address network configuration issues) is that if static IP addresses are used:
The Driver Station subnet mask must be 255.0.0.0 (so that it can reach the FMS on the 10.0.X.Y subnet without requiring a gateway)
Other IP endpoints on the robot (RoboRIO, Vision Processor CPU, IP Cameras) only require access to the robot subnet (10.TE.AM.XX) and thus can use a more restrictive subnet mask of 255.255.255.0
All that said, a 255.255.255.0 subnet mask will work on any IP endpoint on the robot, since that subnet mask is less restrictive.
