RoboRIO / FMS / mDNS / lessons learned

[x86_4819]

Originally Posted by fovea1959

I can testify that running Avahi on a rPi works just fine in the pits, on the field, and in practice.

Aside from Avahi, how did you configure your network interface? (Static IP, DHCP, Link local, etc)

[jwhite]

Originally Posted by x86_4819

Aside from Avahi, how did you configure your network interface? (Static IP, DHCP, Link local, etc)

We initially used DHCP with our Pi and had the D-link assign it to a known address. Later, we switched that to static. Avahi worked both ways.

Cheers,

Jeremy

[Mr. Lim]

Having competed at GTRC last weekend, I noticed sometimes my students had a lot of difficulty getting a tethered connection to the robot in the pits, possibly due to some of the items raised in this thread.

It's still speculation at this point, but I think it has to do with stale DHCP leases from when the robot and DS were connected to the FMS (10.xx.yy.zz) vs falling back to link-local (169.254.xxx.xxx) in the pits.

If you are connected to the FMS and play a match, the DHCP server will give all your devices a 10.xx.yy.zz address. After leaving the field, when you power cycle the robot, or disable/enable the network connection on the DS, or wait an unspecified amount of time, those devices get a new 169.254.xxx.xxx link-local IP.

If you have one device that still has the IP from the FMS's DHCP, and another with the link-local, the two won't communicate.

The solution we stumbled up is to make sure the robot is power cycled, and the DS's network connection is disabled and re-enabled after every match (to refresh the DHCP lease).

Has anyone else run into this?

[billbo911]

Originally Posted by Mr. Lim

Having competed at GTRC last weekend, I noticed sometimes my students had a lot of difficulty getting a tethered connection to the robot in the pits, possibly due to some of the items raised in this thread.

It's still speculation at this point, but I think it has to do with stale DHCP leases from when the robot and DS were connected to the FMS (10.xx.yy.zz) vs falling back to link-local (169.254.xxx.xxx) in the pits.

If you are connected to the FMS and play a match, the DHCP server will give all your devices a 10.xx.yy.zz address. After leaving the field, when you power cycle the robot, or disable/enable the network connection on the DS, or wait an unspecified amount of time, those devices get a new 169.254.xxx.xxx link-local IP.

If you have one device that still has the IP from the FMS's DHCP, and another with the link-local, the two won't communicate.

The solution we stumbled up is to make sure the robot is power cycled, and the DS's network connection is disabled and re-enabled after every match (to refresh the DHCP lease).

Has anyone else run into this?

This actually makes a lot of sense.
I worked as a FTAA at CVR last week. We had several teams that would show up to the field with static addresses set, teams that I personally had set to DHCP on their previous match.
The reason they gave was that they could not connect in the pits or practice field unless they switched to a static address.

Cycling the status of the NIC (Disabled/Enabled) would indeed address the issue.

The other issue we saw a lot was that the Wireless had been magically re-enabled. As you should be aware, this is a BIG No-No at competition and should never happen on the DS. Just disableing the Wireless would usually force the NIC to acquire a DHCP address and all was well.

[ozrien]

http://www.chiefdelphi.com/forums/sh...61&postcount=4
This might help .

[Mr. Lim]

Thanks Omar! Seems to corroborate exactly what we saw at GTRC.

I guess I would put another vote towards exploring the RoboRIO acting as a DHCP + DNS server when not connected to the FMS. That would keep all devices on 10.xx.yy.zz IPs at all times.

[GeeTwo]

Originally Posted by jwhite

Yeah, okay, calling it obvious is perhaps a stretch. But if you presume the FMS is running on a 10.0.0.X ip, and you have no default gateway, then a netmask 255.0.0.0 makes sense. (And yes, I agree, habit dictates 255.255.255.0).

No, if the first three octets are fixed, 255.255.255.0 makes sense. I'm not saying it's correct.

[Mr. Lim]

Originally Posted by GeeTwo

No, if the first three octets are fixed, 255.255.255.0 makes sense. I'm not saying it's correct.

Actually there is a convention for what subnet SHOULD be used based on the first number in an IP address:

http://en.wikipedia.org/wiki/IPv4_subnetting_reference

A network that starts with 10 has a leading 0 when converted to 8-bit binary, and thus should be a 255.0.0.0 subnet. You'll see that's what the subnet field defaults to when you type in a 10.xx.yy.zz IP.

Conversely, one that starts with 192 leads with 110 when converted to 8-bit binary, and thus should be a 255.255.255.0 subnet.

We're breaking this convention if we use anything other than a 255.0.0.0 actually, but it's just a convention and not really a rule. There are some good reasons to break convention in our case, like preventing teams from interfering with each other.

[Travis Hoffman]

Originally Posted by Mr. Lim

Having competed at GTRC last weekend, I noticed sometimes my students had a lot of difficulty getting a tethered connection to the robot in the pits, possibly due to some of the items raised in this thread.

It's still speculation at this point, but I think it has to do with stale DHCP leases from when the robot and DS were connected to the FMS (10.xx.yy.zz) vs falling back to link-local (169.254.xxx.xxx) in the pits.

If you are connected to the FMS and play a match, the DHCP server will give all your devices a 10.xx.yy.zz address. After leaving the field, when you power cycle the robot, or disable/enable the network connection on the DS, or wait an unspecified amount of time, those devices get a new 169.254.xxx.xxx link-local IP.

If you have one device that still has the IP from the FMS's DHCP, and another with the link-local, the two won't communicate.

The solution we stumbled up is to make sure the robot is power cycled, and the DS's network connection is disabled and re-enabled after every match (to refresh the DHCP lease).

Has anyone else run into this?

*Raises hand*

Nothing new to add. Our experience was as you described.

[FrankJ]

Not sure what the 2015 FMS DHCP does, but in years past with static IPs... Every thing except the crio had 255.0.0.0 subnet masks (the older robot routers didn't, that is a special case & they didn't have to talk to anybody.) That means FMS, driver stations could see and talk to each other. crios had a subnet mask of 255.255.255.0 puts them in a different sub net class & they should not talk to anybody in a different subnet. But one quirkieness of IP subnets 10.te.am.xx subnet mask 255.0.0.0 looks be the same subnet as 10.te.am.xx subnet mask 255.255.255.0. So the CRIO will see and talk to the team driver station, camera, anything 10.te.am.xx and nothing else. Which is generally why you or not supposed to mix subnet classes on the same network.

Back in the pit, if you somehow have not cycled power to the radio, the roborio will hold its IP. Or if they are slow reconfiguring the field radio & it reconnects to the robot. It also takes a while for your computer to drop to the default IP address because it is hoping that a DHCP will assign one to it. You could really confuse things and run a DHCP server on your drive station.

[rjmah]

Originally Posted by Mr. Lim

Thanks Omar! Seems to corroborate exactly what we saw at GTRC.

I guess I would put another vote towards exploring the RoboRIO acting as a DHCP + DNS server when not connected to the FMS. That would keep all devices on 10.xx.yy.zz IPs at all times.

I'll add my vote to this. We experienced similar problems at Toronto East this weekend. It seemed the Classmate would always connect but our HP laptop would only connect maybe once in 4 times after we had been on the field. It had all our java programming in it and I hadn't read this thread before now.

[Alan Anderson]

Originally Posted by Mr. Lim

I guess I would put another vote towards exploring the RoboRIO acting as a DHCP + DNS server when not connected to the FMS.

The problem is that the roboRIO itself doesn't connect to the FMS, and it can't know whether or not to be a DHCP server until after it is already talking to the Driver Station.

I can't think of a reliable way to have a DHCP server decide to be active only when the FMS is not present. There are too many chances for it to be running before the DS has made the FMS connection. My preference would be for a separate physical device in the Ethernet tether between DS and robot bridge.

[fovea1959]

Originally Posted by x86_4819

Aside from Avahi, how did you configure your network interface? (Static IP, DHCP, Link local, etc)

We set it up DHCP. One of the Avahi daemons assigns a link local if (and only if) it sees the DHCP client does not assign an address; I don't remember the details, but could probably dig them up (as can you).

[Mr. Lim]

Originally Posted by Alan Anderson

The problem is that the roboRIO itself doesn't connect to the FMS, and it can't know whether or not to be a DHCP server until after it is already talking to the Driver Station.

I can't think of a reliable way to have a DHCP server decide to be active only when the FMS is not present. There are too many chances for it to be running before the DS has made the FMS connection. My preference would be for a separate physical device in the Ethernet tether between DS and robot bridge.

But the RoboRIO does get its IP address from the FMS's DHCP server. I think it may be possible (although I have personally never done it) to configure a DHCP server on the RoboRIO to only hand out leases if there isn't already a DHCP server on the same network. If it detects another DHCP server (i.e. the FMS's) on the network it simply won't do anything, and let the existing (FMS's) DHCP handle everything.

Redundant DHCP servers that are configured for failover have been around for a long time. Maybe this is an approach worth exploring:

https://kb.isc.org/article/AA-00502/...-Failover.html

Also, if the RoboRIO's DHCP server hands out a bunch of leases BEFORE connecting to the FMS, would this necessarily be a bad thing? I'm not sure there is a downside. All the devices would get proper 10.te.am.zz IPs, and assuming mDNS is still configured, everything should be addressable by hostname, even if the FMS doesn't know everyone's IP address via DHCP.

[plnyyanks]

I thought I'd write up some of the things I saw as FTAA at NYC this weekend.

Major Things:
- UPDATE YOUR DRIVER STATIONS/FIRMWARE. We still spent way too much time running around updating teams' laptops. Save everybody the time and update it before you compete (and you'll pass inspection that much faster)
- PUSH IN THE PDP FUSES. Offhand, I think at least quarter of the field connectivity issues we saw involved robots with the fuses not pushed in all the way, which increases the likelihood of power issues elsewhere on the robot (brownouts, reboots, and whatnot). Look at them sideways, if you can see the contacts between the top of the fuse, and the PDP, it's not in all the way. Push harder.
- CHECK YOUR ETHERNET CABLES. I was surprised at the number of times the ethernet connection to the roboRIO was not plugged in all the way, not plugged in at all, or nonexistent. If you unplug the ethernet cable in the pits, make sure it gets plugged back in (and verify the link lights on the port when you turn the robot on)
- DON'T TOUCH THE BRIDGE/AP MODE SWITCH. Once you configure your bridge, there is absolutely no reason to move the switch on the bridge back to AP mode until after the event. Wifi isn't allowed in the pits, and it wont' work on the field, so it's not particularly useful to touch...
- BUY A BATTERY BEAK. Power brownouts are no fun, and pretty common. Load test your batteries so they don't happen.
- HAVE YOUR STATUS LIGHTS VISIBLE. Both bridge and roboRIO. If I can't see the lights from 20 feet away, I can't help you debug when your robot drops.


General Field Connectivity Notes:
- Connectivity seemed pretty good this year, once a robot linked up, they very rarely dropped (and the vast majority of those were caused by something above). So if you don't want to drop, do all the above things and you should be fine.
- Driver stations were sometimes slow to connect (probably related to some of the mDNS issues above). It works the best when the laptop is plugged in after the field has completed prestart - you know this has happened when the stack lights on the scoring table turn on blue and red to signify the field is not ready.
- If you have plugged your driver station in, and it's not showing a connection, the fastest fixes were to either close and reopen the DS program, or to change your team number to a different one and back to your (both of these worked equally well for us). Then it should link right up. If not, run through the list below.
- An alternate would be to not open the driver station program until the field has been prestarted and the stack light is lit red/blue

Common DS Issues:
- Version. Update it.
- Wifi on. It shouldn't be. Turn it off.
- Firewall; just disable it.
- If no robot connection, close/reopen or change team number briefly
- If still no robot connection, restart NI mDNS Responder Service

Overall, things ran pretty smoothly - just keep an eye on the things above, and nobody should have any problems.