RoboRIO / FMS / mDNS / lessons learned
 

Hi folks,

We're having a good time in Duluth. We learned a few things about the Roborio and the new mDNS stuff the hard way. In hindsight, they're fairly obvious, but I thought I'd share our experience in the hope it saves someone else some of our puzzlement. (Nicely, the good FIRST people helped us figure it out in the practice rounds, so no Q round harmed).

We learned two things the hard way:

1. Once you get to the regional, and have the D-link configured, you are in bridge mode, and there is no DHCP server.

2. If you use static IPs, you *must* use a netmask of 255.0.0.0 or you won't work on the field.

Some subtleties of #1:
a. The mDNS stuff will keep working. But the Robo Rio and your windows laptop will use a 169.XXX address, and other devices (like our Axis camera and our Pi) will use the last leased address, in our case 10.28.23.X. So you won't be able to communicate in the Pit. Ironically, there is a dhcp server on the field, so you *will* be able to communicate on the field.
b. If you had the D-link assign a particular IP to a device, you lose that once the router is reconfigured. For example, we had the d-link give the axis a set address. The camera was still in DHCP mode, but it always got assigned .11. That meant we could keep using the basic smart dashboard widget, which only accepts a numeric IP address. Of course, that falls apart when you no longer have a DHCP server...

So the solution to #1 was to switch over to using static IPs. *It turns out that this is really easy*. Hit roborio-XXXX.local with a web browser, click on the network configuration, one easy choice, and you are done. (This presumes you're still good at doing static ips for your other devices).

The *catch* is that the default netmask the RoboRio will suggest is 255.255.0.0. That will work great in the pits, but the FMS requires a netmask of 255.0.0.0. (If you think about it, that's obvious :-/).

At any rate, I hope this helps someone else, and good luck to you all.

And a shout out to the Lake Superior staff for doing a great job of running a smooth week 1.

Cheers,

Jeremy

Re: RoboRIO / FMS / mDNS / lessons learned
 

While what you describe can work, the simpler configuration that teams are expected to use when at an event in the pits is to set all devices to DHCP + Link Local (the default for most devices) and use mDNS to find them even when they get a 169.254 address. If there is a tool that was released that doesn't support mDNS, then you should report that as a bug on the collab.net site.

I'm not sure why that would be obvious. It's not like you are allowed / able / expected to send packets among teams. 255.255.255.0 is what I would select for the "obvious" netmask.

Re: RoboRIO / FMS / mDNS / lessons learned
 

It's the Smart Dashboard IP camera widget. We were told it was a known problem that it only
accepts numeric ips. I didn't find a bug report for it, so I've entered it here:
https://usfirst.collab.net/sf/go/artf4028
(and boy that is not especially easy or obvious; you have to 'Plan' to add an artifact...)

Yeah, okay, calling it obvious is perhaps a stretch. But if you presume the FMS is running on a 10.0.0.X ip, and you have no default gateway, then a netmask 255.0.0.0 makes sense. (And yes, I agree, habit dictates 255.255.255.0).

Cheers,

Jeremy

Re: RoboRIO / FMS / mDNS / lessons learned
 

This only applies if your team isn't using macs, which we've found cannot use the mDNS to find the roboRIO reliably.

Thanks for posting this! We'll definitely be setting up a static IP configuration at the comp, as we're using macs.

Re: RoboRIO / FMS / mDNS / lessons learned
 

Very interesting subtleties. I can see those being very annoying.

During Alpha testing I was very postivie of moving to DHCP and taking away the need to set static IPs. This was especially useful for programming laptops that were switching constantly between robot networks (static) and internet connections (DHCP).

I still am a big fan of DHCP, but recognize there are instances like this where a lack of mDNS support by some peripherals (IP-based cameras, embedded computers for co-processing, etc) could be an issue.

The pros outweigh the cons for sure, however.

I'd have to agree with jhersh in saying that the best situation is to get all devices on dynamic IPs via DHCP + mDNS, then address them only by their hostnames.

Of course not everything supports mDNS.

The RPi can probably do it with some additional software:
http://www.howtogeek.com/167190/how-...-raspberry-pi/

The newer Axis Cameras (M1011 onwards) should also support Apple Bonjour, which actually should provide mDNS services.

I haven't tried either so I'm not speaking from experience, but they are both worth a shot trying to configure.

Alternatively, I think a cool potential solution would be to have the RoboRIO detect whether a DHCP server is present on the network, and if not, act as a DHCP+DNS server. It would then need to shut down and hand off the leases to the FMS's DHCP server when connected to the field, which would probably be the hard part. Multiple DHCP servers on the same network aren't ideal.

Re: RoboRIO / FMS / mDNS / lessons learned
 

Not sure what you mean here. Care to elaborate?

If I recall, the FMS runs at 10.100.0.x addresses. The robot never needs to send or receive packets to or from the FMS. The DS takes care of all of that communication. I don't recall what the router configuration on the field is (if it hands out a default gateway that is needed or if it broadens the subnet mask). Based on your original post I would guess it's the latter, but I can't say for sure. The DS should always be configured for DHCP so it will always do what the field requests vis-a-vis network configuration. In your proposed configuration you have to react manually to any field configuration change.

It is a work-around to an actual problem that teams obviously are dealing with. Thanks for sharing!

Re: RoboRIO / FMS / mDNS / lessons learned
 

We considered this from very early on, but rejected it. I don't recall the reason, but we may revisit it.

Re: RoboRIO / FMS / mDNS / lessons learned
 

I remember a bug report about this during beta. The team at NI that is responsible for that part of the image wasn't able to respond with a resolution due to how late it was reported. It's on our radar for next year.

Also, are you aware of this? : https://discussions.apple.com/thread/6607871

Re: RoboRIO / FMS / mDNS / lessons learned
 

Just to be sure everyone who is looking at this thread knows about the documentation for the network configuration in FRC 2015, I want to provide the link: https://wpilib.screenstepslive.com/s...rio-networking

Re: RoboRIO / FMS / mDNS / lessons learned
 

I hadn't seen that, but, I'm pretty sure that isn't the problem. The issue only appears when using the DLink router. I had sent Greg Mckaskle some of my debugging notes, but I'll list them here too:

• When the roborio is connected to my home network's wireless, it resolves just fine
• When running tcpdump on the robot/laptop when connected to the dlink, mDNS packets go out to the robot, and the robot shows that it sends responses. However, the responses don't show up at the laptop
• Interestingly enough, when I try to resolve the router's mdns name (dlinkap.local), the roborio sees the response packets, but the laptop does not
• When I'm connected via ethernet to the DLink, mDNS works instantly
• I've tried changing various settings on the router with no lasting success. Every once in awhile something I change fixes it, but then after a reboot or something it will stop working

A prior thread about this is here: http://www.chiefdelphi.com/forums/sh...d.php?t=132607

Thinking about it, maybe it'll just work for us since we'll always be tethered at the comp, and mDNS worked just fine for me when I was connected via ethernet.

Re: RoboRIO / FMS / mDNS / lessons learned
 

Aha! I want to highlight this portion of your remark, because I didn't fully understand it while at the regional. The DS and the Robo Rio both apparently are set such that if there is no DHCP server, they fall back to 'link local' (also apparently aka stateless address autoconfiguration). I'm not particularly familiar with that mode.

But if you can persuade all your other devices to do the same, and all your software can use names, you should indeed be in great shape. (A lazy 5 minute Google and read of the M1011 manual suggests possibilities, but not clear paths to doing that. The M1011 appears to let you set *two* addresses; not sure which gets the 'axis-camera.local' name. The Pi can use avahi to set a link local address. It's not clear to me if there is an easy or clean way to have the dhcp server fall back. I suspect another 20 minutes of Googling might help...)

Yeah, I agree. I think this is a good change, and hopefully once the bumps are ironed out, the teams will reap the benefits into the future.

Just to confirm - they both do, and it's nice. It's particularly nice because the mDNS continues to work in both static and DHCP modes.

Cheers,

Jeremy

Re: RoboRIO / FMS / mDNS / lessons learned
 

Sure.

Disclaimer, this is all meant in a constructive way, and I know that you have little control over collab.net. Just trying to cheerfully and positively illustrate my experience.


Nit #1 - this page:
http://wpilib.screenstepslive.com/s/...-documentation
At the bottom, there is a list of what to do with an unresolved problem. That list, or elements of it, could be drawn out on the 'Troubleshooting' main page; or be somewhat more visible. I only found it because I knew you wouldn't have asked me to file a bug if I there wasn't a link, so I just kept digging until I found it. And then, really minor nit, I think the list of three action items should be bullets; my eye scans better that way. (I found the link with a ctrl-F on collab, *not* by seeing it). Super dooper teeny nit: the url could be an href...

Nit #2: this page:
https://usfirst.collab.net/sf/projects/wpilib/
Doesn't really have good guidance for someone new to all this. The welcome tells me how to clone the code, and points me back to the same documentation I just came from it I want more help. The 'getting around' and 'how to participate' sections are empty. Be nice to have more newb friendly advice. (The opening could be, if you are a developer, then...to report a bug, then ....)

Okay, now I've been a good boy, and remembered that I'm supposed to do something with Trackers. So I click on 'Trackers' (not bugs; nowhere on this page does it say what to do if I want to report a bug).

As as exercise, click on that with an icognito window to get the newb experience. Nowhere on that page is there an obvious way to enter a bug. You're in list mode. Now let's say you get lucky, and you click 'Plan' mode. Now you're invited to pick a planning folder. Or maybe 3. Who knows? You just want to enter a bug...

But let's say you try it, and you pick Java. Gee, that '+' button is greyed out, with no tooltip.

So then you make an account, repeat the exercise, and then finally you enter, not a bug, mind you, but an 'Artifact'. And then you have apparently done it.

And then you go back and write a disclaimer because you're not really grumpy :-).

Cheers,

Jeremy

Re: RoboRIO / FMS / mDNS / lessons learned
 

Yeah, seriously, they should use github for hosting wpilib. We'd all be happier. Collab is just terrible in so many ways -- this is why new projects rarely choose sourceforge.

Re: RoboRIO / FMS / mDNS / lessons learned
 

I can testify that running Avahi on a rPi works just fine in the pits, on the field, and in practice.

The axis camera would still be a problem. We finally abandoned ours and went to a Lifecam on the roboRIO....

Re: RoboRIO / FMS / mDNS / lessons learned
 

How do we get the default dashboard to use the mDNS in place of the IP addresses. After a match we could sometimes tether and after a few minutes would loose the tether and seemed to have a very dificult time to re establish communications.

Re: RoboRIO / FMS / mDNS / lessons learned
 

Aside from Avahi, how did you configure your network interface? (Static IP, DHCP, Link local, etc)

Re: RoboRIO / FMS / mDNS / lessons learned
 

We initially used DHCP with our Pi and had the D-link assign it to a known address. Later, we switched that to static. Avahi worked both ways.

Cheers,

Jeremy

Re: RoboRIO / FMS / mDNS / lessons learned
 

Having competed at GTRC last weekend, I noticed sometimes my students had a lot of difficulty getting a tethered connection to the robot in the pits, possibly due to some of the items raised in this thread.

It's still speculation at this point, but I think it has to do with stale DHCP leases from when the robot and DS were connected to the FMS (10.xx.yy.zz) vs falling back to link-local (169.254.xxx.xxx) in the pits.

If you are connected to the FMS and play a match, the DHCP server will give all your devices a 10.xx.yy.zz address. After leaving the field, when you power cycle the robot, or disable/enable the network connection on the DS, or wait an unspecified amount of time, those devices get a new 169.254.xxx.xxx link-local IP.

If you have one device that still has the IP from the FMS's DHCP, and another with the link-local, the two won't communicate.

The solution we stumbled up is to make sure the robot is power cycled, and the DS's network connection is disabled and re-enabled after every match (to refresh the DHCP lease).

Has anyone else run into this?

Re: RoboRIO / FMS / mDNS / lessons learned
 

This actually makes a lot of sense.
I worked as a FTAA at CVR last week. We had several teams that would show up to the field with static addresses set, teams that I personally had set to DHCP on their previous match.
The reason they gave was that they could not connect in the pits or practice field unless they switched to a static address.

Cycling the status of the NIC (Disabled/Enabled) would indeed address the issue.

The other issue we saw a lot was that the Wireless had been magically re-enabled. As you should be aware, this is a BIG No-No at competition and should never happen on the DS. Just disableing the Wireless would usually force the NIC to acquire a DHCP address and all was well.

Re: RoboRIO / FMS / mDNS / lessons learned
 

http://www.chiefdelphi.com/forums/sh...61&postcount=4
This might help .

Re: RoboRIO / FMS / mDNS / lessons learned
 

Thanks Omar! Seems to corroborate exactly what we saw at GTRC.

I guess I would put another vote towards exploring the RoboRIO acting as a DHCP + DNS server when not connected to the FMS. That would keep all devices on 10.xx.yy.zz IPs at all times.

Re: RoboRIO / FMS / mDNS / lessons learned
 

No, if the first three octets are fixed, 255.255.255.0 makes sense. I'm not saying it's correct.

Re: RoboRIO / FMS / mDNS / lessons learned
 

Actually there is a convention for what subnet SHOULD be used based on the first number in an IP address:

http://en.wikipedia.org/wiki/IPv4_subnetting_reference

A network that starts with 10 has a leading 0 when converted to 8-bit binary, and thus should be a 255.0.0.0 subnet. You'll see that's what the subnet field defaults to when you type in a 10.xx.yy.zz IP.

Conversely, one that starts with 192 leads with 110 when converted to 8-bit binary, and thus should be a 255.255.255.0 subnet.

We're breaking this convention if we use anything other than a 255.0.0.0 actually, but it's just a convention and not really a rule. There are some good reasons to break convention in our case, like preventing teams from interfering with each other.

Re: RoboRIO / FMS / mDNS / lessons learned
 

*Raises hand*

Nothing new to add. Our experience was as you described.

Re: RoboRIO / FMS / mDNS / lessons learned
 

Not sure what the 2015 FMS DHCP does, but in years past with static IPs... Every thing except the crio had 255.0.0.0 subnet masks (the older robot routers didn't, that is a special case & they didn't have to talk to anybody.) That means FMS, driver stations could see and talk to each other. crios had a subnet mask of 255.255.255.0 puts them in a different sub net class & they should not talk to anybody in a different subnet. But one quirkieness of IP subnets 10.te.am.xx subnet mask 255.0.0.0 looks be the same subnet as 10.te.am.xx subnet mask 255.255.255.0. So the CRIO will see and talk to the team driver station, camera, anything 10.te.am.xx and nothing else. Which is generally why you or not supposed to mix subnet classes on the same network.

Back in the pit, if you somehow have not cycled power to the radio, the roborio will hold its IP. Or if they are slow reconfiguring the field radio & it reconnects to the robot. It also takes a while for your computer to drop to the default IP address because it is hoping that a DHCP will assign one to it. You could really confuse things and run a DHCP server on your drive station.

Re: RoboRIO / FMS / mDNS / lessons learned
 

I'll add my vote to this. We experienced similar problems at Toronto East this weekend. It seemed the Classmate would always connect but our HP laptop would only connect maybe once in 4 times after we had been on the field. It had all our java programming in it and I hadn't read this thread before now.

Re: RoboRIO / FMS / mDNS / lessons learned
 

The problem is that the roboRIO itself doesn't connect to the FMS, and it can't know whether or not to be a DHCP server until after it is already talking to the Driver Station.

I can't think of a reliable way to have a DHCP server decide to be active only when the FMS is not present. There are too many chances for it to be running before the DS has made the FMS connection. My preference would be for a separate physical device in the Ethernet tether between DS and robot bridge.

Re: RoboRIO / FMS / mDNS / lessons learned
 

We set it up DHCP. One of the Avahi daemons assigns a link local if (and only if) it sees the DHCP client does not assign an address; I don't remember the details, but could probably dig them up (as can you).

Re: RoboRIO / FMS / mDNS / lessons learned
 

But the RoboRIO does get its IP address from the FMS's DHCP server. I think it may be possible (although I have personally never done it) to configure a DHCP server on the RoboRIO to only hand out leases if there isn't already a DHCP server on the same network. If it detects another DHCP server (i.e. the FMS's) on the network it simply won't do anything, and let the existing (FMS's) DHCP handle everything.

Redundant DHCP servers that are configured for failover have been around for a long time. Maybe this is an approach worth exploring:

https://kb.isc.org/article/AA-00502/...-Failover.html

Also, if the RoboRIO's DHCP server hands out a bunch of leases BEFORE connecting to the FMS, would this necessarily be a bad thing? I'm not sure there is a downside. All the devices would get proper 10.te.am.zz IPs, and assuming mDNS is still configured, everything should be addressable by hostname, even if the FMS doesn't know everyone's IP address via DHCP.

Re: RoboRIO / FMS / mDNS / lessons learned
 

I thought I'd write up some of the things I saw as FTAA at NYC this weekend.

Major Things:
- UPDATE YOUR DRIVER STATIONS/FIRMWARE. We still spent way too much time running around updating teams' laptops. Save everybody the time and update it before you compete (and you'll pass inspection that much faster)
- PUSH IN THE PDP FUSES. Offhand, I think at least quarter of the field connectivity issues we saw involved robots with the fuses not pushed in all the way, which increases the likelihood of power issues elsewhere on the robot (brownouts, reboots, and whatnot). Look at them sideways, if you can see the contacts between the top of the fuse, and the PDP, it's not in all the way. Push harder.
- CHECK YOUR ETHERNET CABLES. I was surprised at the number of times the ethernet connection to the roboRIO was not plugged in all the way, not plugged in at all, or nonexistent. If you unplug the ethernet cable in the pits, make sure it gets plugged back in (and verify the link lights on the port when you turn the robot on)
- DON'T TOUCH THE BRIDGE/AP MODE SWITCH. Once you configure your bridge, there is absolutely no reason to move the switch on the bridge back to AP mode until after the event. Wifi isn't allowed in the pits, and it wont' work on the field, so it's not particularly useful to touch...
- BUY A BATTERY BEAK. Power brownouts are no fun, and pretty common. Load test your batteries so they don't happen.
- HAVE YOUR STATUS LIGHTS VISIBLE. Both bridge and roboRIO. If I can't see the lights from 20 feet away, I can't help you debug when your robot drops.


General Field Connectivity Notes:
- Connectivity seemed pretty good this year, once a robot linked up, they very rarely dropped (and the vast majority of those were caused by something above). So if you don't want to drop, do all the above things and you should be fine.
- Driver stations were sometimes slow to connect (probably related to some of the mDNS issues above). It works the best when the laptop is plugged in after the field has completed prestart - you know this has happened when the stack lights on the scoring table turn on blue and red to signify the field is not ready.
- If you have plugged your driver station in, and it's not showing a connection, the fastest fixes were to either close and reopen the DS program, or to change your team number to a different one and back to your (both of these worked equally well for us). Then it should link right up. If not, run through the list below.
- An alternate would be to not open the driver station program until the field has been prestarted and the stack light is lit red/blue

Common DS Issues:
- Version. Update it.
- Wifi on. It shouldn't be. Turn it off.
- Firewall; just disable it.
- If no robot connection, close/reopen or change team number briefly
- If still no robot connection, restart NI mDNS Responder Service

Overall, things ran pretty smoothly - just keep an eye on the things above, and nobody should have any problems.

Re: RoboRIO / FMS / mDNS / lessons learned
 

I suspect this might be a bad thing, everyone would have 10.te.am.zz IPs, but would they be *unique*? What if the FMS assigns an address that the RoboRIO previously assigned, but to a different machine?

Would it be correct to say that anything that (correctly) does DHCP, and falls back to link local addresses, and uses mDNS for name lookups works fine? If so, the setup right now works, we just need to take care of the "not correctly" cases (i.e. reusing DHCP-assigned addresses across disconnects/reconnects). Or am I over-simplifying?

Re: RoboRIO / FMS / mDNS / lessons learned
 

I expect the FMS DHCP is in the FMS's radio (access point might be a better name.) Since you are on a VPN at that point, each team would have their own DHCP. The RoboRIO cannot connect to the DS until it has an IP in the DS's subnet so it has to get that from the DHCP first. It would be interesting to know if the RoboRIOs subnet mask was still 255.255.255.0 at this point.

A couple of options for DCHP in the pits would be one on your DS. Probably not a good idea since you would have to remember to turn it on & off. Another would be a wired router. You would have to remember to plug it into the the robot radio before turning it on. Or you could go back to static IPs as outlined in screen steps

Other than it taking annoyingly long, we did not have issues connecting in the pit using to default method.

Re: RoboRIO / FMS / mDNS / lessons learned
 

Here's an example of the PDP fuses working their way out by the time eliminations rolled around.
They were never pushed in all the way.

Properly inserted there should only be about 1/4" showing, and are not removable by hand any longer.
If you can easily remove the fuses by hand, then they are not seated all the way.

Attached are three variations:
- fuses on a robot in Semi-Finals that have worked their way out
- fuses properly seated
- one good/one badly seated fuse to compare

Re: RoboRIO / FMS / mDNS / lessons learned
 

We ran into this problem at LoneStar on a regular basis (i.e. failure to reconnect to robot after returning to the pit, Driver Station showing a 169.254.X.X address). The key here (as already noted) is the 169.254.X.X address. Ever since Windows Vista (/ward against evil eye), the Windows IP stack generates the 169.254.X.X address if it cannot find a DNS server. This default address is also sticky and hard to flush from the standard GUI interfaces.

The problem lies with the driver station running Windows 7+. The roboRIO seems to be doing what its supposed to do, its just not being sent a DNS query by the driver station.

The fix - reset the ethernet adapter IP system.

The solution: Generate a batch file in Notepad that invokes the windows command line net shell interface. Use it to reset the IP system.

One line, should read:

Or you could write it out fully:

Save the file as with a .bat extension somewhere handy with a memorable title (e.g. reset_ip.bat). Execute it by double clicking on it or any other conventional method.

Our experience is that this fixes the problem in seconds.

Hope this helps. Good luck in this last week of District competitions and at Championship if you are fortunate enough to go.