FMS enabling sequence?

[ATannahill]

I suggest you view the FMS White Paper to better understand how the field works.

[Tom Bottiglieri]

Originally Posted by rtfgnow

I suggest you view the FMS White Paper to better understand how the field works.

This paper does not address how the FMS software goes about enabling robots, only the order in which it traverses match state.

We have not measured exactly delay, but have anecdotal evidence of seeing some robots move before others. Field timing issues were pretty prevalent last year and it's unclear to me any of it has been fixed.

Unfortunately, the delay from the FMS may decide Einstein.

[Jared]

Originally Posted by MasterMentor

On the contrary, I think I figured it out directly from the FMS Whitepaper.

From the FAQ, "Does the FMS Control the Robot", FMS sends the commands to the DS, which then forwards the command to your robot to start.

Does the command get sent to all 6 DS laptops at the same time? The server in the scorpian case has one ethernet port for all red and one ethernet port for all blue robots. Does the software send red and blue simultaneously, or is there a single network adapter + a switch for both red and blue, meaning one must be sent before another. How much time is between these two commands? Within the separate red/blue disable commands the 1, 2, and 3 station enable commands must happen serially. Are they sent immediately after each other, and if not, what is the delay.

It might seem a silly question to ask if there is a significant delay between sending red and blue side commands, but hey, that's exactly what happened last year with the hot goal lights, which were +/- 4 seconds in terms of accuracy at times.

These ethernet cables run to control cases (SCC's) under the DS shelf, where they plug into an ethernet switch. This ethernet switch is connected to an Allen Bradley Stratix 8000, which has three other ethernet ports that are dedicated to 1, 2, and 3 driver stations. What goes on here is unknown, but it's not unreasonable to expect that there could be delays here. The Allen Bradley box is doing something other than being a switch because it won't work if you plug the wrong team into the wrong ethernet cable.

While it's not explicitly stated, most command systems have a processing loop that will "process" incoming commands at a known rate (timed so you don't chew up your CPU). Now, even if we knew the loop rate we don't know where in the loop the command will come in; will the command arrive immediately at the beginning of that loop, in the middle, or at the end?

It depends on how the polling is set up. If they check for 'FMS says enable' every time they send the command (20 ms), then timing can't be accurate to more than 20 ms, as you pointed out.

The FMS software and DS are written in C# (or c++, I forget), and LabVIEW, both of which easily support event driven behavior, meaning the driver station may respond to an FMS command as soon as the network buffer reads in the message.

That's probably the difference - I'd bet the difference in time it takes FMS to issue the command across all the VLANs to the DS's would be in the nanosecond (microsecond for sure) range.

-MM

You'd think, but the difference in time it took the FMS to send commands to the phillips lighting controller last year was sometimes greater than 2 or 3 seconds. FMS timing seems to be properly understood by nobody.

[MikeE]

Originally Posted by Tom Bottiglieri

Field timing issues were pretty prevalent last year and it's unclear to me any of it has been fixed.

Originally Posted by Jared

It might seem a silly question to ask if there is a significant delay between sending red and blue side commands, but hey, that's exactly what happened last year with the hot goal lights, which were +/- 4 seconds in terms of accuracy at times.

Originally Posted by Jared

You'd think, but the difference in time it took the FMS to send commands to the phillips lighting controller last year was sometimes greater than 2 or 3 seconds. FMS timing seems to be properly understood by nobody.

I think it's fairly widely known that the FMS has been substantially (completely?) rewritten since last season, so anecdotes from 2014 are of limited applicability to 2015.

One of the questions I was taught to ask early on in Grad school is what scale of effect actually matters in a given experiment, and to focus only on those factors that have meaningful impact.
If there are no systematic biases towards particular team stations, how much does a worst case 20ms delay due to polling actually affect a mechanism compared to all the other variables in play?

[Tom Bottiglieri]

Originally Posted by MikeE

If there are no systematic biases towards particular team stations

That is the part I am interested in learning more about.

[plnyyanks]

On a different note, I would love an update to the FMS Whitepaper that goes into much greater technical detail. Or maybe a 1 day recorded conference, where someone on FRC engineering can do a very detailed, technical walkthrough of how the field works (and maybe show some code snippets), including a Q&A.

I don't like how the field electronics are a 'black box' to most teams - I gained most of my knowledge about how the field works from volunteering and working with it regularly, something teams don't do.

A greater understanding about how the competition field works will reduce the amount of anger and frustration often (mis)directed at FMS.

[Citrus Dad]

Originally Posted by MikeE

If there are no systematic biases towards particular team stations, how much does a worst case 20ms delay due to polling actually affect a mechanism compared to all the other variables in play?

If the serial enabling differences are cumulative, the first bot will move 0.1 second before the last one. That's more than the difference in speed between several of the can grabbers at the moment.

[Thad House]

Originally Posted by Citrus Dad

If the serial enabling differences are cumulative, the first bot will move 0.1 second before the last one. That's more than the difference in speed between several of the can grabbers at the moment.

It would never be that high. Even serially, I bet it still sends all the enable packets within 1ms of each other. The 20ms polling loops happen on the DS, and the FMS does not depend on it. So the FMS wouldnt accumulate based on those. It would only matter to the DS at which point in the 20ms period it recieves the enable packet, and thats if its not interrupt based. So worst case is that 1 robot gets an enabled packet 19ms after another one.

Now its entirely possible something else in code is delaying more, but I cant see how the FMS would take 60+ms to send enable packets. If so, we would visually see it.

[FrankJ]

Originally Posted by Jared

...
These ethernet cables run to control cases (SCC's) under the DS shelf, where they plug into an ethernet switch. This ethernet switch is connected to an Allen Bradley Stratix 8000, which has three other ethernet ports that are dedicated to 1, 2, and 3 driver stations. What goes on here is unknown, but it's not unreasonable to expect that there could be delays here. The Allen Bradley box is doing something other than being a switch because it won't work if you plug the wrong team into the wrong ethernet cable.
...
FMS timing seems to be properly understood by nobody.

The switches are managed. Which means they only route packets to specific ports based on the IP address (or MAC). Managed switches are always higher end than generic switches. You are not losing any significant time in the switches. The travel time for Ethernet packets between the DS & robot are logged on the DS. I expect the travel times between FMS & the DS are the same or better.

I expect the FMS timing is understood by the people involved in writing the programs. It just not public information.

[themccannman]

So, since this doesn't seem resolved is there any way we can figure this out aside from FIRST publicly posting the FMS code? They only other way I can think is setting up two robots with known mechanism speeds and comparing them at each possible driver station combination. This would be pretty difficult to do though as you would need to reserve an entire field for only 2 teams for at least 30-60 mins, likely not possible with time constraints and demand for practice matches at most competitions.

[MrRoboSteve]

I have no knowledge of the internals of the FMS software, so these observations are based completely on the whitepaper and other public sources.

1. The minimum interpacket gap for 100mbps Ethernet is 0.96 µs.

2. The FMS code could take longer than 0.96 µs to send the "auto enable" command to each DS, depending on what the logic looks like in the FMS code. For example, there could be logging being written to disk between each initiate command.

3. The protocol between the FMS and the DS isn't documented in the whitepaper.

4. The FMS whitepaper says that robot <--> DS control packets are prioritized over video packets. I read that as saying that UDP 1130 and 1140 are prioritized over the other ports available for robot <--> DS communication. The whitepaper is silent about the relative priority of FMS <--> DS packets. A reasonable guess is that FMS <--> DS packets are at the same as robot <--> DS control packets. If this guess is true, it's unlikely that robot -> DS data on the open ports (e.g. camera packets) from the robot will interfere with the "auto enable" command from the FMS.

5. It is likely that one of these two statements is true:
a. The "auto enable" command is sent to each DS in the same order in every match (e.g., Red 1/2/3, Blue 1/2/3)
b. The "auto enable" command is sent to each DS in a purposely randomized order in every match

Why are these likely? Because a) is the easiest to code, and b) would be the logical alternative if they made a specific design decision here.

6. The FMS developers are likely to be curious about a thread on CD entitled "FMS enabling sequence."

[MikeE]

Caveat: I have had training on use of the FMS but do not know details of it's implementation. I'm engaging in this discussion because it is a interesting topic and educated speculation is fun!

Originally Posted by Citrus Dad

If the serial enabling differences are cumulative, the first bot will move 0.1 second before the last one. That's more than the difference in speed between several of the can grabbers at the moment.

That's a huge assumption. Not only would that require serial enabling, but it assumes only one packet is being sent anywhere per 20ms time slot. As other posters have stated it's much more likely that packets are being sent to all DS within the same 20ms frame.
Just from a safety perspective the overall architecture should allow EStop commands to be acted on quickly, so I feel a 100ms scale of potential delay is very unlikely.

I agree with Thad and other's suggestion that all DSs are likely sent an enable packet within a much smaller window - let's assume within 5ms. With that assumption, the critical issue is where the DS is in the polling cycle when it received the enable, and that's not under FMS control. Big picture: fewer than half of matches would have one or more teams receive the enable one polling period later. With the worst case design of fixed enable order, some specific stations would have a higher probability of getting the one period delay.

So back to my earlier question: given all the other sources of variability, does a finite probability of a 20ms delay matter in the "real world"?
I don't know how consistent the top teams are at can burgling, but my gut sense is that this order of potential delay is of secondary or tertiary concern. However I'm happy to be corrected by members of those inspirational teams.

Originally Posted by MrRoboSteve

6. The FMS developers are likely to be curious about a thread on CD entitled "FMS enabling sequence."

Hi guys!

[Citrus Dad]

Originally Posted by MikeE

That's a huge assumption.

So back to my earlier question: given all the other sources of variability, does a finite probability of a 20ms delay matter in the "real world"?
I don't know how consistent the top teams are at can burgling, but my gut sense is that this order of potential delay is of secondary or tertiary concern. However I'm happy to be corrected by members of those inspirational teams.

I was positing the question because I'm not technically knowledgeable about the FMS. But it appears that a delay of 20ms, particularly if it is cumulative across stations, will be significant in the canburglar race. Teams are already under 300ms and expect much, much faster. That delay could be a high % of the time.

[Bennett548]

20 ms matters a great deal. I expect in that much time some teams would be 1/4 of the way to the cans already.

A seperate question: why are we all guessing about how FMS works? Why isn't the code 100% available to everyone? Was it developed by an external company that doesn't want their IP shared?

[Bennett548]

We have some evidence of FMS enabling sequence timing issues. At msc, in playoffs, 548 and our partner 1711 ran very similar autonomous modes (i.e. cheesecake). We both run java, and used talons srx connected by can.

The troubling issue is that 548 consistently started moving 10-30 ms before 1711. 548 was always red 1, and 1711 was red 3.

We are working with 1711 to look into DS logs to find out if there was some delay recorded there.

Can anything be done about this? As far as specific details FMS is a black box to us teams.

Hopefully there is a Snowden among the FMS developers that will spill the beans and let the community suggest improvements.