FMS enabling sequence?

[Citrus Dad]

The question came up this weekend in Sacramento as there were at least 3 fast auto can grabbers in the field: Does the FMS enable robots in sequence during auto, or simultaneously? And if its in sequence, is it known which stations enable in sequence and what is the delay between each station?

High speed video of 1678 and 971 in a semifinal match show that a delay could affect match outcome.

[AdamHeard]

The question came up this weekend in Sacramento as there were at least 3 fast auto can grabbers in the field: Does the FMS enable robots in sequence during auto, or simultaneously? And if its in sequence, is it known which stations enable in sequence and what is the delay between each station?

High speed video of 1678 and 971 in a semifinal match show that a delay could affect match outcome.

Do you have video?

[ATannahill]

I suggest you view the FMS White Paper (http://wpilib.screenstepslive.com/s/4485/m/24193/l/291972-fms-whitepaper) to better understand how the field works.

[Kyler Hagler]

Same thing happened on Dallas week one. There were a couple of can grabbers and 118 showed us high speed footage that showed us not moving for ~60ms from when the opposing alliances robot was moving. We were told that the FMS does enable in a sequence but that's all we were told. Don't quote me on the last part.

[rich2202]

There is one Ethernet connection between the FMIS Server and the rest of the world. The commands have to go out serially.

I'm guessing the Cisco Wifi router has 3 channels, and 2 robots are assigned to each channel. Wifi Contention could delay the receipt of commands by a robot.

If the command is routed through the Driver Station, then there is minimal delay between when FMIS sends the Auto Start, and when your DS is first able to react.

How fast your robot can react to the DS commands depends upon how much wifi traffic there is on your signal. Also, the Cisco Router has one ethernet connection, so how fast your DS responds will determine where in the que is your command.

[Tom Bottiglieri]

I suggest you view the FMS White Paper (http://wpilib.screenstepslive.com/s/4485/m/24193/l/291972-fms-whitepaper) to better understand how the field works.
This paper does not address how the FMS software goes about enabling robots, only the order in which it traverses match state.

We have not measured exactly delay, but have anecdotal evidence of seeing some robots move before others. Field timing issues were pretty prevalent last year and it's unclear to me any of it has been fixed.

Unfortunately, the delay from the FMS may decide Einstein.

[FrankJ]

There is one Ethernet connection between the FMIS Server and the rest of the world. The commands have to go out serially.

I'm guessing the Cisco Wifi router has 3 channels, and 2 robots are assigned to each channel. Wifi Contention could delay the receipt of commands by a robot.


The FMS AP has 6 SSIDs Each robot & DS is own its own VLAN & AP. Presumably there is only one ethernet connection to the FMS server. The FMS server sends the enable to the Driver stations which then send the enable to the robots. I don't know how you would be able to predict how much jitter is in all that what testing. Since the enable is being processed by the DS, maybe the speed of the DS has some effect as well?

[dcarr]

I'd be interested to hear how this was approached with Cheesy Arena.

[Thad House]

Based on things I've seen in the past, its a mix between ds, fms, fpga and the robot code itself. Based on my understanding I would say the fms actually affects it very little.

[Jared Russell]

I would think that each DS is running its own polling loop asynchronously at ~50Hz (the rate of driver control packets). So even if the disabled bit toggles simultaneously for all six bots from the FMS (which it doesn't since the FMS->DS packets must be sent serially, but it ought to be close), there could be a 20ms swing in when each DS sends the first enabled packet depending on the phasing.

( I am just guessing at how the FMS and DS are architected here since I don't think it is publicly documented; please set me straight if I have made an incorrect assumption. )

Realistically, with our wireless links, synchronization is always going to be hard.

[wireties]

It could be multicast to begin to deal with the delay of serially issuing the "change of state" messages.

[Thad House]

I would think that each DS is running its own polling loop asynchronously at ~50Hz (the rate of driver control packets). So even if the disabled bit toggles simultaneously for all six bots from the FMS (which it doesn't since the FMS->DS packets must be sent serially, but it ought to be close), there could be a 20ms swing in when each DS sends the first enabled packet depending on the phasing.

( I am just guessing at how the FMS and DS are architected here since I don't think it is publicly documented; please set me straight if I have made an incorrect assumption. )

Realistically, with our wireless links, synchronization is always going to be hard.

There's also another 20+ ms delay caused by the wpilibs if you use anything but the sample robot template. Could end up being up to 40ms depending on how the fpga loops.

[Jared Russell]

There's also another 20+ ms delay caused by the wpilibs if you use anything but the sample robot template. Could end up being up to 40ms depending on how the fpga loops.

Please elaborate?

[Jared]

Here's a frame by frame of the Dallas Q39 (118 and 3310 in same match).

118 is going for their 3 tote auto, 3310 is going for containers, and a team 932 (middle of field red side) is driving forward at a high speed in auto mode.

Frame 1 - Yellow lights on field illuminate
Frame 2 - Red/Blue DS stack lights illuminate
Frame 3 - Nothing
Frame 4 - 932 tips back and begins movement
Frame 5 - 932 continues to move
Frame 6 - 3310 begins to drop arms

Another frame by frame on Dallas F1 (118 grabbing from step, 148 auto)

Frame 1 - Yellow lights
Frame 2 - stack lights
Frame 3 - Nothing
Frame 4 - 148's arm moves
Frame 5 - 118 drops arms

From watching a few other matches, there is usually a different number of frames between lights and robot movement for teams like 118 and 3310.

[Thad House]

Please elaborate?

In LV I know that it runs disabled for 1 extra loop after it receives an enabled packet. Then it will run auto on the next enabled packet.

Based on the last time I looked in the Java code, it will run auto init, then the next packet run auto periodic. So unless you are starting your sequence in auto init, it wont run motors until it gets a second enabled packet.

Note that the whole time this is happening, the fpga is enabled, so it will run 20ms with the values it last receives. This is why there are times the robot jumps between disabled and enabled.

[Jared Russell]

Based on the last time I looked in the Java code, it will run auto init, then the next packet run auto periodic. So unless you are starting your sequence in auto init, it wont run motors until it gets a second enabled packet.

I do not think this is currently the case in Java (for IterativeRobot at least). It does check for the new data flag (via nextPeriodReady()), but this flag is not cleared after calling autonomousInit(), so autonomousPeriodic() should be invoked on the first packet

[Bryan Herbst]

I would think that each DS is running its own polling loop asynchronously at ~50Hz (the rate of driver control packets). So even if the disabled bit toggles simultaneously for all six bots from the FMS (which it doesn't since the FMS->DS packets must be sent serially, but it ought to be close), there could be a 20ms swing in when each DS sends the first enabled packet depending on the phasing.

( I am just guessing at how the FMS and DS are architected here since I don't think it is publicly documented; please set me straight if I have made an incorrect assumption. )

Realistically, with our wireless links, synchronization is always going to be hard.

I came here to say the same thing. Regardless of whether the FMS sends out the signals in parallel or in serial, the time between the FMS sending out each packet is going to be negligble compared to any number of other factors in the system. An obvious unknown here is that each team's driver station has its own limitations due to wildly varying hardware and drivers.

Any anecdotal or video "evidence" is rather worthless in this discussion without knowing a lot more about each robot (including seeing its code). If team X's arm appears to move at 0:00:00, but team Y's wheels start spinning at 0:00:20, how do we know which team got the "start auto" signal first? It is entirely possible that team Y's code is simply slower or explicitly waits 20ms. It is also possible that there are mechanical limitations that caused team Y to start visibly moving later.


It would certainly be interesting to see some hard data on when robots across a number of matches receive the signals, but I wouldn't rely on any such data being universally accurate, nor would I hope to have perfect synchronicity in robot start times.

[Pat Fairbank]

I'd be interested to hear how this was approached with Cheesy Arena.
Cheesy Arena sends DS packets at match period boundaries (with 10ms resolution) and at 250ms intervals. A single thread of execution decides whether a packet needs to be sent, and sends all six in sequence (by iterating through a hashmap of the driver stations, so the order is not deterministic). The DS packets are UDP, so all six are probably sent over a period spanning far less than a millisecond. I would guess that the six driver stations would receive the "enable" signal within a similar total time frame.

It's well known that the FMS doesn't communicate directly with the robot; the FMS informs the DS what state the robot should be in and the DS passes it along with the joystick information. What I don't know is whether the DS syncs to send a robot packet whenever it receives an FMS packet, or if it only passes on the enable signal next time its regular loop runs. If it's the latter case, then the order and timing of robot enablement is determined presumably by when each driver station process was started and by the DS-to-robot packet interval.

Ideally, the DS should send a packet to the robot as soon as it receives an FMS packet that changes its disable/enable/auto/teleop state, regardless of when it last sent a packet. Someone could probably test whether it does this or not using Wireshark.

[Jared]

On the contrary, I think I figured it out directly from the FMS Whitepaper.

From the FAQ, "Does the FMS Control the Robot", FMS sends the commands to the DS, which then forwards the command to your robot to start.

Does the command get sent to all 6 DS laptops at the same time? The server in the scorpian case has one ethernet port for all red and one ethernet port for all blue robots. Does the software send red and blue simultaneously, or is there a single network adapter + a switch for both red and blue, meaning one must be sent before another. How much time is between these two commands? Within the separate red/blue disable commands the 1, 2, and 3 station enable commands must happen serially. Are they sent immediately after each other, and if not, what is the delay.

It might seem a silly question to ask if there is a significant delay between sending red and blue side commands, but hey, that's exactly what happened last year with the hot goal lights, which were +/- 4 seconds in terms of accuracy at times.

These ethernet cables run to control cases (SCC's) under the DS shelf, where they plug into an ethernet switch. This ethernet switch is connected to an Allen Bradley Stratix 8000, which has three other ethernet ports that are dedicated to 1, 2, and 3 driver stations. What goes on here is unknown, but it's not unreasonable to expect that there could be delays here. The Allen Bradley box is doing something other than being a switch because it won't work if you plug the wrong team into the wrong ethernet cable.



While it's not explicitly stated, most command systems have a processing loop that will "process" incoming commands at a known rate (timed so you don't chew up your CPU). Now, even if we knew the loop rate we don't know where in the loop the command will come in; will the command arrive immediately at the beginning of that loop, in the middle, or at the end?


It depends on how the polling is set up. If they check for 'FMS says enable' every time they send the command (20 ms), then timing can't be accurate to more than 20 ms, as you pointed out.

The FMS software and DS are written in C# (or c++, I forget), and LabVIEW, both of which easily support event driven behavior, meaning the driver station may respond to an FMS command as soon as the network buffer reads in the message.


That's probably the difference - I'd bet the difference in time it takes FMS to issue the command across all the VLANs to the DS's would be in the nanosecond (microsecond for sure) range.

-MM
You'd think, but the difference in time it took the FMS to send commands to the phillips lighting controller last year was sometimes greater than 2 or 3 seconds. FMS timing seems to be properly understood by nobody.

[MikeE]

Field timing issues were pretty prevalent last year and it's unclear to me any of it has been fixed.

It might seem a silly question to ask if there is a significant delay between sending red and blue side commands, but hey, that's exactly what happened last year with the hot goal lights, which were +/- 4 seconds in terms of accuracy at times.


You'd think, but the difference in time it took the FMS to send commands to the phillips lighting controller last year was sometimes greater than 2 or 3 seconds. FMS timing seems to be properly understood by nobody.

I think it's fairly widely known that the FMS has been substantially (completely?) rewritten since last season, so anecdotes from 2014 are of limited applicability to 2015.

One of the questions I was taught to ask early on in Grad school is what scale of effect actually matters in a given experiment, and to focus only on those factors that have meaningful impact.
If there are no systematic biases towards particular team stations, how much does a worst case 20ms delay due to polling actually affect a mechanism compared to all the other variables in play?

[Tom Bottiglieri]

If there are no systematic biases towards particular team stations
That is the part I am interested in learning more about.

[plnyyanks]

On a different note, I would love an update to the FMS Whitepaper that goes into much greater technical detail. Or maybe a 1 day recorded conference, where someone on FRC engineering can do a very detailed, technical walkthrough of how the field works (and maybe show some code snippets), including a Q&A.

I don't like how the field electronics are a 'black box' to most teams - I gained most of my knowledge about how the field works from volunteering and working with it regularly, something teams don't do.

A greater understanding about how the competition field works will reduce the amount of anger and frustration often (mis)directed at FMS.

[YPatrick]

Slow-Mo: https://docs.google.com/file/d/0B9RPxLljFfi-TlVOSUU2dXVkSEU/edit

Real Time: https://docs.google.com/file/d/0B9RPxLljFfi-Uy1aVFhIWlotdEE/edit

There is an interpolated clip that is a little bit slower, so if people are interested I could post the link to that one as well.

[Citrus Dad]

Cheesy Arena sends DS packets at match period boundaries (with 10ms resolution) and at 250ms intervals. A single thread of execution decides whether a packet needs to be sent, and sends all six in sequence (by iterating through a hashmap of the driver stations, so the order is not deterministic). The DS packets are UDP, so all six are probably sent over a period spanning far less than a millisecond. I would guess that the six driver stations would receive the "enable" signal within a similar total time frame.

It's well known that the FMS doesn't communicate directly with the robot; the FMS informs the DS what state the robot should be in and the DS passes it along with the joystick information. What I don't know is whether the DS syncs to send a robot packet whenever it receives an FMS packet, or if it only passes on the enable signal next time its regular loop runs. If it's the latter case, then the order and timing of robot enablement is determined presumably by when each driver station process was started and by the DS-to-robot packet interval.

Ideally, the DS should send a packet to the robot as soon as it receives an FMS packet that changes its disable/enable/auto/teleop state, regardless of when it last sent a packet. Someone could probably test whether it does this or not using Wireshark.

That's interesting. So an alliance might want to be careful as to the order in which it connects drivers stations. (And this can be a real world issue.)

[Thad House]

I do not think this is currently the case in Java (for IterativeRobot at least). It does check for the new data flag (via nextPeriodReady()), but this flag is not cleared after calling autonomousInit(), so autonomousPeriodic() should be invoked on the first packet

Ah. They might have changed that then.

However, LV still does do this, and runs disabled one last time, and still has to wait for another packet to be sent before actually enabling auto. so auto in LV will have a delay of one packet before actually running auto code.

[Citrus Dad]

If there are no systematic biases towards particular team stations, how much does a worst case 20ms delay due to polling actually affect a mechanism compared to all the other variables in play?

If the serial enabling differences are cumulative, the first bot will move 0.1 second before the last one. That's more than the difference in speed between several of the can grabbers at the moment.

[Thad House]

If the serial enabling differences are cumulative, the first bot will move 0.1 second before the last one. That's more than the difference in speed between several of the can grabbers at the moment.

It would never be that high. Even serially, I bet it still sends all the enable packets within 1ms of each other. The 20ms polling loops happen on the DS, and the FMS does not depend on it. So the FMS wouldnt accumulate based on those. It would only matter to the DS at which point in the 20ms period it recieves the enable packet, and thats if its not interrupt based. So worst case is that 1 robot gets an enabled packet 19ms after another one.

Now its entirely possible something else in code is delaying more, but I cant see how the FMS would take 60+ms to send enable packets. If so, we would visually see it.

[FrankJ]

...
These ethernet cables run to control cases (SCC's) under the DS shelf, where they plug into an ethernet switch. This ethernet switch is connected to an Allen Bradley Stratix 8000, which has three other ethernet ports that are dedicated to 1, 2, and 3 driver stations. What goes on here is unknown, but it's not unreasonable to expect that there could be delays here. The Allen Bradley box is doing something other than being a switch because it won't work if you plug the wrong team into the wrong ethernet cable.
...
FMS timing seems to be properly understood by nobody.

The switches are managed. Which means they only route packets to specific ports based on the IP address (or MAC). Managed switches are always higher end than generic switches. You are not losing any significant time in the switches. The travel time for Ethernet packets between the DS & robot are logged on the DS. I expect the travel times between FMS & the DS are the same or better.

I expect the FMS timing is understood by the people involved in writing the programs. It just not public information.

[themccannman]

So, since this doesn't seem resolved is there any way we can figure this out aside from FIRST publicly posting the FMS code? They only other way I can think is setting up two robots with known mechanism speeds and comparing them at each possible driver station combination. This would be pretty difficult to do though as you would need to reserve an entire field for only 2 teams for at least 30-60 mins, likely not possible with time constraints and demand for practice matches at most competitions.

[MrRoboSteve]

I have no knowledge of the internals of the FMS software, so these observations are based completely on the whitepaper and other public sources.

1. The minimum interpacket gap for 100mbps Ethernet is 0.96 µs.

2. The FMS code could take longer than 0.96 µs to send the "auto enable" command to each DS, depending on what the logic looks like in the FMS code. For example, there could be logging being written to disk between each initiate command.

3. The protocol between the FMS and the DS isn't documented in the whitepaper.

4. The FMS whitepaper says that robot <--> DS control packets are prioritized over video packets. I read that as saying that UDP 1130 and 1140 are prioritized over the other ports available for robot <--> DS communication. The whitepaper is silent about the relative priority of FMS <--> DS packets. A reasonable guess is that FMS <--> DS packets are at the same as robot <--> DS control packets. If this guess is true, it's unlikely that robot -> DS data on the open ports (e.g. camera packets) from the robot will interfere with the "auto enable" command from the FMS.

5. It is likely that one of these two statements is true:
a. The "auto enable" command is sent to each DS in the same order in every match (e.g., Red 1/2/3, Blue 1/2/3)
b. The "auto enable" command is sent to each DS in a purposely randomized order in every match

Why are these likely? Because a) is the easiest to code, and b) would be the logical alternative if they made a specific design decision here.

6. The FMS developers are likely to be curious about a thread on CD entitled "FMS enabling sequence."

[MikeE]

Caveat: I have had training on use of the FMS but do not know details of it's implementation. I'm engaging in this discussion because it is a interesting topic and educated speculation is fun!

If the serial enabling differences are cumulative, the first bot will move 0.1 second before the last one. That's more than the difference in speed between several of the can grabbers at the moment.

That's a huge assumption. Not only would that require serial enabling, but it assumes only one packet is being sent anywhere per 20ms time slot. As other posters have stated it's much more likely that packets are being sent to all DS within the same 20ms frame.
Just from a safety perspective the overall architecture should allow EStop commands to be acted on quickly, so I feel a 100ms scale of potential delay is very unlikely.

I agree with Thad and other's suggestion that all DSs are likely sent an enable packet within a much smaller window - let's assume within 5ms. With that assumption, the critical issue is where the DS is in the polling cycle when it received the enable, and that's not under FMS control. Big picture: fewer than half of matches would have one or more teams receive the enable one polling period later. With the worst case design of fixed enable order, some specific stations would have a higher probability of getting the one period delay.

So back to my earlier question: given all the other sources of variability, does a finite probability of a 20ms delay matter in the "real world"?
I don't know how consistent the top teams are at can burgling, but my gut sense is that this order of potential delay is of secondary or tertiary concern. However I'm happy to be corrected by members of those inspirational teams.

6. The FMS developers are likely to be curious about a thread on CD entitled "FMS enabling sequence."

Hi guys!

[Citrus Dad]

That's a huge assumption.

So back to my earlier question: given all the other sources of variability, does a finite probability of a 20ms delay matter in the "real world"?
I don't know how consistent the top teams are at can burgling, but my gut sense is that this order of potential delay is of secondary or tertiary concern. However I'm happy to be corrected by members of those inspirational teams.


I was positing the question because I'm not technically knowledgeable about the FMS. But it appears that a delay of 20ms, particularly if it is cumulative across stations, will be significant in the canburglar race. Teams are already under 300ms and expect much, much faster. That delay could be a high % of the time.

[AdamHeard]

So, since this doesn't seem resolved is there any way we can figure this out aside from FIRST publicly posting the FMS code? They only other way I can think is setting up two robots with known mechanism speeds and comparing them at each possible driver station combination. This would be pretty difficult to do though as you would need to reserve an entire field for only 2 teams for at least 30-60 mins, likely not possible with time constraints and demand for practice matches at most competitions.

It would be easier to add LEDs to all robots.

Unfortunately (as far as I can tell) the RSL doesn't visually convey exactly when teams are enabled.

[orangelight]

It would be easier to add LEDs to all robots.

Unfortunately (as far as I can tell) the RSL doesn't visually convey exactly when teams are enabled.

I think that would be a great idea. We always have to guess what time the robot is enabled when we run our canburglars.

[Wetzel]

6. The FMS developers are likely to be curious about a thread on CD entitled "FMS enabling sequence."

Fine entertainment on the internet super highway.:cool:

[Bennett548]

20 ms matters a great deal. I expect in that much time some teams would be 1/4 of the way to the cans already.

A seperate question: why are we all guessing about how FMS works? Why isn't the code 100% available to everyone? Was it developed by an external company that doesn't want their IP shared?

[Bennett548]

We have some evidence of FMS enabling sequence timing issues. At msc, in playoffs, 548 and our partner 1711 ran very similar autonomous modes (i.e. cheesecake). We both run java, and used talons srx connected by can.

The troubling issue is that 548 consistently started moving 10-30 ms before 1711. 548 was always red 1, and 1711 was red 3.

We are working with 1711 to look into DS logs to find out if there was some delay recorded there.

Can anything be done about this? As far as specific details FMS is a black box to us teams.

Hopefully there is a Snowden among the FMS developers that will spill the beans and let the community suggest improvements.

[MikeE]

20 ms matters a great deal. I expect in that much time some teams would be 1/4 of the way to the cans already.

We will actually know in a couple of days. If they do exist then 80ms canburglers will be a sight to behold! In the meantime there are hundreds of things a team can directly control.

Hopefully there is a Snowden among the FMS developers that will spill the beans and let the community suggest improvements.

While I don't think that's an apt comparison, after this past winter Moscow might be a welcome respite (http://en.wikipedia.org/wiki/Climate_of_Moscow#Snow_cover) from the New Hampshire weather.