Issues receiving field configuration data and how to mitigate them

FIRST General Forum
1

At the Finger Lakes Regional in week 3, our alliance ran into some odd issues regarding receiving the field configuration from getGameSpecificMessage. I’m going to outline the problems and ways to mitigate these problems so that you don’t run the wrong auto like we did. Here is the complete story of what happened:

In 340s last qual match – qm82 – we had the field config of RRR. Everything ran as expected and we went into the elimination rounds. In our first quarterfinal match, the match started, and to the observers, we saw the field config was LLL. However, 340 then drove to the right side of the scale and scored the cube, and 1507 didn’t move at all. (It’s worth noting that 1507 uses C++ and we use Java, however, it doesn’t really change anything. It’s the same problem.)

The first thing 1507 noticed is that in autonomousInit, they had not received any game data – more specifically, that GetGameSpecificMessage returned an empty string. Since they did not handle that case, they didn’t move. We were already going through our code to try and find out why we scored on the wrong side, but we hadn’t changed anything, and we’d been running the same auto code for nearly 30 matches at that point. We found that, while 1507 had gotten an empty string from the method, we had actually gotten RRR – the last field config we got, in our previous qual match.

We immediately jumped to the question box, but (after a long delay) were not granted a replay. This isn’t really important, aside from letting you all know that if this issue happens to you, you probably won’t get a replay. The reason for this is that we were told they verified the FMS sent the correct data to the driver stations, but had no way of verifying the driver stations sent the correct data to the robots. Thus it was not a field fault.

Okay, so you might be wondering why 1507 and 340 got totally different data. This is because there are two different legal versions of the Driver Station – 1507 had the first version, and we had the second. In the first version, if the DS has not yet received anything from the FMS, it returns an empty string. In the second, if the DS has not received anything, it returns the last field config that it got (in our case, the field config from qm82 - RRR). As far as I know, this was an intentional change. This is just the info that I’ve received from others – if there is more information necessary here that I am unaware of (or wrong about), please inform/correct me.

So, what can you do, if you’re scared about this happening to you? I hate to say this – but honestly, not much. As far as I know, closing the driver station and re-opening it doesn’t clear the cache. We got LRL in our last match at FLR, and when we opened the driver station at our next meeting, it still returned LRL (after a complete reboot of the machine). We are going to downgrade the driver station because we would rather get an empty string than the wrong field config.

Okay, so if you can an empty string… now what? Technically you could use CV to look for LED colors on the switch/scale but that’s probably not realistic for most teams (and it’s definitely not what I’m investing our time into). Since we took the side for most elimination matches, we’ve elected to treat an empty string as such: If we get a field config, do the auto. If not, start a timer. Continue to ask for a field config while the timer is going, and if we get a field config, start the auto. If after 8-10 seconds, if we still don’t have a field config, run a drive forwards auto.

Aside from this, you can try to reduce your network usage. We had one print statement running in disabledPeriodic which apparently was too much. I trust the people that are smarter than I am that have told me that’s the issue, but we ran the same code for close to 30 matches before it became an issue even once.

This issue did not re-appear after the match it happened. It happened in the very first match after alliance selections. The FLR field traveled to the Pittsburgh Regional this week and it looks like the exact same thing happened there, in the first match after alliance selections, causing the 1 alliance to (attempt to / successfully) score in the wrong side of both the switch and the scale.

I’ve attached our DS logs to the post. There are logs from both the first quarterfinal match (where things went haywire) and our last elimination match (where things behaved correctly) if you would like to compare them. The log from the qf match is rather sizable, so looking through it can be tricky. The logs with the timestamp 13_17_24 are from the first match, and the final match has the timestamp 17_08_30. Unfortunately, in the heat of the moment, we did not think to create a git commit of the exact code we were running. We have a commit from after practice day and after eliminations, but anything between there is mostly guessing what we were running.

If anyone has more info on this interesting situation, please post it. I honestly do not have a solution and it worries me with the championship approaching.

logs.zip (697 KB)


logs.zip (697 KB)

2

At the greater Pittsburg regional, we experienced not only the exact same issue, but it also occured during the first quarterfinal match exclusively. Seems a bit too odd to be coincident in my opinion, might be worthwhile to look into this issue and a potential solution, field-side as opposed to team-side

3

It’s very odd, especially since it’s the same field as FLR (or so I’ve heard). Unfortunately, from a programmer’s perspective, there is not much in our control, which makes it frustrating that it isn’t grounds for a replay. I have heard grumblings of it having to do with the field trip time – but that’s a field problem, not a team problem.

There is not really a way to verify that the game data received on the robot is the correct one, which is the problem.

4

At the NC UNC-Pembroke Event last week, a team reported they got incorrect data from the FMS. I checked their logs (am a CSA) ans, in fact, they had gotten no game data from the FMS in that match. It looks like the beginning of the logs were truncated in the event viewer for some reason. Since it was their first match, I verified with their practice match logs and they had indeed gotten data then. So, the robot ended up using the stake data from their practice match.

The team then reported that their alliance member had gotten stake data, too. I checked their logs and it was missing from their logs for that match, too, but not for their practice match. I checked with their final alliance member, a team on the other alliance, a team in a subsequent match who said they didn’t receive the data, and another team in another match. They all didn’t receive game data.

I brought this to the attention of the FTA and the initial team that reported the problem was granted a replay. Thankfully, it was the beginning matches and the issue was rectified for the subsequent matches and we didn’t have issues afterwards.

I would cross-check your logs with other teams’ logs to make sure the issue wasn’t present with just your bot or one other member on your alliance. If the problem is present in other teams, I would bring it to the attention to your FTA and CSA because the one I had was the first case of the FMS not sending the game data.

5

Hi, none of the three robots on our alliance received data by the time auto started. However, our third robot, 3550, was set to drive forwards regardless of the field config, so it did not matter. But we did check their logs and they had not received the field config by the time the match started. It was an alliance-wide problem. I am not sure whether or not the opposing alliance had the same problem (they did not appear to, but that could be sheer luck, I’m not sure).

6

I am the technician (and Honorary Mechanical Deputy) for team 4028 - The Beak Squad, the captain of the first seed alliance from the Greater Pittsburgh Regional. The same issues that appeared to come up in Finger Lakes did, in fact, happen to us and, as expected, we were not granted a rematch. Team 217 - The Thunder Chickens, our first pick, ended up receiving their game data from their previous match, and the FTAs assumed that some issue with BOTH of our alliance mates ended up causing the game data to not come through. After nearly 30 minutes of investigating all THREE of our Driver Stations, the FTAs did confirm that our partners, team 5740 - Trojanators, and us did not receive ANY Game Data. Another potential explanation that they gave us was packet loss, but neither of these was, obviously, enough to call a field fault and play a rematch. We ended up losing the quarterfinals based on other issues, but this certainly heavily contributed to our loss in the first quarterfinals. As soon as my team saw this thread we shared it with our alliance partners, team 217, who were understandably as distressed as we are about this pattern. I am not able to confirm with such level of detail the Driver Station versions that everyone used, but that does seem like a valid cause of this issue. It is curious that two teams, us and 217, who have been running our autons with considerable success all weekend, would coincidentally fail in the same way as the other first seeded team on the same field from a different regional. We offered up evidence that proved that nothing in our code changed, and I am confident that nothing in the Thunder Chickens’ code would have caused this FMS failure. Between the Finals Match 1 and Finals Match 2, I approached the head FTAs of the event and informed them of this post. They begrudgingly told me that they would look into it, and I urged them to investigate further and inform FIRST Headquarters of this potential issue so it will not to another team again. I also made it extremely clear that we were not looking for them to overturn our match, but we merely wanted to protect any other teams that were going to play on this field at some point. Whether or not they act on my suggestion will be up to their discretion.
**
AFTER THIS POINT IS PURELY SPECULATION:**
While this is unconfirmed and I was not able to confirm with any volunteers of the event, my guess would be that this field is heading to Buckeye Regional next weekend. I will certainly be watching Quarterfinals 1 of Buckeye to see if there is a definitive trend.

7

Juan rocks. Just sayin’.

8

Question. I have to double to check our code to be sure, but why do you not check the string once auto starts?

9

I am the technician (and honorary mechanical deputy) or team 4028 - The Beak Squad, the alliance captain of the first seeded alliance at the Greater Pittsburgh Regional. I can confirm that the same thing that happened at Finger Lakes Regional appears to have happened to us. In our first quarterfinal match, here is what happened in auton:

  1. Team 217 - The Thunder Chickens
    , placed the cube in the wrong side of the switch
  2. We ran to the wrong side of the scale but were fortunately stopped by team 5740 - The Trojanators
    who accidentally ran into us during the auton period

Two teams, both us and 217, had been running the exact same autons with considerable success, yet we both managed to have catastrophic failures during the same quarterfinal match. We approached the question box following the match and, as stated in this thread, we were not granted a rematch. After nearly 30 minutes of deliberation, we were given two primary explanations by the Head Referee and FTAs:

  1. Something in the Thunder Chickens’ code caused them to receive their previous match’s game data, and this somehow caused both us and Trojanators to not receive any game data
  2. Packet Loss of the game data string could have caused both us and Trojanators to fail to receive

We offered to present information that nothing in our code had been changed and I am confident that Thunder Chickens’ code was not the culprit. This being said, I do not believe that the outcome of the match would have changed were the autons to go correctly. I would like to congratulate team 5811 - The Bonds and their alliance mates on the win which they certainly deserved. The failure of the autons was not the only issue we were facing, but it did contribute heavily to our poor performance in the first quarterfinal.
After first seeing this post, my team brought it to the attention of 217 who were understandably as distressed as we were. During the official timeout between Finals Match 1 and Finals Match 2, I approached the question box and had a brief discussion with the two, head FTAs at the event. I made it extremely clear that we were not looking for them to overturn the match, but merely to read this post and investigate this apparent issue. I urged them to inform FIRST Headquarters of this issue to protect other teams from experiencing this same problem and potentially losing them the tournament. Whether there is any action to prevent this issue from happening further is now up to FIRST’s discretion.

AFTER THIS POINT IS PURELY SPECULATION:
My personal guess would be that this field would be heading to the Buckeye Regional next weekend, so I will certainly be looking out at quarterfinals 1 to see if this issue persists. Unfortunately, before my team left I was not able to confirm this with anyone working, so it has not been confirmed where this field is going.

10

Do you mean in autoInit or autoPeriodic? We check in autoInit. The data hadn’t arrived to the driver station by that time.

11

I wonder if to avoid the issue of old game data if it would be wise to set
gameData = null; before gameData = DriverStation.getInstance().getGameSpecificMessage();

or is the issue that the DriverStation is stuck? More Likely since the Robot had been rebooted.

Is there other data we could get to verify fresh data like match number, Alliance color… Then using the dash board monitor these or enter them so the robot can verify it has the same data before auto?

I know that SuffleBoard has a field widget the populate the plate colors and alliance colors from the drivers point of view so the data is there for that. But alliance color may not change for you each round and plate color is to late by the time it matters.

12

The problem here is that according to the WPILib docs, the function call may return data from a prior match if called before the FMS sends the data. The problem is, as exhibited in week 1, the FMS may not have sent the data at the time autonomousInit() is called, or even for a few cycles of periodic.

If you’re testing for null, the stale data returned may look legit. On the other hand if you check to see if there is data during the disabled period prior to init() and wait for it to change, there’s no clear indication that the current match is any different from the last match. (And with fewer valid combinations, this is far from an unlikely occurrence.)

Our testing showed that we were not getting stale values after a DS restart, but clearly that’s not consistent with what everyone else is seeing…

At this point I believe it’s critical for the WPILib people and NI to figure a way to make this function call deterministic – or give us some other way to validate the stale-ness of the data.

13

Ah, we don’t use the periodic framework.

We use the sample/simple robot framework. We’ve been using it a long time, and have not had any reason to change. I still don’t know why they deprecated it this year…

We don’t check game data until the actual autonomous function is called. In your case, autonomous periodic.

In 14 matches during week 1 we never saw any game data issues.

14

Is it possible to go in and manually delete the old data from the driver station after each match?

15

Yep. We were nailed by this as well, once. We look for null and then the first change to have data. Never saw the null so ran our default.

We also echo the data to our dash, because we figured something like this would happen. The data we had did not match the match data and was clearly our data from the previous match.

I’m at a loss as to how to fix this on our side.

16

I just checked our logs from Miami Valley because we were on the number one alliance as well. We got the correct data, RLR, in quarterfinal 1.

17

Unfortunately, there is not a lot it seems a team can do from their side to prevent this from happening, and at the same time, teams aren’t being granted replays when it does happen.

It seems like it isn’t an FMS fault, but instead a DS fault. Then is the DS at the hands of the field, or the teams? Since the DS is closed-source and not at all under the control of teams, it seems like it wouldn’t be the teams fault for it breaking. But if it isn’t an FMS/field fault (and thus not grounds for a replay), then who is at fault? What happens if this happens in a championship division? What about Einstein? What happens then?

18

I am sorry I didn’t post this sooner. I was hoping I would get a clearer explanation speaking with some other people. I was the one to review the NYRO-QF1 match. It is tough to say why the robots acted the way they did.

And yes, 340 looks for the GameData in autonomousInit() the way FIRST suggests.

Corrections - If you look at the logs posted, you’ll see that “Game Data MessageLLL” in logged at 1:37:47.929. This is the time the DriverStation program received the GameData. This is not when the robot received the GameData, there is no log for that. As a second point of correction(sorry here Justin) there is no explicit proof that 340 ran code based on RRR because the team does not print back the Game Data. Though I agree that it looks like 340 ran code based on “RRR” GameData. 1507’s code had a clear error about not getting GameData. And lastly, to clarify 340 had one print statement printing very often, because it is once a code loop.

Because the logs for all three teams showed “Game Data MessageLLL” prior to the start of the match there was no replay. But this isn’t saying that the robots got that message, there is no built in system to prove that. In Beta testing it took 3-4 seconds for the data changes to be passed to the robot. This delay is the point behind the PowerUp sound prior to autonomous starting.

Understand your robot can not talk with FMS, all communication with FMS is through your own DriverStation program. So the GameData is transmitted to the DriverStation program, logged, then after which it is FIRST policy that it is on the team. The expectation is that the TCP packet from the DriverStation program to the robot will not get lost and that it will be sent and received in the ~3 seconds the field takes to do the PowerUp sound.

In the match Justin brings up, NYRO-QF1, there odd trip time/packet loss on the red alliance. Unclear if it was on the blue alliance too. So this might be why the GameData did not seem to make it to the robots in time.

As to checking the Alliance Color or Match Number, cpapplefamily, it is my understanding that this is updated when the FMS runs pre-start and then it is transmitted to the bot as soon as the DriverStation is plugged into the field after that point.

My understanding of the DriverStation program, you can delete the GameData field, so that it is null again. Just be sure to do it before you connect to FMS, as the GameData field becomes hidden from the user then.(at least that is my understanding Classified*)

Best Practices:
1. Print back the GameData your robot receives, as you make your choice, this can help with explaining why your robot reacts a certain way. Use Print Statement so it appears in the logs. (It might even be used to help you prove your robot didn’t receive the GameData)

System.out.println("GameData Is "+ DriverStation.getInstance().getGameSpecificMessage());

2. Delete the GameData on your DriverStation program BEFORE a match. Even BEFORE charging your pneumatic and the like. (clean slate)
3. Write your code to wait for valid GameData before making a choice as to what autonomous to do.(as Justin mentions)
4. Before you score in autonomous, double check the GameData that you are scoring for yourself.(once 340 scored against themselves, it was hard to comeback)
5. Reduce your network traffic. Print Statements, SmartDashboard, Error calls, anything that uses a packet, reduce the total number of packets. (there is no direct proof this caused the trip time/packet loss seen in NYRO-QF1 or that those things caused GameData packets to delay, but couldn’t hurt)

I would not suggest down grading your DriverStation program, especially if you are running Java, there were fixes to avoid crashes added, I believe.

A report was written about the NYRO-QF1 GameData. I’ll release it in the coming week, but I left time for FIRST to comment on it as it is only fair and a good way to eliminate potential bias and alternative explainations.

For instance, you might say the same issue came up in PACA-QF1, and yes that is the same field, but in NYUT-QF1 340, nor its partners claim problems with GameData and it is still the same field.

19

Thank you for this thread and these notes. We had the same general behavior described happen to us in two matches in the Sundome event (despite an entire previous event of perfect auto runs and every other time at the Sundome). It was quite frustrating for our students. We will review the suggestions in this thread and implement accordingly.

20

You are welcome.

But remember these are edge cases, and are infrequent. For instance it might look like 3986 had experienced a similar problem in NYRO-QF5(or QF1m2) that the red alliance complained about in NYRO-QF1(or QF1m1). But their own print statements fed back the proper “LLL” in their own logs. Sometimes it is a coding error, but give help to the field personal when you can.