Errors at Colorado Regional

[walruspunch]

Team 1977 was wondering about the error of some kind at the 2014 Colorado Regional. By our count, somewhere between 10 and 20 teams had issues during matches. Some of that is likely coincidence, but there was way too much interference to mark it all as something from a team. Our robot worked perfectly on the practice matches and in the pit while tethered, but for the first day of qualifiers, we could do nothing. No autonomous, no teleop, nothing. As soon as we put it on the field, we went dead, with no robots hitting us or interference. When we brought it back to the pit, it worked perfectly every time. We had several officials look it over and find no explanation. We also had several people check over the code and find no explanation. We finally made it work, albeit with lag, by disabling the autonomous feature and switching out the CRIO. However, other teams that had communication issues had been cleared up on their own, leading to the possibility that making these changes was coincidence. Our robot was the worst off, losing a full day of competition, but another team was only able to run for half of every match before losing connection. Several times, a team would be dead at the start of a match and gain connection about 10 seconds into teleop mode. Rebooting the CRIO helped for a time, but it takes a lot of time within each match and only seems to be a temporary fix. It seems too big of a deal involving too many teams to be marked down as an error on every team. (Unsure, but it may have happened to a team in the finals as well. Last 10 seconds they were not moving, but it could have been driver error.)

It's possible that all this could be coincidence, it just doesn't seem likely. If anyone has had a similar problem where a robot works tethered in the pit but not on the field, please reply.

[Nick Lawrence]

When you were testing in the pit, did you put the Driver Station into 'Practice' mode?

- Nick

[Mackenzie W]

Team 1305 also had a similar CRIO issue at Waterloo. It worked perfectly fine the first day, but on the final day it suddenly stopped working when connected to FMS. It worked perfectly in our pit as well in all of the modes, none of the volunteers or other teams could replicate the problem off of the field. We solved it by switching out the CRIO as well, and were able to compete in our next two regionals without issue.

To any teams who have this issue and need a quick fix, reboot the CRIO in the driver station before the match starts while connected to FMS. In our driver station logs, we determined that the CRIO never entered disabled mode, somehow dying in the middle of starting up, no matter what the code was. By rebooting the CRIO, it somehow let the robot enter this disabled mode, and therefore function normally.

We have no clue what triggered it to fail though, we only left it overnight.

[AshWalker]

Team 4293 experienced problems at the Colorado Regional. Our cRio would reboot during matches with no clear explanation.

Our problem was traced down to debris in the cRio and a flaky power connector.

We owe a huge thanks to the FTAs and the CSAs for all of their help trying to troubleshoot everything!

[jbsmithtx]

I noticed this happened a lot at the Dallas regional, as well as Hub City too. We had code that would run perfectly while tethered, and then when we put the robot on the field, we would run into all sorts of issues. I don't know what happened, and I don't know why it did, but I think that FIRST should be a little more transparent with the FMS. Yes, it must be secure, but I know for a fact that there was a driver station that caused more robots to be down than others. I can pull scouting data and show that the robots that occupied this station had problems. And even some scouts came up to me and asked what was going on because they noted it too. It was tough to play qualification matches with this colored alliance, because often we were having to run two robots instead of the three.

I don't want to blame FIRST, but things like this seem a little suspicious to me. And often all I got from asking an FTA is "It must be problems with your robot", Granted, I did sit down and find my errors at our first regional, but there were still several times in which this happened, even after scrutinizing my code between regionals.

[E Dawg]

Our robot was working tethered in the pit but not on the field. Towards the end of the match we would lose connection to our robot. Originally we thought it was the field. However, after many hours of looking it over, we figured out that our PD board was outputting voltage incorrectly and causing all sorts of issues. So the moral of the story is: in all likelihood the teams appearing to have difficulty with the field just had electrical or programming errors that were giving them trouble. The FMS itself was probably not the problem. Because you were able to fix it by disabling your autonomous, I'm guessing that somewhere in there you were calling something that was causing a whole bunch of errors and stopping you from connection to the field.

Thanks to the FTA's and CSA's for helping us figure this out!

[pyroslev]

If you can provide the following information, perhaps we can provide a differential diagnosis (without the House sarcasm/snarking)

Programming Language
Which CRIO
Pneumatics? If so, how much?
Vision processing? If so, on or offboard via Raspberry Pi or such things?
Drive Train/wheels
Electronics correctly wired, including the dedicated wiring?
What the CSA/FTA/FTAA advised?
When you swapped the CRIO, were the modules swapped too?
Digital Sidecar tested good?
No tripped breakers?

I had a team with similar issues. They were using Java with offboard Raspberry Pi Vision processing, mecanum drive and LARGE 24 Solenoids wired off of the CRIO.
In the end, we corrected the wiring, disabled the Pi, Went to a basic code and swapped all the modules. By then, she was purring like a kitten and rebuilt the code. We didn't open the CRIO or swap it but I think if I had opened it up there would have been some debris in it.

I know it is a lot to ask post mortem but it helps to put issues to bed before St. Louis and the "Off Season".

[Mark McLeod]

In general, scouters and anyone sitting in the stands for that matter are really bad sources of random data on robot/field troubles.
I used to be one of them and I remember how wrong it all looks from the distance of the stands.
All they ever take notice of is that a robot isn't moving (yes/no).

Folks in the stands can't know the difference between a drive team choosing not to move (strategic, threw a chain, because the robot will break), a Ref disabling a robot for transgressions (dragging bumpers, outside starting configuration), a Driver Station issue (poor default power mgt settings, low battery, bad USB/Eth ports, unplugged USB/Eth), or a robot issue (wiring loosening up after being hit all day, code errors, DSC shorts or CAN wiring glitches that prevent driving but leave all other functions running, low battery dips rebooting everything).

Because the robot side of FMS is so simple, network problems are easy for the field crew to spot and deal with.

If you want to be scientific and gather actual data rather than anecdotal evidence, then scouters have to:

1. Ignore driver station positions in their reports (avoid confirmation bias-position can always be looked up later if it's important to you).
2. Note the RSL (solid off or solid on - not blinking) when a robot stops moving.
3. Note the team light (e-Stops are highlighted by the bottom yellow light)
4. Send drive team scouters to inquire at the team pits for the reason for the robot non-movement (only the drivers know, often not the pit crew and certainly never random team members)
5. Use pit scouters to ask how much testing time the robot had before the event. Face it, the vast majority of us build a robot in 6 weeks and collision test it only at the competition - that's why teams are shaking bugs out all through Championships.
6. The absolute best is to inspect the team Driver Station logs, not that teams are likely to let you look at them, but you can check your own. The robot is actually providing the data, so any loss of comms is pretty obvious, because the log goes blank. If it comes back again, then your robot rebooted. If the data is there all through the match, then your comms were good.
7. Volunteer for field duty - they keep logs on all robot failures.

[Conor Ryan]

Originally Posted by Mark McLeod

In general, scouters and anyone sitting in the stands for that matter are really bad sources of random data on robot/field troubles.
I used to be one of them and I remember how wrong it all looks from the distance of the stands.
All they ever take notice of is that a robot isn't moving (yes/no).

Folks in the stands can't know the difference between a drive team choosing not to move (strategic, threw a chain, because the robot will break), a Ref disabling a robot for transgressions (dragging bumpers, outside starting configuration), a Driver Station issue (poor default power mgt settings, low battery, bad USB/Eth ports, unplugged USB/Eth), or a robot issue (wiring loosening up after being hit all day, code errors, DSC shorts or CAN wiring glitches that prevent driving but leave all other functions running).

Because the robot side of FMS is so simple, network problems are easy for the field crew to spot and deal with.

If you want to be scientific and gather actual data rather than anecdotal evidence, then scouters have to:

1. Ignore driver station positions in their reports (avoid confirmation bias-position can always be looked up later if it's important to you).
2. Note the RSL (solid off or solid on - not blinking) when a robot stops moving.
3. Note the team light (e-Stops are highlighted by the bottom yellow light)
4. Send drive team scouters to inquire at the team pits for the reason for the robot non-movement (only the drivers know, often not the pit crew and certainly never random team members)
5. Use pit scouters to ask how much testing time the robot had before the event. Face it, the vast majority of us build a robot in 6 weeks and collision test it only at the competition - that's why teams are shaking bugs out all through Championships.
6. The absolute best is to inspect the team Driver Station logs, not that teams are likely to let you look at them, but you can check your own. The robot is actually providing the data, so any loss of comms is pretty obvious, because the log goes blank. If it comes back again, then your robot rebooted.
7. Volunteer for field duty - they keep logs on all robot failures.

I cannot emphasize the importance of all of this enough. If you have further questions about specific on field issues at your event I recommend going to the question box.

[Jon Stratis]

Originally Posted by Mark McLeod

In general, scouters and anyone sitting in the stands for that matter are really bad sources of random data on robot/field troubles.
I used to be one of them and I remember how wrong it all looks from the distance of the stands.
All they ever take notice of is that a robot isn't moving (yes/no).

Folks in the stands can't know the difference between a drive team choosing not to move (strategic, threw a chain, because the robot will break), a Ref disabling a robot for transgressions (dragging bumpers, outside starting configuration), a Driver Station issue (poor default power mgt settings, low battery, bad USB/Eth ports, unplugged USB/Eth), or a robot issue (wiring loosening up after being hit all day, code errors, DSC shorts or CAN wiring glitches that prevent driving but leave all other functions running).

Because the robot side of FMS is so simple, network problems are easy for the field crew to spot and deal with.

If you want to be scientific and gather actual data rather than anecdotal evidence, then scouters have to:

1. Ignore driver station positions in their reports (avoid confirmation bias-position can always be looked up later if it's important to you).
2. Note the RSL (solid off or solid on - not blinking) when a robot stops moving.
3. Note the team light (e-Stops are highlighted by the bottom yellow light)
4. Send drive team scouters to inquire at the team pits for the reason for the robot non-movement (only the drivers know, often not the pit crew and certainly never random team members)
5. Use pit scouters to ask how much testing time the robot had before the event. Face it, the vast majority of us build a robot in 6 weeks and collision test it only at the competition - that's why teams are shaking bugs out all through Championships.
6. The absolute best is to inspect the team Driver Station logs, not that teams are likely to let you look at them, but you can check your own. The robot is actually providing the data, so any loss of comms is pretty obvious, because the log goes blank. If it comes back again, then your robot rebooted.
7. Volunteer for field duty - they keep logs on all robot failures.

This is a great post. I can't count the number of times I saw a robot not moving and started looking for one of my inspectors to send back to their pit, only to see that they were actually disabled by the field. Of course, that only meant I had to ask the head ref/FTA why they were disabled and send an inspector back anyways to help them avoid that in the future...

[AnonymousMarvin]

I agree most of the issues by robots were probably due to team, rather than FMS error. I would also like to mention though that even though most of the errors happened to robots, there were plenty of errors on the parts of the Referee's at Colorado. From what I could tell, and the general consensus I have found is that the Referees waited till the end of the match to score or log fouls. This caused some major issues in Refs signaling for fouls during the match and then not logging them at the end of the match. The most obvious case happened in SF 2-1, nothing against these teams just an observation, 2996 had team 4153 pinned against the one point goal the entire match. 2996 was halfway into 4153's robot and could not back out. The ref on the field closest to the goal counted off the foul, waved her flag, and signaled for the foul. Then the head ref came over and counted off an additional foul for a continuous pin. Neither foul was counted at the end of the Match, and cost 1987, 1619, and 4153 the match. Again no offense to 2996, 1410, and 662 but that was the clear cut definition of a pin, the refs called it on the field during the match, but failed to score it, to me that is a huge error, and think that it should be considered that fouls have to be logged the instance that they are called.

[RallyJeff]

Originally Posted by AnonymousMarvin

I agree most of the issues by robots were probably due to team, rather than FMS error. I would also like to mention though that even though most of the errors happened to robots, there were plenty of errors on the parts of the Referee's at Colorado. From what I could tell, and the general consensus I have found is that the Referees waited till the end of the match to score or log fouls. This caused some major issues in Refs signaling for fouls during the match and then not logging them at the end of the match. The most obvious case happened in SF 2-1, nothing against these teams just an observation, 2996 had team 4153 pinned against the one point goal the entire match. 2996 was halfway into 4153's robot and could not back out. The ref on the field closest to the goal counted off the foul, waved her flag, and signaled for the foul. Then the head ref came over and counted off an additional foul for a continuous pin. Neither foul was counted at the end of the Match, and cost 1987, 1619, and 4153 the match. Again no offense to 2996, 1410, and 662 but that was the clear cut definition of a pin, the refs called it on the field during the match, but failed to score it, to me that is a huge error, and think that it should be considered that fouls have to be logged the instance that they are called.

FYI (without speaking to specific matches or events): the way the ref tablet works this year, the ref has to switch away from the goal/assist page to a foul page to enter fouls. With this switching, it means that when a ref is entering a foul, there's a 5 to 10 second period when he or she can't enter a goal. We want to make sure we enter goals as soon as they happen (since otherwise, the pedestal doesn't light up when it should), so refs will often wait until the area around their goal is clear before entering a foul... but I've seen matches where it takes more than a minute for this to happen.

Alternatively, the ref who flagged the foul might radio another ref at the less-busy end of the field to enter it into the system, but at a noisy event, radio communications can be difficult.

[JohnSchneider]

Originally Posted by AnonymousMarvin

I agree most of the issues by robots were probably due to team, rather than FMS error. I would also like to mention though that even though most of the errors happened to robots, there were plenty of errors on the parts of the Referee's at Colorado. From what I could tell, and the general consensus I have found is that the Referees waited till the end of the match to score or log fouls. This caused some major issues in Refs signaling for fouls during the match and then not logging them at the end of the match. The most obvious case happened in SF 2-1, nothing against these teams just an observation, 2996 had team 4153 pinned against the one point goal the entire match. 2996 was halfway into 4153's robot and could not back out. The ref on the field closest to the goal counted off the foul, waved her flag, and signaled for the foul. Then the head ref came over and counted off an additional foul for a continuous pin. Neither foul was counted at the end of the Match, and cost 1987, 1619, and 4153 the match. Again no offense to 2996, 1410, and 662 but that was the clear cut definition of a pin, the refs called it on the field during the match, but failed to score it, to me that is a huge error, and think that it should be considered that fouls have to be logged the instance that they are called.

No it was not the "clear cut definition of a pin", as 2996 was pushed into the pin by 1619. They were not at fault directly and so neither team should receive a foul under mutual cause. The refs are instructed "When in doubt throw the flag", because they can always take a foul back after discussion. After the match the refs had a conversation and obviously concluded that the pin was not directly 2996's fault and therefore no foul.

http://youtu.be/HQdVzEqjz7E?t=47s

[AnonymousMarvin]

Thanks for the clarification to both, RallyJeff and animenerdjohn, both posts were very helpful. Particularly animenerdjohn because, even though I watched the match live, and several times on video, didn't consider that 1619 forced them into the foul, I will check on that again, and if that is the reason then I at least understand the referee's decision better. Not that I agree, but I will understand. I still believe that the system of refereeing could have been better. At several other regionals, the fouls were logged on the spot, and then were also explained after the match, even if the fouled was called back after the match they would still give the reason for the foul. Just an observation. Thanks for the feedback.

[AnonymousMarvin]

I just watched the match in question again, SF1-1, and I see that 1619 did indeed push 2996, however, I still believe that a foul could have been called for a pin, because of the fact that 2996 put themselves in that position to be pushed into 4153. Again thanks to animenerdjohn for your help, with this. I find it interesting though that no announcements were made about fouls. Would have been helpful for those of us watching the eliminations.