A Thank You and a Complaint

I’m currently watching the Central Illinois Regional on stream, and in match 58 the FTA allowed 2338 a pretty significant amount of time to figure out pre-match field connection issues. In addition to the FTA, there were multiple other volunteers out there helping where they could. I’d like to thank the FTA for going out of his way to make sure the match would be fair for all teams.

When I was a student, there was nothing more demoralizing than losing a match because your robot, or a partner’s robot, didn’t connect to the field. It still felt bad when you won against an alliance with a disconnected robot- it doesn’t feel like you earned the win. It was even worse to find out it was a problem which would have needed no more than 5-ish minutes to fix, and could easily have been fixed on the field. I probably saw this happen a dozen times or more in my 4 years as a student, and I know I have seen it since.

I understand that we don’t want to get too far behind, but it’s not like this kind of problem occurs on a regular basis. In a broad sense, the goal of the FRC program is to provide personal fulfillment to the students through a high-octane engineering competition. I feel that this purpose is diminished when the desire to stay on schedule prevails over the desire to provide all teams with an even playing field.

Win win solution, make every team understand the incredible importance of getting inspected early and into the practice field. The very special feature you’re working on doesn’t matter if your robot is disconnected. More functioning teams gets a smoother schedule and more leeway to help the teams that need it without going off schedule too much.

I do agree that doing inspections early is good, and it’s important to get to at least one practice match to make sure you can at least move.

In your opinion, what makes a team qualify for “needing help,” if not the teams I was discussing? More often than not, a team that misses all or most of their practice matches is working on a feature that’s mission-critical. The teams who need help are the same teams who aren’t going to pass inspection in a timely fashion, and will miss most/all of their practice matches. That fact doesn’t change even if pretty much everyone gets on the field before quals. Yes, teams should obviously be asking for help when they need it, but an RI who’s busy inspecting robots isn’t going to help a team put their drivetrain together, nor are they going to spend an hour and a half helping a team find the short circuit in the rats nest of wires in their robot.

Not to mention, comms issues can still crop up on well-made robots which were inspected early. Sometimes, symptoms of an issue don’t arise until multiple matches in, despite everything working fine in practice matches. People getting inspected faster/going to their practice matches might help pick out wiring problems earlier, but it does nothing to solve the on-again-off-again comms issues that have historically affected plenty of teams who *do *get inspected early and *do *go to their practice matches. This was the issue I was getting at; apologies for the lack of clarity.

Because inspection is mandatory, but practice match attendance is not, I think FIRST could look into improving the electrical portion of inspections to be more proactive in addressing comms problems. I don’t recall an RI ever checking to see if the ethernet cable connecting the RIO to the radio had the locking clips on both ends. I also know that many RIs don’t suggest that teams secure the power cable to the radio, even though a foot of electrical tape goes miles to prevent the cable from coming loose. For a more ambitious idea, an emulated FMS kept at the weighing station could be used to quickly see if the robot will connect. This is no replacement for a practice match, but it would be a great start.

I’m gonna throw this in as an aside:

I don’t appreciate the “very special feature” comment. It doesn’t add anything to your argument that I couldn’t otherwise understand, and it just sounds condescending. I don’t want to be the grammar police, but it’s very easy to make that statement without the condescending tone. I try to avoid semantics here because it helps promote reasonable and objective discussion, and it hinders the haphazard slinging of inflammatory rhetoric that silently plagues this website. I ask you to do the same.

This is a perfect reply!

Getting into the practice field is more important than you’d think it is. We use it to get in and make sure we can connect, and just get used to driving for the new event so when quals start we are as ready as possible.

1836 also had weird connection issues at AZN delaying start times and causing us to die in-match in our first and second matches. While more teams getting on practice matches would be nice, it’s not an all-fixing solution. IIRC, we were in 6-8 practice matches at AZN with no communication issues. After our second quals match, we replaced the radio and had no communications problems for the rest of the event. The team still doesn’t quite understand why those issues happened though we’re swapping our radio for a new one at OCR. Huge props to Nate and the other volunteers at AZN for helping us diagnose these issues and giving us that time to connect (and at least run part of auto) in those first two matches.

FTA’s are awesome.

Having spent a considerable amount of time helping teams connect to the field, I’ll share a bit of a counterpoint.

FTAs, FTAAs, CSAs, RIs, and even Refs helping robots connect while the key volunteers are busy answering question box conversations all WANT to see all 6 robots connect, every time. It absolutely sucks to bypass a robot.

It’s rare that the fix is a known five minute fix. If it was a known fix, it’d already be fixed. You suggest it’s condescending to claim teams are in the back working on a special feature instead of basic functionality. It’s similarly condescending to assume the fix is trivial and the staff are lacking empathy. I won’t say that never happens. But, it’s the vast minority of the examples you’re pointing to.

Context matters. If the morning runs more than 60 min late, you’re essentially skipping lunch for all of the field crew. That’s how the crew gets time back. Once they’ve spent time struggling to connect and fail, you’re essentially asking them to skip lunch to ensure the team gets to connect. You’re also asking everyone else there (including the 5 successful teams) to stand around anxiously. There are pros and cons to getting the team connected. You’re focusing solely on the pros.

If the schedule is ahead, it’s actually beneficial to spend extra time on that robot. Running 10-15 minutes behind, as counter-intuitive as it may sound, is optimal. Depending on the situation, it might make more sense to spend a few extra minutes, even if you think they’ll prove fruitless, to put the event on a better schedule. If you’re an hour behind, that becomes less feasible.

As a perfect example, take Alamo two years ago. That event ran so long that teams were in danger of missing travel back home. Teams that SHOULD have spent time in the playoff rounds as backup bots weren’t in the arena because they were hurrying to the airport. Would it be fair to them to push them late enough they have to decide between missing a flight and taking part in their last match?

Ideally, we agree. Every team should get a chance to connect for every match. Field staff should do their best to ensure this happens, even if that means a little extra latitude. In practice, it’s not always that easy and it’s an unfortunate thing.

To clarify, I was discussing the teams you were discussing. Being ahead of schedule means you have the time to spare to connect that robot with comms issues, whether that robot went to the practice field or not.

I want to preface by saying that I come from a district, and your experience may vary if you go to regionals.

Firstly, the idea that teams are plagued by communication issues that are unfixable and which FTAs or other field personnel don’t help with is a myth. The most frustrating thing as a field volunteer is when you tell a team exactly what it is that is causing the problem and how to fix it, the team tells you that they understand, and then they come back with the same issue their next match. This happens with at least 6 teams at every single event I have ever been to. When these teams come back match after match with the same issues, the FTAs will slowly stop allowing them to delay matches.

I would strongly challenge this assertion. The main reasons I see teams miss practice matches are that they aren’t inspected, they want to go home, or their programmers have stolen their robot in an attempt to code an autonomous routine.

Yes, teams should obviously be asking for help when they need it, but an RI who’s busy inspecting robots isn’t going to help a team put their drivetrain together, nor are they going to spend an hour and a half helping a team find the short circuit in the rats nest of wires in their robot.

Precisely the opposite should happen. After the initial rush of inspections in the two hours after load-in, RIs are fully expected to help teams with those things. The rat’s nest of wire leads to so many communication issues down the road that RIs should be working with teams to fix the issue.

Because inspection is mandatory, but practice match attendance is not, I think FIRST could look into improving the electrical portion of inspections to be more proactive in addressing comms problems. I don’t recall an RI ever checking to see if the ethernet cable connecting the RIO to the radio had the locking clips on both ends. I also know that many RIs don’t suggest that teams secure the power cable to the radio, even though a foot of electrical tape goes miles to prevent the cable from coming loose.

A good RI should be doing all of these things.

For a more ambitious idea, an emulated FMS kept at the weighing station could be used to quickly see if the robot will connect. This is no replacement for a practice match, but it would be a great start.

This will not happen ever.

At regionals, the period after practice matches, before pit closing, is for precisely this purpose.

Do you have reasons why?

Your post reads like “we’re fine the way we are, no need to change”, but the results suggest that the system does need some change.

Let’s start with there being no known FMS emulator. There’s FMS… and FMS Off-season, which has fewer bells and whistles.

And any wireless emulator may cause interference with the real FMS.

Now add in budget and transportation for said emulator.

Can it be done? Yes.
Should it be done? I think that that’s something to be considered carefully. Root Cause Analysis of the “failure to connect” issues could show some surprising stuff…

At the St Louis Regional, on the Thursday we had quite a few practice matches and did not have any connection issues. On Friday, we were Qualifying match number 3 and we were not able to connect to the field with the same Classmate and same robot (no code changes). The FTA wanted us to sit out the match which was unacceptable, since we were paying to compete. We brought over a second Classmate and were able to connect. 2 matches later, with the replacement PC we were once again not able to connect. We brought in a third PC and were able to connect and complete our scheduled matches. Their appears to be communication issues with the fields and the FTA do not seem willing to admit it.

I has a CSA in the pit later, and we could not find an explanation for the connection issues. It seems strange that you can connect one day and then the next day not be able to connect. We made sure our robot was passed inspection and we got in the practice matches to confirm that we could connect. I can see the need for keeping to the schedule, but a team that follows FIRST rules, using FIRST software should get some leeway into resolving the issue.

Does the FTA have some sort of debugging tools for communication? If a team has had communication issues, why doesn’t the FTA visit with that team when they are not on the field to assist with the communication issue. That would not take time away from the matches.

Meeting in the pits is the CSA’s job. The FTA must be on the field in order for any matches to run.

The CSA needs should have some debugging tools. After the match, in the pit, we were unable to connect the Classmate that did not work with the robot without setting a static IP.

On the field, there would be a layer of software between the DS and the robot’s radio that we would not have in the pits, and since that software has the ability to change settings on the DS, the CSA should have access to some sort of emulator that would allow them to see possible issues with connection.

At this point the FMS hardware is pretty complicated and I’m not going to pretend like I understand it all. But I suspect that an “emulator” is much easier said than done, especially in a way that doesn’t interfere with the main field.

I agree, the emulator would be complicated. I think the FTA should then work with the teams on the field, especially when they were able to connect the previous day. Add some down time in the schedule for these types of unexpected issues.

We followed the rules and made sure we could connect to the field, we should not be penalized for possible differences that occur when the competition is going on. Having a Classmate work one match and then stop working 2 matches laters seems to be an issue with the field. Other teams were having connection issues during the competition also.

I mean, a laptop with FMS Off-season could probably do the trick for the setup, but I agree that it’s going to interfere with the main field. Everyone’s better off if teams can get onto the practice field early and test in a match setting.

Perhaps the time between pits opening and opening ceremonies on Day 2 could be used for additional connection tests?

This isn’t strictly related but I did want to post an anecdote from last season.

We were on the field for QF1 at NYC, and we were noticing a 3-4 second lag on our joystick. We brought it to the FTA and said we didn’t know what to do, and were sent away with a CSA. Looking at our DS logs, there was some crazy high packet loss/ping (can’t remember which - likely both), and our alliance partner’s DS logs showed something similar. We (us and the CSA) brought it to the FTA and were told that the reason was that the field’s bandwidth was maxed out that match. Having programmed our camera setup, I knew that we never exceeded 3Mbits/s, and was confused why that should be our problem to bear. It turned out someone on the opposing alliance had three cameras running simultaneously, and was destroying the bandwidth for everyone else. Eventually, after some back and forth, we got a replay.

The point I’m trying to bring to the table here is that FTAs, while not in the pits, will work closely with CSAs to make sure things go smoothly. Sadly, given the complexity of the system, the FMS can get hairy to debug, and I certainly wouldn’t mind seeing it overhauled (easier said than done, I know).

I’m sorry your team was having trouble getting connected at your event. There’s a lot of flexibility in the configuration of the driver station in FRC, and with that flexibility can come system integration issues.

There’s been a huge increase in the availability of help at events for controls issues in the last several years. I look at the corps of CSAs we have available at MN events, and compare it to the year when I was the sole CSA at the Minneapolis double regional. They’re smarter than me too :]

Practice matches are there in part to verify the performance of the robot on the field. When a team decides to instead work on a mission-critical feature, they’re making a risk-reward tradeoff. They might be unhappy later about making that tradeoff, but they should not be surprised if it doesn’t pan out.

These tradeoffs are a common occurrence in engineering and software. We have the same discussions at work. Near a new release of software, the PMs will come with a “mission critical” feature that we MUST HAVE in the new release. Then there’s a discussion about the risk-reward tradeoff, and we decide whether to take the risk. Sometimes the risk pays off, sometimes it doesn’t.

They do. The CSAs and FTAAs receive detailed training on troubleshooting the very issues that you experienced. There’s a limit, though, based on time, and then your option is “use a loaner Classmate or don’t run in the match.”

There’s actually no layer of software between the DS and the radio. Here’s what the FMS Whitepaper has to say:

The Driver Station to Robot Communication is identical to that of a system without the FMS in terms of packets. The only difference is that on an FMS network, the packets are routed through the FMS Field Network, then to the Robot. These packets include control data for your robot, telling it what state it should be in and what the values of the joysticks are. The FMS does not send any packets to your robot.

That’s not to say there aren’t differences between tethered and the field, particularly around how data is routed – but the FMS doesn’t modify or inspect any robot to DS traffic.

Agreed. If the driver station isn’t connecting to the field, it’s nearly always an issue with the laptop. The most common source of issues is configuration changes to the laptop between matches, particularly over lunch or in the evening between competition days.

Getting teams connected is a top priority for FTAs. That said, the farther into an event you go, the less time is available for troubleshooting.

There’s a misperception in this thread that there’s no time allowed for troubleshooting. Here’s what I know about regional scheduling. Practice day has longer match cycle times than “real” matches to allow troubleshooting, and for inspected teams they can go to the filler line and get several opportunities to connect. After practice, there’s time before pits close where teams who haven’t yet connect to come to the field and verify their connection. The FTAs, FTAAs, CSAs and RIs actively seek out teams that haven’t connected to get them to the field.

Our team has a goal every year of being the first team inspected, playing in all of our practice matches, and collecting many frequent flyer miles in the filler line. Changes to the robot need to fit in with that.

On the first day of qualification matches, there’s a latent buffer built in to allow time for troubleshooting. I’ve also seen volunteers stay through lunch and after the awards on Friday to help teams troubleshoot issues like what you describe. By Saturday, the expectation is that the bugs are worked out and there’s less time available.

If you’ve had issues with the driver station I’d recommend looking at the Driver Station Best Practices whitepaper. It has a bunch of useful suggestions to improve the reliability of this part of the control system.

Finally, I see from your profile that you’ve been on CD since 2006, which means you’re probably out of high school now ;>) You should come volunteer as a CSA at an event, and get an inside view of all of the help that’s given to teams at events. Besides lots of troubleshooting experience, you get a badge that lets you watch matches from near the field.

In regards to no FMS emulator: The biggest issue would be setting up and running a whole extra FMS network. Because of how FMS works right now, once your robot connects, you are “locked in.” If, for example, the emulator was set up to accept team 1234, it is very possible that it would interfere with team 1234 connecting to the actual FMS.

With regards to the second part: I don’t think the system is perfect as it is. But I think the problems are different than what most people here on CD complain about and the solutions for those problems are difficult and complex.