[FVC]: Analysis Shows Improvement Possible in Ranking System

[gblake]

Ahhhh - A debate worth sinking one's teeth into

Originally Posted by Lil' Lavery

There is one major kink in your analysis, it negates defense (especially in the points scored sector).

It doesn't negate defense. It does ignore the effect defense can have on match scores.

Bill’s “model” (in the mathematical sense) for what an FVC robot does is purposefully one-dimensional. Any model is imperfect. This one has a substantial imperfection, in general, but I have yet to see anyone demonstrate that that imperfection is important in this particular discussion.

Because in this sort of analysis one can play the part of a god and can know the proper ranking of every robot in the simulation, Bill chose to make the ranking metric simple. This allowed the simulation to clearly focus on whether a few rounds of 4-team, 3-match, mini-tournaments would create a ranking that is close to the correct one (known by the god-like analyst who set up the experiment).

His results are that if you want to employ a system that lets you discover what that correct ranking is, his system of having randomly chosen groups of 4 teams stick together for three quick matches appears to figure out the correct rankings faster than randomly selecting a totally new group of four teams for each and every match.

And… he has a body of results that can be (and should be) reproduced. No amount of anyone's opinions or arm-waving can make his results untrue. If anyone wants to reject them they need to choose one of these options and then do the math.

1) Devise a simulation that includes the effects of defense and show that Bill’s hypothetical match scheduling method does not reveal true rankings as well as the new-teams-every-single-match method(s), once defense has been included. I personally do not think that this is a foregone conclusion.

2) Show that one of his assumptions, inputs or formulae is off-target in a way that makes his results inaccurate.

3) Something else equally rigorous.

Originally Posted by Lil' Lavery

If you don't think defense has an impact during a FVC event, you've apparently never been to an FVC event.

You don't want to go there. Trust me - Bill has seen an FVC event.

Originally Posted by Lil' Lavery

In your over-all ranking analysis, you can claim that the "baseline score" is not actually points scored, but a ranking on ability, but I find that claim hard to support, and even harder to accurately quantify, especially considering the ability of the defender is also heavily dependent on the offensive robot.

Read it again. You are looking through the wrong end of the telescope.

Because Bill (or whoever duplicates/checks his very repeatable work) sets up the experiment, before the simulation runs he/they can perfectly sort/rank to the simulated robots according to their scoring ability. He knows exactly how good each robot is (using scoring ability as his metric) because he assigned those scoring abilities. The simulated tournament’s job, much like the jobs of a real tournament, is to attempt to (re)discover what that perfect ranking is.

Originally Posted by Lil' Lavery

In addition, I have a question. How were the groupings formed? If it was randomly, it still has a large chance of dropping a well-qualified team to the bottom of the rankings, as the four best (or well-qualified) teams have a chance to all be in the same grouping. In that scenario you could see one of the best dropped to the bottom, and/or multiple (if not all four) lodged in the middle of the rankings. The inverse applies as well, as it can still propel a sub-par robot to the top. I don't see how that would solve much (even if it slightly reduces it).

Monte Carlo simulations run the same problem over and over again, scrambling (in appropriate ways) the initial conditions and inputs. At the end of the simulation (limited by the fidelity of your simulation’s models) you have a fairly good picture of how well the algorithm you are testing will perform over the long-haul.

Pointing out a few situations where the algorithm works poorly is not adequate evidence that the algorithm isn't useful. I dare say that current methods and other alternatives all fail in some unusual situations. What is more important is how well they perform in most situations, and/or whether any of the algorithms being weighed produce completely unacceptable results in any pathological situations.

Originally Posted by Lil' Lavery

A large part of FIRST (imo, although I know many who support my claim) is being able to play against every (or many) different teams and robots. I wouldn't want to interact with the same three teams the entire time.

Bill does not recommend interacting with the same three teams all the time.

Bill suggests a quick 3-match mini-tournament for a first group of four FVC teams; followed by another 3-match mini-tournament for a different group of four teams, followed by …. until all teams have played in a mini-tournament. Then you scramble the teams and do it again... until time runs out for the entire tournament. In each/any mini-tournament the teams would be matched up like this:
Match 1 AB vs CD
Match 2 AC vs BD
Match 3 AD vs BC
You can see that in any given mini-tournament, a team never has the same ally more than once and faces an ever changing opposing alliance in the three matches (played in rapid succession).

Originally Posted by Lil' Lavery

This change would not be well received by the FIRST community as a whole. It has had little exposure and reaction because of the particular time of the year and sub-forum it is in.

Bill has not necessarily suggested it to the FIRST community as a whole. The strong implication of his original message is that FVC tournaments might benefit from using it. Even if he had recommended it to all of FRC, FVC, FLL, JFLL; I think it is a bit presumptuous for you to assert that “the FIRST community” would dislike a method that could be proven to generally produce more accurate “top 8”s than current methods (and that is likely to be no worse than current methods in extreme situations).

PS: The results of the math will not change if they are posted at some other time or under a more high-falootin heading.

Originally Posted by Lil' Lavery

But if you look at the similar situation from the Week 1 FRC regionals, you will see how the FRC community reacted. At the NASA/VCU, Pacific Northwest, and Granite State regionals each team had a "partner" they competed against in every qualification match. My team, FRC 116, was included, as we competed against FRC 122 in all eight qualification matches. The reception by the teams involved, and over-all FRC community was almost entirely negative. St. Louis and New Jersey also had similar situations that weekend, but not 100% (same reactions though).
http://www.chiefdelphi.com/forums/sh...5178&highlight

As you may remember, I was there with you at NASA/VCU and I agree that that match scheduling method was a mistake (http://www.chiefdelphi.com/forums/sh...&postcount=55). This one is very different. Also a pure application of this one might not be well-suited for use with 3-team FRC alliances. I don’t think Bill suggested using it in an FRC context.

Originally Posted by Lil' Lavery

In closing. FIRST will not accept a schedule that isn't random, and for good reason.

That’s an interesting opinion you have there; but I’m not sure which schedule you are talking about. Bill’s thought experiment is chock full of randomness.

And... the last I checked, neither you nor I nor anyone else was authorized to speak for all of the participants in FIRST activities or for all of the folks who organize and run the FIRST corporation.

Originally Posted by Lil' Lavery

In addition it would be next to impossible to impose a ranking criteria based on individual team performance (and unfair if it does not include some form of defensive ranking ability), nor would all of FIRST support it.

If I understand it correctly, Bill’s approach does not attempt to do this. Bill assigns the alliance’s score to the robots in the alliance. He uses a running total of these points to rank the robots at the end of a full tournament. He does this instead of assigning 2 qual points to each robot on a winning alliance.

Blake
PS: My large font "Bravo!" reply to Bill's original message was intended to commend the methods he employed. Whether the results stand up in a more sophisticated analysis is beside the point. Well-written presentations of results obtained through proper use of the scientific method, computer science and math should be applauded.