**FIRST EMAIL**/Looking for Beta Test Teams

[EricH]

Jack, I think it is worth while to remind you that there was an NDA in place last year, too. That did not stop the flow of information; it merely directed it such that teams got the information they needed in a place that could be monitored by all parties.

After the beta test ended, some teams involved were asked to continue their beta work through part of the build season. The reason was so that they could more easily find and eliminate issues that weren't simply in teams' programming. (Not to mention stretch the system further...)

This time around, with the system already somewhat understood, they're probably going to be shooting for some of the capabilities that were not allowed to be used in 2009--the CAN, the Jaguar limit function, and some other similar items. As pointed out earlier, it's a lot easier to pull the plug on a beta program than to come out in mid-February and say, "Due to massive complaints, XYZ is no longer legal." The beta test can find the issues and attempt to resolve them, and failure is certainly an option under the "Fail more often to succeed sooner" doctrine during this time.

I'd rather go into an FRC season with stuff I know will work in actual conditions, rather than "Well, it worked in the lab, and theory says...". Hence, the beta test and the NDA to keep a little bit of a lid on the stuff that works in the lab but not on the field.

[Joe Ross]

Originally Posted by Jack Jones

If there was no advantage to knowing beforehand, then why did they need an NDA? Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest? Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

You can read the NDA from the previous beta test here: http://www.usfirst.org/uploadedFiles...st-Rev-0-5.pdf (the last page). In steps 2 and 4, it mentions written consent. That consent is given in section 0.4 of the document, a few pages earlier. As was noted earlier, not much was witheld, but there were guidelines to make sure that all teams had equal opportunity to the information discovered and the code developed.

Now you don't have to speculate what was allowed or not allowed.

[Daniel_LaFleur]

Originally Posted by Chris is me

Neither my team nor any other beta test team got a significant head start last year.

Please quantify this with hard evidence.

Specifically:
1. Compare the average win-loss ratio for teams that had beta tested the cRIO vs those who didn't.
2. Compare ratio of teams beta testing the cRIO and winning a regional to those not beta testing a cRIO and winning a regional.

I don't have the numbers in front of me, but I'd bet that those ratios will show a significant advantage to those who did beta test.

[Chris is me]

I'll do that, but I doubt it will prove anything, as the beta test teams are made up of well-established FRC teams and thus likely have experience building robots and are more likely to "have their act together" than Random FRC Team X. All it would prove is that FIRST picked beta test teams that happen to do better than average at regionals than a "typical team". Teams like 67, 1114, 254 do a lot of winning in FRC.

If the beta test teams were drawn randomly out of a hat and there were more of them you could probably get a definitiveish answer, but there really isn't a numerical way to quantify "advantage".

What reason do you have to believe we had an advantage?

An interesting statistic that probably also doesn't mean anything would be the number of teams that had never won regionals before the new control system that were beta test teams this year compared to the number of "new" regional winners, or the number of years since the last regional win for a team, though that doesn't guarantee the control system was what mattered or if the team is better at ball manipulation rather than other objects or that the game design did not influence the team to win or the team did not improve.

[Daniel_LaFleur]

Originally Posted by Chris is me

What reason do you have to believe we had an advantage?

My reasoning for my belief that beta teams have an advantage is based on 'First hand experiance' vs. '3rd hand experiance'.

How many times have you tried to repeat an experiance to someone, only to end up saying "you had to be there". This is part of human nature and the fact that our systems for communication (language, etc) are flawed by our own paradiems (sp?).

Many things that I do naturally (and think nothing of it), others may never do (or conceive of doing). Because of this I may not transmit that information acccurately, properly, or even at all. However, if the others had 'hands on' training then they might have picked up on those small details.

Let me ask you a couple of question:
How many beta teams screwed up setting up the cRIO (downloading, etc) when they got theirs for competition? Probably none since they already had done it previously and knew where the mistakes were.

How many rookie teams screwed up setting up their cRIO (downloading, etcc) when they got theirs? If you read the threads here on CD, there were quite a few ... and those were the ones that knew about CD.

[Andrew Schreiber]

Originally Posted by Daniel_LaFleur

My reasoning for my belief that beta teams have an advantage is based on 'First hand experiance' vs. '3rd hand experiance'.

How many times have you tried to repeat an experiance to someone, only to end up saying "you had to be there". This is part of human nature and the fact that our systems for communication (language, etc) are flawed by our own paradiems (sp?).

Many things that I do naturally (and think nothing of it), others may never do (or conceive of doing). Because of this I may not transmit that information acccurately, properly, or even at all. However, if the others had 'hands on' training then they might have picked up on those small details.

Let me ask you a couple of question:
How many beta teams screwed up setting up the cRIO (downloading, etc) when they got theirs for competition? Probably none since they already had done it previously and knew where the mistakes were.

How many rookie teams screwed up setting up their cRIO (downloading, etcc) when they got theirs? If you read the threads here on CD, there were quite a few ... and those were the ones that knew about CD.

Your insinuation that somehow the beta teams had a leg up on the competition because they had a leg up on the control system is a pointless conversation to have. Teams chosen for beta testing were chosen because they would have a leg up to begin with. This may seem counter intuitive, giving the teams that would have succeeded anyway and forcing the teams to help other teams through the disclosure agreement made sense.

The data you are asking is quite easy for Michigan, for example, 67 WAS a beta test team. However, correlating their early experience with the control system to their success this year is simply impossible. In the real world there are too many variables from year to year to say that any one factor contributed to a given team's success. Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team. Naturally these teams will be ones that have a marked history of winning events. To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.

[Andrew Schuetze]

Just to add a bit more clarification on the NDA, it was important that early code and libraries did not go out because many of them contained errors and bugs. During the 6 - 8 week window that we were developing code, our code was broken multiple times by updates sent to us. So the code that was working somewhat on Tuesday was non-functional on Thursday because of a needed fix to the libraries ... If all the beta test teams went about sending out libraries and code every other week, some non-beta test team might end up using that broken system for the competition and be in worse shape then if we all waited until a stable platform was released by FIRST.
As far as a leg up goes, yes we were well versed in the process for downloading firmware, code, resetting the cRIO becuase we did so way too many times during the testing.

However, we were at a disadvantage last season because we invested all our mentor and student efforts into the beta test project and did not do any new student workshops for the team, we reduced our out-reach within the community prior to traveling 2 - 3 hours to multiple out of town workshops to get veteran and rookie teams up to speed in late November and December. (Only Beta test team in Texas yields the following: SA to El Paso 600 miles, SA - Houston 180 miles, SA - Dallas 240 miles, SA - Brownsville 240 miles)

So all in all, some pluses, some minuses, and probably all washed out to some slight net positive. 1600 beta testers in not manangable by an all volunteer staff. Not all 1600 teams would have been able to help improve the process and would have slowed development down. So the line was drawn and it was what it was.

I know several non beta test teams that invested heavily in learning Labview on thier own using Lego NXT robots. So what did we know that a non-beta team didn't. Some of the hardware a bit earlier, some of the libraries a bit early but most of those changed several times and then once more after we had to return the cRIO and equipment.

Hard to be much more definative and IMHO the only course of action to take.

[Daniel_LaFleur]

Originally Posted by Andrew Schreiber

Your insinuation that somehow the beta teams had a leg up on the competition because they had a leg up on the control system is a pointless conversation to have. Teams chosen for beta testing were chosen because they would have a leg up to begin with. This may seem counter intuitive, giving the teams that would have succeeded anyway and forcing the teams to help other teams through the disclosure agreement made sense.

The data you are asking is quite easy for Michigan, for example, 67 WAS a beta test team. However, correlating their early experience with the control system to their success this year is simply impossible. In the real world there are too many variables from year to year to say that any one factor contributed to a given team's success. Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team. Naturally these teams will be ones that have a marked history of winning events. To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.

The tone of your post suggests that you are getting defensive. I did not mean to offend anyone or insinuate that the success of the beta testers was soley due to being a beta tester.

However,

Originally Posted by Chris is me

Neither my team nor any other beta test team got a significant head start last year.

The above quote does not take into account that the beta testers had 6 weeks of familiarity with the cRIO and thus did not have to learn the basics from scratch just before build season. Remember: many rookie teams may not know where the FIRST forums are ... nevermind CD.

Also, suggesting that reading a blog or post about how something is done is as good as getting your hands on it and experimenting is just not true. You cannot easily replace first hand knowledge with third hand knowledge, it just doesn't translate that well ... especially with technical nuances.

Additionally,

Originally Posted by Andrew Schreiber

Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team

By your own admission, the beta testers were very skilled. By contrast, many of the teams that needed extra time with the cRIO (rookies, small teams, teams without software mentors) were not afforded the time.

Originally Posted by Andrew Schreiber

To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.

Please state where I made this claim.
All I claimed is that they recieved a significant advantage over non-beta teams.

As always, the Above is JMHO.

[Chris is me]

Originally Posted by Daniel_LaFleur

The above quote does not take into account that the beta testers had 6 weeks of familiarity with the cRIO and thus did not have to learn the basics from scratch just before build season. Remember: many rookie teams may not know where the FIRST forums are ... nevermind CD.

Not to be elitist or anything, but if the rookies haven't been looking for help and for the resources commonly available to all teams, why should anyone be surprised when they have issues or are at a disadvantage? Yes we had an advantage over those teams, but so did you.

What I'm trying to say is, given the nature of who is picked to be beta testing, a team beta testing is not much better off knowledge wise than if the same team did not beta test. Teams like 67 / 1114 would definitely do everything they could to learn about what the beta teams are doing.

Regardless of CD or FIRST Forum use, all beta test teams held seminars and Q&As in the real world.

By your own admission, the beta testers were very skilled. By contrast, many of the teams that needed extra time with the cRIO (rookies, small teams, teams without software mentors) were not afforded the time.

The purpose of the beta test program is not entirely "to familiarize a team with an early version of the new control system that is not identical to the final version". We had work to do. Would the same teams that do not use Chief Delphi be able to help give back to FIRST as effectively?