**FIRST EMAIL**/Looking for Beta Test Teams

[EricH]

Jack, I think it is worth while to remind you that there was an NDA in place last year, too. That did not stop the flow of information; it merely directed it such that teams got the information they needed in a place that could be monitored by all parties.

After the beta test ended, some teams involved were asked to continue their beta work through part of the build season. The reason was so that they could more easily find and eliminate issues that weren't simply in teams' programming. (Not to mention stretch the system further...)

This time around, with the system already somewhat understood, they're probably going to be shooting for some of the capabilities that were not allowed to be used in 2009--the CAN, the Jaguar limit function, and some other similar items. As pointed out earlier, it's a lot easier to pull the plug on a beta program than to come out in mid-February and say, "Due to massive complaints, XYZ is no longer legal." The beta test can find the issues and attempt to resolve them, and failure is certainly an option under the "Fail more often to succeed sooner" doctrine during this time.

I'd rather go into an FRC season with stuff I know will work in actual conditions, rather than "Well, it worked in the lab, and theory says...". Hence, the beta test and the NDA to keep a little bit of a lid on the stuff that works in the lab but not on the field.

[Joe Ross]

Originally Posted by Jack Jones

If there was no advantage to knowing beforehand, then why did they need an NDA? Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest? Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

You can read the NDA from the previous beta test here: http://www.usfirst.org/uploadedFiles...st-Rev-0-5.pdf (the last page). In steps 2 and 4, it mentions written consent. That consent is given in section 0.4 of the document, a few pages earlier. As was noted earlier, not much was witheld, but there were guidelines to make sure that all teams had equal opportunity to the information discovered and the code developed.

Now you don't have to speculate what was allowed or not allowed.

[Daniel_LaFleur]

Originally Posted by Chris is me

Neither my team nor any other beta test team got a significant head start last year.

Please quantify this with hard evidence.

Specifically:
1. Compare the average win-loss ratio for teams that had beta tested the cRIO vs those who didn't.
2. Compare ratio of teams beta testing the cRIO and winning a regional to those not beta testing a cRIO and winning a regional.

I don't have the numbers in front of me, but I'd bet that those ratios will show a significant advantage to those who did beta test.

[Chris is me]

I'll do that, but I doubt it will prove anything, as the beta test teams are made up of well-established FRC teams and thus likely have experience building robots and are more likely to "have their act together" than Random FRC Team X. All it would prove is that FIRST picked beta test teams that happen to do better than average at regionals than a "typical team". Teams like 67, 1114, 254 do a lot of winning in FRC.

If the beta test teams were drawn randomly out of a hat and there were more of them you could probably get a definitiveish answer, but there really isn't a numerical way to quantify "advantage".

What reason do you have to believe we had an advantage?

An interesting statistic that probably also doesn't mean anything would be the number of teams that had never won regionals before the new control system that were beta test teams this year compared to the number of "new" regional winners, or the number of years since the last regional win for a team, though that doesn't guarantee the control system was what mattered or if the team is better at ball manipulation rather than other objects or that the game design did not influence the team to win or the team did not improve.

[Daniel_LaFleur]

Originally Posted by Chris is me

What reason do you have to believe we had an advantage?

My reasoning for my belief that beta teams have an advantage is based on 'First hand experiance' vs. '3rd hand experiance'.

How many times have you tried to repeat an experiance to someone, only to end up saying "you had to be there". This is part of human nature and the fact that our systems for communication (language, etc) are flawed by our own paradiems (sp?).

Many things that I do naturally (and think nothing of it), others may never do (or conceive of doing). Because of this I may not transmit that information acccurately, properly, or even at all. However, if the others had 'hands on' training then they might have picked up on those small details.

Let me ask you a couple of question:
How many beta teams screwed up setting up the cRIO (downloading, etc) when they got theirs for competition? Probably none since they already had done it previously and knew where the mistakes were.

How many rookie teams screwed up setting up their cRIO (downloading, etcc) when they got theirs? If you read the threads here on CD, there were quite a few ... and those were the ones that knew about CD.

[Andrew Schreiber]

Originally Posted by Daniel_LaFleur

My reasoning for my belief that beta teams have an advantage is based on 'First hand experiance' vs. '3rd hand experiance'.

How many times have you tried to repeat an experiance to someone, only to end up saying "you had to be there". This is part of human nature and the fact that our systems for communication (language, etc) are flawed by our own paradiems (sp?).

Many things that I do naturally (and think nothing of it), others may never do (or conceive of doing). Because of this I may not transmit that information acccurately, properly, or even at all. However, if the others had 'hands on' training then they might have picked up on those small details.

Let me ask you a couple of question:
How many beta teams screwed up setting up the cRIO (downloading, etc) when they got theirs for competition? Probably none since they already had done it previously and knew where the mistakes were.

How many rookie teams screwed up setting up their cRIO (downloading, etcc) when they got theirs? If you read the threads here on CD, there were quite a few ... and those were the ones that knew about CD.

Your insinuation that somehow the beta teams had a leg up on the competition because they had a leg up on the control system is a pointless conversation to have. Teams chosen for beta testing were chosen because they would have a leg up to begin with. This may seem counter intuitive, giving the teams that would have succeeded anyway and forcing the teams to help other teams through the disclosure agreement made sense.

The data you are asking is quite easy for Michigan, for example, 67 WAS a beta test team. However, correlating their early experience with the control system to their success this year is simply impossible. In the real world there are too many variables from year to year to say that any one factor contributed to a given team's success. Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team. Naturally these teams will be ones that have a marked history of winning events. To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.

[Andrew Schuetze]

Just to add a bit more clarification on the NDA, it was important that early code and libraries did not go out because many of them contained errors and bugs. During the 6 - 8 week window that we were developing code, our code was broken multiple times by updates sent to us. So the code that was working somewhat on Tuesday was non-functional on Thursday because of a needed fix to the libraries ... If all the beta test teams went about sending out libraries and code every other week, some non-beta test team might end up using that broken system for the competition and be in worse shape then if we all waited until a stable platform was released by FIRST.
As far as a leg up goes, yes we were well versed in the process for downloading firmware, code, resetting the cRIO becuase we did so way too many times during the testing.

However, we were at a disadvantage last season because we invested all our mentor and student efforts into the beta test project and did not do any new student workshops for the team, we reduced our out-reach within the community prior to traveling 2 - 3 hours to multiple out of town workshops to get veteran and rookie teams up to speed in late November and December. (Only Beta test team in Texas yields the following: SA to El Paso 600 miles, SA - Houston 180 miles, SA - Dallas 240 miles, SA - Brownsville 240 miles)

So all in all, some pluses, some minuses, and probably all washed out to some slight net positive. 1600 beta testers in not manangable by an all volunteer staff. Not all 1600 teams would have been able to help improve the process and would have slowed development down. So the line was drawn and it was what it was.

I know several non beta test teams that invested heavily in learning Labview on thier own using Lego NXT robots. So what did we know that a non-beta team didn't. Some of the hardware a bit earlier, some of the libraries a bit early but most of those changed several times and then once more after we had to return the cRIO and equipment.

Hard to be much more definative and IMHO the only course of action to take.

[Daniel_LaFleur]

Originally Posted by Andrew Schreiber

Your insinuation that somehow the beta teams had a leg up on the competition because they had a leg up on the control system is a pointless conversation to have. Teams chosen for beta testing were chosen because they would have a leg up to begin with. This may seem counter intuitive, giving the teams that would have succeeded anyway and forcing the teams to help other teams through the disclosure agreement made sense.

The data you are asking is quite easy for Michigan, for example, 67 WAS a beta test team. However, correlating their early experience with the control system to their success this year is simply impossible. In the real world there are too many variables from year to year to say that any one factor contributed to a given team's success. Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team. Naturally these teams will be ones that have a marked history of winning events. To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.

The tone of your post suggests that you are getting defensive. I did not mean to offend anyone or insinuate that the success of the beta testers was soley due to being a beta tester.

However,

Originally Posted by Chris is me

Neither my team nor any other beta test team got a significant head start last year.

The above quote does not take into account that the beta testers had 6 weeks of familiarity with the cRIO and thus did not have to learn the basics from scratch just before build season. Remember: many rookie teams may not know where the FIRST forums are ... nevermind CD.

Also, suggesting that reading a blog or post about how something is done is as good as getting your hands on it and experimenting is just not true. You cannot easily replace first hand knowledge with third hand knowledge, it just doesn't translate that well ... especially with technical nuances.

Additionally,

Originally Posted by Andrew Schreiber

Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team

By your own admission, the beta testers were very skilled. By contrast, many of the teams that needed extra time with the cRIO (rookies, small teams, teams without software mentors) were not afforded the time.

Originally Posted by Andrew Schreiber

To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.

Please state where I made this claim.
All I claimed is that they recieved a significant advantage over non-beta teams.

As always, the Above is JMHO.

[Chris is me]

Originally Posted by Daniel_LaFleur

The above quote does not take into account that the beta testers had 6 weeks of familiarity with the cRIO and thus did not have to learn the basics from scratch just before build season. Remember: many rookie teams may not know where the FIRST forums are ... nevermind CD.

Not to be elitist or anything, but if the rookies haven't been looking for help and for the resources commonly available to all teams, why should anyone be surprised when they have issues or are at a disadvantage? Yes we had an advantage over those teams, but so did you.

What I'm trying to say is, given the nature of who is picked to be beta testing, a team beta testing is not much better off knowledge wise than if the same team did not beta test. Teams like 67 / 1114 would definitely do everything they could to learn about what the beta teams are doing.

Regardless of CD or FIRST Forum use, all beta test teams held seminars and Q&As in the real world.

By your own admission, the beta testers were very skilled. By contrast, many of the teams that needed extra time with the cRIO (rookies, small teams, teams without software mentors) were not afforded the time.

The purpose of the beta test program is not entirely "to familiarize a team with an early version of the new control system that is not identical to the final version". We had work to do. Would the same teams that do not use Chief Delphi be able to help give back to FIRST as effectively?

[Akash Rastogi]

[sarcasm] Yes let's give everyone a system before its been tested so they'll all have the same problems, that'll save everyone so much time and frustration.[/sarcasm]

Really now? Its Beta Testing for a reason. Testing, recording, and reporting is ALL beta testing is about. I've worked for engineering companies before over the summer too and we sent out products such as new ecoders, gyros, and potentiometers for them to TEST and REPORT. When the product was released to all manufacturing companies those who didn't test product were happy that in the new iteration of each product, there were NO problems in the device. They didn't complain or argue about another tester company having more experience with the product.

Haven't we always said that FIRST is a microcosm of life? Its not fair, nothing is. Live with it. NI is still a corporation first. We are the consumers; who, if given a faulty product that hasn't been tested within the target audience (oh hey Luminary Micro) will complain and moan and cry about the problems we encountered.

Sorry if that sounded rude or harsh.

[Jared Russell]

I contend that the beta test teams did not have a significant leg up over anybody else. I cite the following reasons:

1. Early on, the beta test teams were working with buggy hardware and software. Many of the things they learned about the nuances of the early control system and API had to be unlearned as updates were made available.

2. The beta test teams did not just get to play with their system for pre-season code prototyping. They had to fulfill specific objectives as a condition of their participation in the beta testing program. Much of their work in these areas was not even used this season (e.g. both 330 and 1114 demonstrated camera tracking code, but neither one wound up using tracking on their Lunacy bots).

3. All of the findings, code, and lessons learned from the beta test teams were made available to all FRC teams. For example, team 103 brought their beta control system to our off season event Ramp Riot, where they gave a presentation and answered questions from a live audience of 100 and an even larger online audience. Before kickoff, I already knew the WPILib API like the back of my hand from diligent studying.

4. The teams selected had the least to gain through early exposure to the cRIO. The programmers and mentors of these elite teams would have very quickly reached an expert level either during or immediately before build season (recall that many teams who were not beta testers still got their control systems early). If you take an experienced software engineer and give him the WPILib API spec, he could crank out good software in a matter of a few days if not hours. We're not solving NP Complete problems in FIRST, folks.

In short, I urge those who criticize the beta testing program or the team selection criteria used by FIRST/NI to see the program in a different light. It was not designed to give any particular team an advantage. It was designed to quickly find and fix bugs and to spread introductory cRIO knowledge to all of FIRST. The teams that were chosen had excellent histories of both technical accomplishment and community mentorship.

We all benefited from their experiences.

[Chris is me]

Originally Posted by Akash Rastogi

When the product was released to all manufacturing companies those who didn't test product were happy that in the new iteration of each product, there were NO problems in the device. They didn't complain or argue about another tester company having more experience with the product.

Though to be fair, those companies are not in a direct sporting competition where 2 minutes decides your fate.

As long as we're debating friendly FIRST style, what alternate solution would others propose? Considering the amount of (completely legitimate) complaining that's happened in previous years with faults in the field system, this seems like a better way to release a better product on time than to not test.

[Daniel_LaFleur]

I reversed your quotes so that I could answer them (i feel) in an appropriate logical succession.

Originally Posted by Chris is me

Considering the amount of (completely legitimate) complaining that's happened in previous years with faults in the field system, this seems like a better way to release a better product on time than to not test.

I completely agree that testing is better than not testing. That is not the issue I am trying to bring up.

The issue I'm bringing up is that the teams with the most experiance, the most support, and the best chances at success were given an advantage (extra time with the cRIO to get an understanding of it's capabilities) while those who really needed the extra time and help got none. I don't believe that that was FIRSTs intent, but it is what happened (IMHO)

Originally Posted by Chris is me

As long as we're debating friendly FIRST style, what alternate solution would others propose?

I would look for outside groups that are friendly to FIRST to do the beta testing. Possibly colleges that give FIRST scholorships but do not sponsor any particular team.

[nathanww]

As long as we're debating friendly FIRST style, what alternate solution would others propose?

Well, depending on the type of beta required, many beta tests could be made open betas. For example, what would a public beta of CAN require? New software libraries and a few cables. FIRST/WPI/whoever would need to provide libraries anyway for a closed beta.

Yes, that does mean more strain on FIRST/WPI to respond to bug reports,provide support, etc. But...that's really what has to happen for the product/system to mature. If 300 teams are emailing because they can't figure out how to configure something, it means the documentation's not clear. If FIRST gets 1000 bug reports for various things, it's a lot more work for them, but as long as they can be prioritized and dealt with, it means that teams AREN'T going to hit those issues during build season.

[Jared Russell]

Originally Posted by nathanww

Well, depending on the type of beta required, many beta tests could be made open betas. For example, what would a public beta of CAN require? New software libraries and a few cables. FIRST/WPI/whoever would need to provide libraries anyway for a closed beta.

Yes, that does mean more strain on FIRST/WPI to respond to bug reports,provide support, etc. But...that's really what has to happen for the product/system to mature. If 300 teams are emailing because they can't figure out how to configure something, it means the documentation's not clear. If FIRST gets 1000 bug reports for various things, it's a lot more work for them, but as long as they can be prioritized and dealt with, it means that teams AREN'T going to hit those issues during build season.

There aren't as many people available to work on the beta test on the FIRST/NI/WPI/Oracle side as you may think. Opening the beta up to 1000 teams is going to result in a lot more work without a lot more results.

From experience I can tell you that inexperienced users make lousy beta testers - their feedback can be vague, contradictory, and all in all detrimental to getting everything working. But if teams 67, 111, and 254 can't figure something out, then there's a good chance that somebody on the development side is going to need to fix something.