**FIRST EMAIL**/Looking for Beta Test Teams

[Joe Ross]

Greetings Teams,

As FRC prepares for the 2010 season, we are seeking a limited number of teams to
assist us by beta testing new software and hardware elements of the control system.
Interested teams should complete the Beta Test Application available here
http://usfirst.org/forms.aspx?ekfrm=14742 by Monday, August 24th at noon EST.

BETA TEST GOALS

* Test the features and functions of new elements of the control system to
uncover problems and generate solution suggestions
* Develop/refine software libraries, supporting documentation and training materials
* Allow teams to become experts and serve as area leaders to mentor other teams

TEAM SELECTION CRITERIA
* COMMUNITY
o Teams must demonstrate consistent involvement within the FIRST community. A
team's communication network should be in place and will be utilized in this
project.
* VISION
o Teams must prove they 'get' the bigger picture and vision of FIRST and must be
prepared to work collaboratively for the betterment of the FRC program.
* GEOGRAPHICAL LOCATION
o FIRST will consider the locations of all teams that apply in an effort to gather
diverse feedback and to create experts in as many locations as possible for future
mentoring.

REQUIRED BETA TEST TASKS
* Complete beta test tasks as assigned
* Post findings on the FIRST Forums on a bi-weekly basis

* Be available to answer questions from area teams and the broader FRC community
* Release every piece of code developed
* Agree to return beta test hardware to FIRST immediately upon request.

Selected teams will be notified by September 4, 2009 and beta testing will begin
after FIRST receives a signed Non Disclosure Agreement from the beta test team.
Please direct questions regarding submissions to FRCbetatest@usfirst.org.

Go Teams!

[Jack Jones]

I wonder about the need for an NDA. Will the beta test teams have a leg up on the rest of the field all the way up until kick-off? If it's not a major advantage, then why not make a full disclosure on the control system at the same time as the beta sites get it?

Many teams were unhappy with the way it went last year, mostly the ones who could not pay until late and got the controller along with their KoP. Their learning curve coincided with the build. There's a whole lot of difference between having the hardware, and knowing people who do!

[Andrew Schreiber]

Originally Posted by Jack Jones

I wonder about the need for an NDA. Will the beta test teams have a leg up on the rest of the field all the way up until kick-off? If it's not a major advantage, then why not make a full disclosure on the control system at the same time as the beta sites get it?

Many teams were unhappy with the way it went last year, mostly the ones who could not pay until late and got the controller along with their KoP. Their learning curve coincided with the build. There's a whole lot of difference between having the hardware, and knowing people who do!

Realize that the NDA may not mean they can't disclose anything but may mean they can't disclose portions of things. Additionally, this is a BETA, technically speaking this could remain unused this year should FIRST decide that the system is not ready.

Also, would you prefer a world where everyone was essentially beta testing a system during build? I would prefer teams that have resources to research stuff DO and then we can stand on their backs come build season. I have always made a rule that I never reinvent the wheel, if I can use someone else's research and build on it I think that is more useful than redoing what has already been done.

[Chris is me]

Originally Posted by Jack Jones

I wonder about the need for an NDA. Will the beta test teams have a leg up on the rest of the field all the way up until kick-off? If it's not a major advantage, then why not make a full disclosure on the control system at the same time as the beta sites get it?

I can certainly say that my team definitely did not have a significant leg up on anyone else in terms of control system issues last year. The NDA is more of a "Disclosure Agreement" that says how and when you can discuss things about the control system. As soon as we could, our team (and every other beta test team) spread everything they knew far and wide. I'm fairly certain this will happen again, especially with this

* Be available to answer questions from area teams and the broader FRC community
* Release every piece of code developed

Being part of the email.

Realize that the NDA may not mean they can't disclose anything but may mean they can't disclose portions of things. Additionally, this is a BETA, technically speaking this could remain unused this year should FIRST decide that the system is not ready.

This is actually a lot of what happened last year. We had to relearn a lot of things when we got the "real" control system.

[Jack Jones]

Also, would you prefer a world where everyone was essentially beta testing a system during build?

Yes, as opposed to a world where a few had a head start, many did not, and most were somewhere in between.

[Akash Rastogi]

Originally Posted by Jack Jones

Yes, as opposed to a world where a few had a head start, many did not, and most were somewhere in between.

I wouldn't necessarily make that conclusion. There were still Beta Test teams who had more problems with their systems than those who did not Beta Test. In our area at least, the team who was beta testing did a fantastic job of relaying information to other area teams (Thanks once again 103). There were barely any teams in the area who had significant problems nor were there teams who had any valuable advantage during the season. Any advantage they found, they shared.

[Chris is me]

Originally Posted by Jack Jones

Yes, as opposed to a world where a few had a head start, many did not, and most were somewhere in between.

Neither my team nor any other beta test team got a significant head start last year. We kept no secrets and published or presented everything we knew about said systems and started no work that went toward the robot. In fact even if we wanted a head start on programming or learning everything abuot the system, a lot of what was learned was rendered null and void with the release of the final control system. I don't even think we took early delivery on it...

[Jack Jones]

If there was no advantage to knowing beforehand, then why did they need an NDA? Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest? Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

In my opinion, information delayed is information denied. But giving select teams the system early is not the only reason I’m against it. I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

I’m not saying that the beta test was worthless. Who knows, maybe we’d still be trying to finish week one with out it? What I am saying is that should not be unnecessary. Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

OAO

[Akash Rastogi]

But this is exactly what FIRST and NI are trying to do by beta testing, they want to give teams the best product possible.

From what I understand, the way you made your statement it sounds like someone saying that Microsoft doesn't need to beta test Windows 7. Their software engineers should just come up with what they think is the best product and release it. What this would fail at is being evaluated by the consumer beforehand. FIRST and NI did the same thing. The beta testers were all expert "consumers" who made the system better and worked out flaws along with NI instead of FIRST and NI assuming things the teams would like. They made a better product suited for the consumer (the rest of the FRC teams) int he end. I can almost guarantee you that if there was no beta testing and FIRST and NI released the system to all the teams at the same time, there would be too many systems out in the community to get proper feedback and any problems would be chaotic to fix.

To the point- in the end, beta testing by a few creates a better product for the rest of us to use.

[Chris is me]

Originally Posted by Jack Jones

If there was no advantage to knowing beforehand, then why did they need an NDA?

Those are completely unrelated, and again, the NDA doesn't stop teams from revealing anything that gives us an advantage. It's honestly more of an "agreement to disclose at this time".

Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest?

If you don't know, why are you alleging that the NDA is somehow designed to stop you guys from knowing stuff about the game that we get to know?

Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

I don't have it in front of me right now, so I can't say, but no, beta testing is not something that should be done by a team looking for an "edge". The feedback given was very useful to FIRST.

In my opinion, information delayed is information denied.

You still got everything we knew, before the game came out. My team hosted a webcast AND went to Illinois and Indiana to demonstrate all of what we learned. Us having it for a slightly longer amount of time did not give us an edge in any way as it's not like we could do anything with the information about the control system before the game came out. Even if it was the same control system.

But giving select teams the system early is not the only reason I’m against it.

Again, we don't get the system early. We get a different system, which we have to return to FIRST. We get the system at the same time as you.

I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

The NI people aren't "passing the buck". They supplied tons of information to teams, including my own and others during the build season. We're merely trying to make sure as many people as possible learn as much as possible, as efficiently as possible.

I’m not saying that the beta test was worthless. Who knows, maybe we’d still be trying to finish week one with out it? What I am saying is that should not be unnecessary. Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

OAO

We're not trying to "figure out" the control system. The beta test program does not exist to inform 16 FIRST Robotics teams on the control system. It's just that, a test. We have to set up things and find out how they don't work and how to fix them, then we give these details to NI for the final control system. As we've said repeatedly, any knowledge we get, we share, and a lot of it became obsolete anyway...

For "proof" we didn't have a massive advantage that let us crush the competition, if you really want to know, we spent the entire first regional we attended attempting to implement working camera tracking. Our team beta tested the camera. In fact, other than an intake roller, the only failures on the robot were programming related. It's a big part of why my team's IRI record was 4 - 4 and not 5 - 3 or maybe even 6 - 2, actually (It was my fault, by the way. )

It's frankly a little insulting to my and every other beta test team when you make assumptions and implications about our motivation and actions when volunteering to attempt to make the transition to a new control system as smooth as possible. My team worked for 3 months on this project, then webcasted a conference on it, then drove through 3 states just to make sure as many people as possible know what we did about it. We've done everything we could possibly do to ensure we didn't have an advantage (this was before learning we didn't get to use the exact same system), and that has never been my nor any other beta test team's intention when doing this program.

PS: 1714's applying again.

[EricVanWyk]

I agree that it should be called a Disclosure Agreement, rather than a Non Disclosure Agreement.

The only things that the Beta teams did not disclose was contact information for developers and FIRST employees.

[Alan Anderson]

Originally Posted by Jack Jones

Was the NDA in place to keep them from misinforming the rest?

That was a small but significant part of it. The distribution of information from beta testing wasn't so much restricted as it was regulated. Things were changing constantly, so immediate disclosure of small details could easily be completely irrelevant to the non-testing teams. A big part of what came out of the beta test process was the documentation provided with the FRC control system, with corrections and clarifications incorporated.

But giving select teams the system early is not the only reason I’m against it. I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

The experts who designed the system shouldn't be the ones to find bugs in the actual environment. That task should fall to people who work in the actual environment -- in other words, beta testers.

Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

The game is tested, debugged, and tweaked a lot before kickoff by people who didn't invent it. The control system should be no different.

[Andrew Schreiber]

Originally Posted by Jack Jones

I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

Jack, I am not sure what your expertise is but I can be reasonably sure it isn't in software or hardware design based on the utter naivety of this statement. Writing any non-trivial piece of software is a difficult process, finding all the bugs in it on your own is near impossible. Not only is it impossible to test all possible scenarios it is impossible to know how the software will be used. Note that I said IMPOSSIBLE. The reason we have bugs in code isn't because the people who created were lazy and didn't test their code but because they did the best they could to test all scenarios and missed some. The purpose of a beta test is to increase the chance that the majority of the bugs will have been found and fixed (or at least noted with a workaround) Under no circumstances can any non-trivial program be perfect. The same concepts apply to hardware.

I apologize if this seems harsh but I know several of the people at NI who work with FIRST, they are passionate and genuinely want to help us all out. I view your comments as slander against these hard working folks and do not feel that is appropriate.

[Zflash]

Unfortuantley this world is not able to give an equal opportunity to us all. So not all teams will become Beta Testers. That is why it is imperative for teams that do become one to go out and help the "have nots" become the "haves" as quick as posssible. Now if all of us outside Michigan could figure out how to get the deal they have we would be set.

[JaneYoung]

I thought the opportunities provided by and through the beta test teams were no less than awesome and I'm glad to see this initiative being made available again this year. Innovation doesn't always come along in ways we can expect or anticipate and the beta test teams were a great innovative idea, offering an opportunity to kick the tires a few times.

The requirements placed on a beta test team should not be, and to my knowledge, are not taken lightly. Also, the geographical areas make a big difference. Nothing is the same in any given area, including mentor support and availability. Many of the beta test teams help mentor teams that would suffer otherwise. It would be great to have initiatives in place like this for other aspects of team development - to help teams succeed and achieve. Whether they take advantage of the opportunities provided by applying to be a beta test team or by following the findings of the beta test teams is up to the individual teams.

A few may grouse among the many but in the end, all potentially benefit.