**FIRST EMAIL**/Looking for Beta Test Teams

[Jack Jones]

If there was no advantage to knowing beforehand, then why did they need an NDA? Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest? Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

In my opinion, information delayed is information denied. But giving select teams the system early is not the only reason I’m against it. I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

I’m not saying that the beta test was worthless. Who knows, maybe we’d still be trying to finish week one with out it? What I am saying is that should not be unnecessary. Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

OAO

[Akash Rastogi]

But this is exactly what FIRST and NI are trying to do by beta testing, they want to give teams the best product possible.

From what I understand, the way you made your statement it sounds like someone saying that Microsoft doesn't need to beta test Windows 7. Their software engineers should just come up with what they think is the best product and release it. What this would fail at is being evaluated by the consumer beforehand. FIRST and NI did the same thing. The beta testers were all expert "consumers" who made the system better and worked out flaws along with NI instead of FIRST and NI assuming things the teams would like. They made a better product suited for the consumer (the rest of the FRC teams) int he end. I can almost guarantee you that if there was no beta testing and FIRST and NI released the system to all the teams at the same time, there would be too many systems out in the community to get proper feedback and any problems would be chaotic to fix.

To the point- in the end, beta testing by a few creates a better product for the rest of us to use.

[Chris is me]

Originally Posted by Jack Jones

If there was no advantage to knowing beforehand, then why did they need an NDA?

Those are completely unrelated, and again, the NDA doesn't stop teams from revealing anything that gives us an advantage. It's honestly more of an "agreement to disclose at this time".

Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest?

If you don't know, why are you alleging that the NDA is somehow designed to stop you guys from knowing stuff about the game that we get to know?

Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

I don't have it in front of me right now, so I can't say, but no, beta testing is not something that should be done by a team looking for an "edge". The feedback given was very useful to FIRST.

In my opinion, information delayed is information denied.

You still got everything we knew, before the game came out. My team hosted a webcast AND went to Illinois and Indiana to demonstrate all of what we learned. Us having it for a slightly longer amount of time did not give us an edge in any way as it's not like we could do anything with the information about the control system before the game came out. Even if it was the same control system.

But giving select teams the system early is not the only reason I’m against it.

Again, we don't get the system early. We get a different system, which we have to return to FIRST. We get the system at the same time as you.

I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

The NI people aren't "passing the buck". They supplied tons of information to teams, including my own and others during the build season. We're merely trying to make sure as many people as possible learn as much as possible, as efficiently as possible.

I’m not saying that the beta test was worthless. Who knows, maybe we’d still be trying to finish week one with out it? What I am saying is that should not be unnecessary. Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

OAO

We're not trying to "figure out" the control system. The beta test program does not exist to inform 16 FIRST Robotics teams on the control system. It's just that, a test. We have to set up things and find out how they don't work and how to fix them, then we give these details to NI for the final control system. As we've said repeatedly, any knowledge we get, we share, and a lot of it became obsolete anyway...

For "proof" we didn't have a massive advantage that let us crush the competition, if you really want to know, we spent the entire first regional we attended attempting to implement working camera tracking. Our team beta tested the camera. In fact, other than an intake roller, the only failures on the robot were programming related. It's a big part of why my team's IRI record was 4 - 4 and not 5 - 3 or maybe even 6 - 2, actually (It was my fault, by the way. )

It's frankly a little insulting to my and every other beta test team when you make assumptions and implications about our motivation and actions when volunteering to attempt to make the transition to a new control system as smooth as possible. My team worked for 3 months on this project, then webcasted a conference on it, then drove through 3 states just to make sure as many people as possible know what we did about it. We've done everything we could possibly do to ensure we didn't have an advantage (this was before learning we didn't get to use the exact same system), and that has never been my nor any other beta test team's intention when doing this program.

PS: 1714's applying again.

[EricVanWyk]

I agree that it should be called a Disclosure Agreement, rather than a Non Disclosure Agreement.

The only things that the Beta teams did not disclose was contact information for developers and FIRST employees.

[Alan Anderson]

Originally Posted by Jack Jones

Was the NDA in place to keep them from misinforming the rest?

That was a small but significant part of it. The distribution of information from beta testing wasn't so much restricted as it was regulated. Things were changing constantly, so immediate disclosure of small details could easily be completely irrelevant to the non-testing teams. A big part of what came out of the beta test process was the documentation provided with the FRC control system, with corrections and clarifications incorporated.

But giving select teams the system early is not the only reason I’m against it. I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

The experts who designed the system shouldn't be the ones to find bugs in the actual environment. That task should fall to people who work in the actual environment -- in other words, beta testers.

Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

The game is tested, debugged, and tweaked a lot before kickoff by people who didn't invent it. The control system should be no different.

[Andrew Schreiber]

Originally Posted by Jack Jones

I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

Jack, I am not sure what your expertise is but I can be reasonably sure it isn't in software or hardware design based on the utter naivety of this statement. Writing any non-trivial piece of software is a difficult process, finding all the bugs in it on your own is near impossible. Not only is it impossible to test all possible scenarios it is impossible to know how the software will be used. Note that I said IMPOSSIBLE. The reason we have bugs in code isn't because the people who created were lazy and didn't test their code but because they did the best they could to test all scenarios and missed some. The purpose of a beta test is to increase the chance that the majority of the bugs will have been found and fixed (or at least noted with a workaround) Under no circumstances can any non-trivial program be perfect. The same concepts apply to hardware.

I apologize if this seems harsh but I know several of the people at NI who work with FIRST, they are passionate and genuinely want to help us all out. I view your comments as slander against these hard working folks and do not feel that is appropriate.

[Zflash]

Unfortuantley this world is not able to give an equal opportunity to us all. So not all teams will become Beta Testers. That is why it is imperative for teams that do become one to go out and help the "have nots" become the "haves" as quick as posssible. Now if all of us outside Michigan could figure out how to get the deal they have we would be set.

[JaneYoung]

I thought the opportunities provided by and through the beta test teams were no less than awesome and I'm glad to see this initiative being made available again this year. Innovation doesn't always come along in ways we can expect or anticipate and the beta test teams were a great innovative idea, offering an opportunity to kick the tires a few times.

The requirements placed on a beta test team should not be, and to my knowledge, are not taken lightly. Also, the geographical areas make a big difference. Nothing is the same in any given area, including mentor support and availability. Many of the beta test teams help mentor teams that would suffer otherwise. It would be great to have initiatives in place like this for other aspects of team development - to help teams succeed and achieve. Whether they take advantage of the opportunities provided by applying to be a beta test team or by following the findings of the beta test teams is up to the individual teams.

A few may grouse among the many but in the end, all potentially benefit.

[EricH]

Jack, I think it is worth while to remind you that there was an NDA in place last year, too. That did not stop the flow of information; it merely directed it such that teams got the information they needed in a place that could be monitored by all parties.

After the beta test ended, some teams involved were asked to continue their beta work through part of the build season. The reason was so that they could more easily find and eliminate issues that weren't simply in teams' programming. (Not to mention stretch the system further...)

This time around, with the system already somewhat understood, they're probably going to be shooting for some of the capabilities that were not allowed to be used in 2009--the CAN, the Jaguar limit function, and some other similar items. As pointed out earlier, it's a lot easier to pull the plug on a beta program than to come out in mid-February and say, "Due to massive complaints, XYZ is no longer legal." The beta test can find the issues and attempt to resolve them, and failure is certainly an option under the "Fail more often to succeed sooner" doctrine during this time.

I'd rather go into an FRC season with stuff I know will work in actual conditions, rather than "Well, it worked in the lab, and theory says...". Hence, the beta test and the NDA to keep a little bit of a lid on the stuff that works in the lab but not on the field.

[Joe Ross]

Originally Posted by Jack Jones

If there was no advantage to knowing beforehand, then why did they need an NDA? Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest? Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

You can read the NDA from the previous beta test here: http://www.usfirst.org/uploadedFiles...st-Rev-0-5.pdf (the last page). In steps 2 and 4, it mentions written consent. That consent is given in section 0.4 of the document, a few pages earlier. As was noted earlier, not much was witheld, but there were guidelines to make sure that all teams had equal opportunity to the information discovered and the code developed.

Now you don't have to speculate what was allowed or not allowed.