**FIRST EMAIL**/Looking for Beta Test Teams

[Joe Ross]

Greetings Teams,

As FRC prepares for the 2010 season, we are seeking a limited number of teams to
assist us by beta testing new software and hardware elements of the control system.
Interested teams should complete the Beta Test Application available here
http://usfirst.org/forms.aspx?ekfrm=14742 by Monday, August 24th at noon EST.

BETA TEST GOALS

* Test the features and functions of new elements of the control system to
uncover problems and generate solution suggestions
* Develop/refine software libraries, supporting documentation and training materials
* Allow teams to become experts and serve as area leaders to mentor other teams

TEAM SELECTION CRITERIA
* COMMUNITY
o Teams must demonstrate consistent involvement within the FIRST community. A
team's communication network should be in place and will be utilized in this
project.
* VISION
o Teams must prove they 'get' the bigger picture and vision of FIRST and must be
prepared to work collaboratively for the betterment of the FRC program.
* GEOGRAPHICAL LOCATION
o FIRST will consider the locations of all teams that apply in an effort to gather
diverse feedback and to create experts in as many locations as possible for future
mentoring.

REQUIRED BETA TEST TASKS
* Complete beta test tasks as assigned
* Post findings on the FIRST Forums on a bi-weekly basis

* Be available to answer questions from area teams and the broader FRC community
* Release every piece of code developed
* Agree to return beta test hardware to FIRST immediately upon request.

Selected teams will be notified by September 4, 2009 and beta testing will begin
after FIRST receives a signed Non Disclosure Agreement from the beta test team.
Please direct questions regarding submissions to FRCbetatest@usfirst.org.

Go Teams!

[Jack Jones]

I wonder about the need for an NDA. Will the beta test teams have a leg up on the rest of the field all the way up until kick-off? If it's not a major advantage, then why not make a full disclosure on the control system at the same time as the beta sites get it?

Many teams were unhappy with the way it went last year, mostly the ones who could not pay until late and got the controller along with their KoP. Their learning curve coincided with the build. There's a whole lot of difference between having the hardware, and knowing people who do!

[Andrew Schreiber]

I wonder about the need for an NDA. Will the beta test teams have a leg up on the rest of the field all the way up until kick-off? If it's not a major advantage, then why not make a full disclosure on the control system at the same time as the beta sites get it?

Many teams were unhappy with the way it went last year, mostly the ones who could not pay until late and got the controller along with their KoP. Their learning curve coincided with the build. There's a whole lot of difference between having the hardware, and knowing people who do!

Realize that the NDA may not mean they can't disclose anything but may mean they can't disclose portions of things. Additionally, this is a BETA, technically speaking this could remain unused this year should FIRST decide that the system is not ready.

Also, would you prefer a world where everyone was essentially beta testing a system during build? I would prefer teams that have resources to research stuff DO and then we can stand on their backs come build season. I have always made a rule that I never reinvent the wheel, if I can use someone else's research and build on it I think that is more useful than redoing what has already been done.

[Chris is me]

I wonder about the need for an NDA. Will the beta test teams have a leg up on the rest of the field all the way up until kick-off? If it's not a major advantage, then why not make a full disclosure on the control system at the same time as the beta sites get it?

I can certainly say that my team definitely did not have a significant leg up on anyone else in terms of control system issues last year. The NDA is more of a "Disclosure Agreement" that says how and when you can discuss things about the control system. As soon as we could, our team (and every other beta test team) spread everything they knew far and wide. I'm fairly certain this will happen again, especially with this


* Be available to answer questions from area teams and the broader FRC community
* Release every piece of code developed

Being part of the email.

Realize that the NDA may not mean they can't disclose anything but may mean they can't disclose portions of things. Additionally, this is a BETA, technically speaking this could remain unused this year should FIRST decide that the system is not ready.

This is actually a lot of what happened last year. We had to relearn a lot of things when we got the "real" control system.

[Jack Jones]

Also, would you prefer a world where everyone was essentially beta testing a system during build?

Yes, as opposed to a world where a few had a head start, many did not, and most were somewhere in between.

[Akash Rastogi]

Yes, as opposed to a world where a few had a head start, many did not, and most were somewhere in between.

I wouldn't necessarily make that conclusion. There were still Beta Test teams who had more problems with their systems than those who did not Beta Test. In our area at least, the team who was beta testing did a fantastic job of relaying information to other area teams (Thanks once again 103). There were barely any teams in the area who had significant problems nor were there teams who had any valuable advantage during the season. Any advantage they found, they shared.

[Chris is me]

Yes, as opposed to a world where a few had a head start, many did not, and most were somewhere in between.

Neither my team nor any other beta test team got a significant head start last year. We kept no secrets and published or presented everything we knew about said systems and started no work that went toward the robot. In fact even if we wanted a head start on programming or learning everything abuot the system, a lot of what was learned was rendered null and void with the release of the final control system. I don't even think we took early delivery on it...

[Jack Jones]

If there was no advantage to knowing beforehand, then why did they need an NDA? Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest? Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

In my opinion, information delayed is information denied. But giving select teams the system early is not the only reason I’m against it. I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

I’m not saying that the beta test was worthless. Who knows, maybe we’d still be trying to finish week one with out it? What I am saying is that should not be unnecessary. Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

OAO

[Akash Rastogi]

But this is exactly what FIRST and NI are trying to do by beta testing, they want to give teams the best product possible.

From what I understand, the way you made your statement it sounds like someone saying that Microsoft doesn't need to beta test Windows 7. Their software engineers should just come up with what they think is the best product and release it. What this would fail at is being evaluated by the consumer beforehand. FIRST and NI did the same thing. The beta testers were all expert "consumers" who made the system better and worked out flaws along with NI instead of FIRST and NI assuming things the teams would like. They made a better product suited for the consumer (the rest of the FRC teams) int he end. I can almost guarantee you that if there was no beta testing and FIRST and NI released the system to all the teams at the same time, there would be too many systems out in the community to get proper feedback and any problems would be chaotic to fix.

To the point- in the end, beta testing by a few creates a better product for the rest of us to use.

[Chris is me]

If there was no advantage to knowing beforehand, then why did they need an NDA?

Those are completely unrelated, and again, the NDA doesn't stop teams from revealing anything that gives us an advantage. It's honestly more of an "agreement to disclose at this time".

Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest?

If you don't know, why are you alleging that the NDA is somehow designed to stop you guys from knowing stuff about the game that we get to know?

Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

I don't have it in front of me right now, so I can't say, but no, beta testing is not something that should be done by a team looking for an "edge". The feedback given was very useful to FIRST.

In my opinion, information delayed is information denied.

You still got everything we knew, before the game came out. My team hosted a webcast AND went to Illinois and Indiana to demonstrate all of what we learned. Us having it for a slightly longer amount of time did not give us an edge in any way as it's not like we could do anything with the information about the control system before the game came out. Even if it was the same control system.

But giving select teams the system early is not the only reason I’m against it.

Again, we don't get the system early. We get a different system, which we have to return to FIRST. We get the system at the same time as you.

I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

The NI people aren't "passing the buck". They supplied tons of information to teams, including my own and others during the build season. We're merely trying to make sure as many people as possible learn as much as possible, as efficiently as possible.

I’m not saying that the beta test was worthless. Who knows, maybe we’d still be trying to finish week one with out it? What I am saying is that should not be unnecessary. Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

OAO

We're not trying to "figure out" the control system. The beta test program does not exist to inform 16 FIRST Robotics teams on the control system. It's just that, a test. We have to set up things and find out how they don't work and how to fix them, then we give these details to NI for the final control system. As we've said repeatedly, any knowledge we get, we share, and a lot of it became obsolete anyway...

For "proof" we didn't have a massive advantage that let us crush the competition, if you really want to know, we spent the entire first regional we attended attempting to implement working camera tracking. Our team beta tested the camera. In fact, other than an intake roller, the only failures on the robot were programming related. It's a big part of why my team's IRI record was 4 - 4 and not 5 - 3 or maybe even 6 - 2, actually (It was my fault, by the way. :( )

It's frankly a little insulting to my and every other beta test team when you make assumptions and implications about our motivation and actions when volunteering to attempt to make the transition to a new control system as smooth as possible. My team worked for 3 months on this project, then webcasted a conference on it, then drove through 3 states just to make sure as many people as possible know what we did about it. We've done everything we could possibly do to ensure we didn't have an advantage (this was before learning we didn't get to use the exact same system), and that has never been my nor any other beta test team's intention when doing this program.

PS: 1714's applying again.

[EricVanWyk]

I agree that it should be called a Disclosure Agreement, rather than a Non Disclosure Agreement.

The only things that the Beta teams did not disclose was contact information for developers and FIRST employees.

[Alan Anderson]

Was the NDA in place to keep them from misinforming the rest?

That was a small but significant part of it. The distribution of information from beta testing wasn't so much restricted as it was regulated. Things were changing constantly, so immediate disclosure of small details could easily be completely irrelevant to the non-testing teams. A big part of what came out of the beta test process was the documentation provided with the FRC control system, with corrections and clarifications incorporated.

But giving select teams the system early is not the only reason I’m against it. I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.

The experts who designed the system shouldn't be the ones to find bugs in the actual environment. That task should fall to people who work in the actual environment -- in other words, beta testers.

Give us a good system and we’re capable of figuring it out. We do it with the game design, the KoP, and all the rest without knowing before kick-off. The control system should be no different.

The game is tested, debugged, and tweaked a lot before kickoff by people who didn't invent it. The control system should be no different. :)

[Andrew Schreiber]

I’m against it because it’s a very poor replacement for having the right information provided by the experts who designed the system and who’ve work out the bugs in the actual environment.


Jack, I am not sure what your expertise is but I can be reasonably sure it isn't in software or hardware design based on the utter naivety of this statement. Writing any non-trivial piece of software is a difficult process, finding all the bugs in it on your own is near impossible. Not only is it impossible to test all possible scenarios it is impossible to know how the software will be used. Note that I said IMPOSSIBLE. The reason we have bugs in code isn't because the people who created were lazy and didn't test their code but because they did the best they could to test all scenarios and missed some. The purpose of a beta test is to increase the chance that the majority of the bugs will have been found and fixed (or at least noted with a workaround) Under no circumstances can any non-trivial program be perfect. The same concepts apply to hardware.

I apologize if this seems harsh but I know several of the people at NI who work with FIRST, they are passionate and genuinely want to help us all out. I view your comments as slander against these hard working folks and do not feel that is appropriate.

[Zflash]

Unfortuantley this world is not able to give an equal opportunity to us all. So not all teams will become Beta Testers. That is why it is imperative for teams that do become one to go out and help the "have nots" become the "haves" as quick as posssible. Now if all of us outside Michigan could figure out how to get the deal they have we would be set.

[JaneYoung]

I thought the opportunities provided by and through the beta test teams were no less than awesome and I'm glad to see this initiative being made available again this year. Innovation doesn't always come along in ways we can expect or anticipate and the beta test teams were a great innovative idea, offering an opportunity to kick the tires a few times.

The requirements placed on a beta test team should not be, and to my knowledge, are not taken lightly. Also, the geographical areas make a big difference. Nothing is the same in any given area, including mentor support and availability. Many of the beta test teams help mentor teams that would suffer otherwise. It would be great to have initiatives in place like this for other aspects of team development - to help teams succeed and achieve. Whether they take advantage of the opportunities provided by applying to be a beta test team or by following the findings of the beta test teams is up to the individual teams.

A few may grouse among the many but in the end, all potentially benefit.

[EricH]

Jack, I think it is worth while to remind you that there was an NDA in place last year, too. That did not stop the flow of information; it merely directed it such that teams got the information they needed in a place that could be monitored by all parties.

After the beta test ended, some teams involved were asked to continue their beta work through part of the build season. The reason was so that they could more easily find and eliminate issues that weren't simply in teams' programming. (Not to mention stretch the system further...)

This time around, with the system already somewhat understood, they're probably going to be shooting for some of the capabilities that were not allowed to be used in 2009--the CAN, the Jaguar limit function, and some other similar items. As pointed out earlier, it's a lot easier to pull the plug on a beta program than to come out in mid-February and say, "Due to massive complaints, XYZ is no longer legal." The beta test can find the issues and attempt to resolve them, and failure is certainly an option under the "Fail more often to succeed sooner" doctrine during this time.

I'd rather go into an FRC season with stuff I know will work in actual conditions, rather than "Well, it worked in the lab, and theory says...". Hence, the beta test and the NDA to keep a little bit of a lid on the stuff that works in the lab but not on the field.

[Joe Ross]

If there was no advantage to knowing beforehand, then why did they need an NDA? Some make it sound like knowing early was a disadvantage. Was the NDA in place to keep them from misinforming the rest? Was the “when” in the NDA an arbitrary date, or was it when the test team demonstrated proficiency? Saying that the test teams had more problems than others suggests it was the former, and is not much of an endorsement for beta testing.

You can read the NDA from the previous beta test here: http://www.usfirst.org/uploadedFiles/0-FRC_Control_System-Beta-Test-Rev-0-5.pdf (the last page). In steps 2 and 4, it mentions written consent. That consent is given in section 0.4 of the document, a few pages earlier. As was noted earlier, not much was witheld, but there were guidelines to make sure that all teams had equal opportunity to the information discovered and the code developed.

Now you don't have to speculate what was allowed or not allowed.

[Daniel_LaFleur]

Neither my team nor any other beta test team got a significant head start last year.


Please quantify this with hard evidence.

Specifically:
1. Compare the average win-loss ratio for teams that had beta tested the cRIO vs those who didn't.
2. Compare ratio of teams beta testing the cRIO and winning a regional to those not beta testing a cRIO and winning a regional.

I don't have the numbers in front of me, but I'd bet that those ratios will show a significant advantage to those who did beta test.

[Chris is me]

I'll do that, but I doubt it will prove anything, as the beta test teams are made up of well-established FRC teams and thus likely have experience building robots and are more likely to "have their act together" than Random FRC Team X. All it would prove is that FIRST picked beta test teams that happen to do better than average at regionals than a "typical team". Teams like 67, 1114, 254 do a lot of winning in FRC.

If the beta test teams were drawn randomly out of a hat and there were more of them you could probably get a definitiveish answer, but there really isn't a numerical way to quantify "advantage".

What reason do you have to believe we had an advantage?

An interesting statistic that probably also doesn't mean anything would be the number of teams that had never won regionals before the new control system that were beta test teams this year compared to the number of "new" regional winners, or the number of years since the last regional win for a team, though that doesn't guarantee the control system was what mattered or if the team is better at ball manipulation rather than other objects or that the game design did not influence the team to win or the team did not improve.

[Daniel_LaFleur]

What reason do you have to believe we had an advantage?


My reasoning for my belief that beta teams have an advantage is based on 'First hand experiance' vs. '3rd hand experiance'.

How many times have you tried to repeat an experiance to someone, only to end up saying "you had to be there". This is part of human nature and the fact that our systems for communication (language, etc) are flawed by our own paradiems (sp?).

Many things that I do naturally (and think nothing of it), others may never do (or conceive of doing). Because of this I may not transmit that information acccurately, properly, or even at all. However, if the others had 'hands on' training then they might have picked up on those small details.

Let me ask you a couple of question:
How many beta teams screwed up setting up the cRIO (downloading, etc) when they got theirs for competition? Probably none since they already had done it previously and knew where the mistakes were.

How many rookie teams screwed up setting up their cRIO (downloading, etcc) when they got theirs? If you read the threads here on CD, there were quite a few ... and those were the ones that knew about CD.

[Andrew Schreiber]

My reasoning for my belief that beta teams have an advantage is based on 'First hand experiance' vs. '3rd hand experiance'.

How many times have you tried to repeat an experiance to someone, only to end up saying "you had to be there". This is part of human nature and the fact that our systems for communication (language, etc) are flawed by our own paradiems (sp?).

Many things that I do naturally (and think nothing of it), others may never do (or conceive of doing). Because of this I may not transmit that information acccurately, properly, or even at all. However, if the others had 'hands on' training then they might have picked up on those small details.

Let me ask you a couple of question:
How many beta teams screwed up setting up the cRIO (downloading, etc) when they got theirs for competition? Probably none since they already had done it previously and knew where the mistakes were.

How many rookie teams screwed up setting up their cRIO (downloading, etcc) when they got theirs? If you read the threads here on CD, there were quite a few ... and those were the ones that knew about CD.

Your insinuation that somehow the beta teams had a leg up on the competition because they had a leg up on the control system is a pointless conversation to have. Teams chosen for beta testing were chosen because they would have a leg up to begin with. This may seem counter intuitive, giving the teams that would have succeeded anyway and forcing the teams to help other teams through the disclosure agreement made sense.

The data you are asking is quite easy for Michigan, for example, 67 WAS a beta test team. However, correlating their early experience with the control system to their success this year is simply impossible. In the real world there are too many variables from year to year to say that any one factor contributed to a given team's success. Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team. Naturally these teams will be ones that have a marked history of winning events. To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.

[Andrew Schuetze]

Just to add a bit more clarification on the NDA, it was important that early code and libraries did not go out because many of them contained errors and bugs. During the 6 - 8 week window that we were developing code, our code was broken multiple times by updates sent to us. So the code that was working somewhat on Tuesday was non-functional on Thursday because of a needed fix to the libraries ... If all the beta test teams went about sending out libraries and code every other week, some non-beta test team might end up using that broken system for the competition and be in worse shape then if we all waited until a stable platform was released by FIRST.
As far as a leg up goes, yes we were well versed in the process for downloading firmware, code, resetting the cRIO becuase we did so way too many times during the testing.

However, we were at a disadvantage last season because we invested all our mentor and student efforts into the beta test project and did not do any new student workshops for the team, we reduced our out-reach within the community prior to traveling 2 - 3 hours to multiple out of town workshops to get veteran and rookie teams up to speed in late November and December. (Only Beta test team in Texas yields the following: SA to El Paso 600 miles, SA - Houston 180 miles, SA - Dallas 240 miles, SA - Brownsville 240 miles)

So all in all, some pluses, some minuses, and probably all washed out to some slight net positive. 1600 beta testers in not manangable by an all volunteer staff. Not all 1600 teams would have been able to help improve the process and would have slowed development down. So the line was drawn and it was what it was.

I know several non beta test teams that invested heavily in learning Labview on thier own using Lego NXT robots. So what did we know that a non-beta team didn't. Some of the hardware a bit earlier, some of the libraries a bit early but most of those changed several times and then once more after we had to return the cRIO and equipment.

Hard to be much more definative and IMHO the only course of action to take.:)

[Daniel_LaFleur]

Your insinuation that somehow the beta teams had a leg up on the competition because they had a leg up on the control system is a pointless conversation to have. Teams chosen for beta testing were chosen because they would have a leg up to begin with. This may seem counter intuitive, giving the teams that would have succeeded anyway and forcing the teams to help other teams through the disclosure agreement made sense.

The data you are asking is quite easy for Michigan, for example, 67 WAS a beta test team. However, correlating their early experience with the control system to their success this year is simply impossible. In the real world there are too many variables from year to year to say that any one factor contributed to a given team's success. Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team. Naturally these teams will be ones that have a marked history of winning events. To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.

The tone of your post suggests that you are getting defensive. I did not mean to offend anyone or insinuate that the success of the beta testers was soley due to being a beta tester.

However,

Neither my team nor any other beta test team got a significant head start last year.

The above quote does not take into account that the beta testers had 6 weeks of familiarity with the cRIO and thus did not have to learn the basics from scratch just before build season. Remember: many rookie teams may not know where the FIRST forums are ... nevermind CD.

Also, suggesting that reading a blog or post about how something is done is as good as getting your hands on it and experimenting is just not true. You cannot easily replace first hand knowledge with third hand knowledge, it just doesn't translate that well ... especially with technical nuances.

Additionally,

Additionally your data will be skewed because teams were chosen because they had demonstrated that they were familiar with how to build a robot and run a FIRST team
By your own admission, the beta testers were very skilled. By contrast, many of the teams that needed extra time with the cRIO (rookies, small teams, teams without software mentors) were not afforded the time.


To make a claim that a team that had a history of winning events would not have won events had they not had the benefit of beta testing the control system is absurd.
Please state where I made this claim.
All I claimed is that they recieved a significant advantage over non-beta teams.

As always, the Above is JMHO.

[Chris is me]

The above quote does not take into account that the beta testers had 6 weeks of familiarity with the cRIO and thus did not have to learn the basics from scratch just before build season. Remember: many rookie teams may not know where the FIRST forums are ... nevermind CD.

Not to be elitist or anything, but if the rookies haven't been looking for help and for the resources commonly available to all teams, why should anyone be surprised when they have issues or are at a disadvantage? Yes we had an advantage over those teams, but so did you.

What I'm trying to say is, given the nature of who is picked to be beta testing, a team beta testing is not much better off knowledge wise than if the same team did not beta test. Teams like 67 / 1114 would definitely do everything they could to learn about what the beta teams are doing.

Regardless of CD or FIRST Forum use, all beta test teams held seminars and Q&As in the real world.

By your own admission, the beta testers were very skilled. By contrast, many of the teams that needed extra time with the cRIO (rookies, small teams, teams without software mentors) were not afforded the time.

The purpose of the beta test program is not entirely "to familiarize a team with an early version of the new control system that is not identical to the final version". We had work to do. Would the same teams that do not use Chief Delphi be able to help give back to FIRST as effectively?

[Akash Rastogi]

Yes let's give everyone a system before its been tested so they'll all have the same problems, that'll save everyone so much time and frustration.

Really now? Its Beta Testing for a reason. Testing, recording, and reporting is ALL beta testing is about. I've worked for engineering companies before over the summer too and we sent out products such as new ecoders, gyros, and potentiometers for them to TEST and REPORT. When the product was released to all manufacturing companies those who didn't test product were happy that in the new iteration of each product, there were NO problems in the device. They didn't complain or argue about another tester company having more experience with the product.

Haven't we always said that FIRST is a microcosm of life? Its not fair, nothing is. Live with it. NI is still a corporation first. We are the consumers; who, if given a faulty product that hasn't been tested within the target audience (oh hey Luminary Micro) will complain and moan and cry about the problems we encountered.

Sorry if that sounded rude or harsh.

[Jared Russell]

I contend that the beta test teams did not have a significant leg up over anybody else. I cite the following reasons:

1. Early on, the beta test teams were working with buggy hardware and software. Many of the things they learned about the nuances of the early control system and API had to be unlearned as updates were made available.

2. The beta test teams did not just get to play with their system for pre-season code prototyping. They had to fulfill specific objectives as a condition of their participation in the beta testing program. Much of their work in these areas was not even used this season (e.g. both 330 and 1114 demonstrated camera tracking code, but neither one wound up using tracking on their Lunacy bots).

3. All of the findings, code, and lessons learned from the beta test teams were made available to all FRC teams. For example, team 103 brought their beta control system to our off season event Ramp Riot, where they gave a presentation and answered questions from a live audience of 100 and an even larger online audience. Before kickoff, I already knew the WPILib API like the back of my hand from diligent studying.

4. The teams selected had the least to gain through early exposure to the cRIO. The programmers and mentors of these elite teams would have very quickly reached an expert level either during or immediately before build season (recall that many teams who were not beta testers still got their control systems early). If you take an experienced software engineer and give him the WPILib API spec, he could crank out good software in a matter of a few days if not hours. We're not solving NP Complete problems in FIRST, folks.

In short, I urge those who criticize the beta testing program or the team selection criteria used by FIRST/NI to see the program in a different light. It was not designed to give any particular team an advantage. It was designed to quickly find and fix bugs and to spread introductory cRIO knowledge to all of FIRST. The teams that were chosen had excellent histories of both technical accomplishment and community mentorship.

We all benefited from their experiences.

[Chris is me]

When the product was released to all manufacturing companies those who didn't test product were happy that in the new iteration of each product, there were NO problems in the device. They didn't complain or argue about another tester company having more experience with the product.

Though to be fair, those companies are not in a direct sporting competition where 2 minutes decides your fate.

As long as we're debating friendly FIRST style, what alternate solution would others propose? Considering the amount of (completely legitimate) complaining that's happened in previous years with faults in the field system, this seems like a better way to release a better product on time than to not test.

[Daniel_LaFleur]

I reversed your quotes so that I could answer them (i feel) in an appropriate logical succession.


Considering the amount of (completely legitimate) complaining that's happened in previous years with faults in the field system, this seems like a better way to release a better product on time than to not test.

I completely agree that testing is better than not testing. That is not the issue I am trying to bring up.

The issue I'm bringing up is that the teams with the most experiance, the most support, and the best chances at success were given an advantage (extra time with the cRIO to get an understanding of it's capabilities) while those who really needed the extra time and help got none. I don't believe that that was FIRSTs intent, but it is what happened (IMHO)

As long as we're debating friendly FIRST style, what alternate solution would others propose?


I would look for outside groups that are friendly to FIRST to do the beta testing. Possibly colleges that give FIRST scholorships but do not sponsor any particular team.

[nathanww]

As long as we're debating friendly FIRST style, what alternate solution would others propose?


Well, depending on the type of beta required, many beta tests could be made open betas. For example, what would a public beta of CAN require? New software libraries and a few cables. FIRST/WPI/whoever would need to provide libraries anyway for a closed beta.

Yes, that does mean more strain on FIRST/WPI to respond to bug reports,provide support, etc. But...that's really what has to happen for the product/system to mature. If 300 teams are emailing because they can't figure out how to configure something, it means the documentation's not clear. If FIRST gets 1000 bug reports for various things, it's a lot more work for them, but as long as they can be prioritized and dealt with, it means that teams AREN'T going to hit those issues during build season.

[Jared Russell]

Well, depending on the type of beta required, many beta tests could be made open betas. For example, what would a public beta of CAN require? New software libraries and a few cables. FIRST/WPI/whoever would need to provide libraries anyway for a closed beta.

Yes, that does mean more strain on FIRST/WPI to respond to bug reports,provide support, etc. But...that's really what has to happen for the product/system to mature. If 300 teams are emailing because they can't figure out how to configure something, it means the documentation's not clear. If FIRST gets 1000 bug reports for various things, it's a lot more work for them, but as long as they can be prioritized and dealt with, it means that teams AREN'T going to hit those issues during build season.

There aren't as many people available to work on the beta test on the FIRST/NI/WPI/Oracle side as you may think. Opening the beta up to 1000 teams is going to result in a lot more work without a lot more results.

From experience I can tell you that inexperienced users make lousy beta testers - their feedback can be vague, contradictory, and all in all detrimental to getting everything working. But if teams 67, 111, and 254 can't figure something out, then there's a good chance that somebody on the development side is going to need to fix something.

[Daniel_LaFleur]

From experience I can tell you that inexperienced users make lousy beta testers - their feedback can be vague, contradictory, and all in all detrimental to getting everything working. But if teams 67, 111, and 254 can't figure something out, then there's a good chance that somebody on the development side is going to need to fix something.

The issue I see with this way of thinking is that, believe it or not, most FIRST teams are fairly inexperianced. And if they cannot figure it out then the documentation needs to be revised so that the average user can use the product effectively.

As a manufacturing engineer, I could write a procedure that was technically correct, yet no one on the production line could follow it because they did not understand the terminology or concepts presented.

By leaving out those who are the most inexperianced, you effectively removed their chances to better understand the technology they are using and put them at a further disadvantage as well as remove any chance FIRST has to reach out and help them by changing the terminology to something they understand.

[EricH]

The issue I'm bringing up is that the teams with the most experiance, the most support, and the best chances at success were given an advantage (extra time with the cRIO to get an understanding of it's capabilities) while those who really needed the extra time and help got none. I don't believe that that was FIRSTs intent, but it is what happened (IMHO)
The teams that really needed the help did not have the resources to do an adequate beta test. They instead got instruction from the beta teams before they even had their control system, continuing until ship day and beyond. What's better: trial and error and maybe not working, or working with someone who knows what they're doing to show you what needs to be done?

The teams that had (have) the best chance of success at the game are also the ones that had (have) the best chance of success in beta testing. If you don't like that, then find software mentors for those teams.

I would look for outside groups that are friendly to FIRST to do the beta testing. Possibly colleges that give FIRST scholorships but do not sponsor any particular team. I doubt that that's a good solution. They have to 1) build a robot (or otherwise obtain one), 2) test the system (learning the languages from scratch, possibly), 3) find a suitable practice area, 4) test. On their students' (workers') time. And ensure reporting. Oh, and there's a decent chance that you might wind up with experts doing the testing, which will not help a rookie team, as the expert might not understand that these guys have no clue what they're doing. See your own comment about writing a procedure.

Sure, you could do it, but you kind of need to have an FRC team involved somewhere.

If most FRC teams are "inexperienced", then the ones that do have experience need to reach out to the ones that don't have the experience.

[nathanww]

But if teams 67, 111, and 254 can't figure something out, then there's a good chance that somebody on the development side is going to need to fix something.

Sure...but the problem with this method(having only very technically capable teams test equipment) is that what you gain in specificity, you lose in sensitivity. For example, take the extreme sensitivity of the 2009
DS to static discharge. There are very simple procedures that can be taken to work around this(i.e.the grounding modification), but it's a bug that did not show up, in a manner that anyone paid attention to, until systems were in use by thousands of teams.

[Joe Ross]

Well, depending on the type of beta required, many beta tests could be made open betas. For example, what would a public beta of CAN require? New software libraries and a few cables. FIRST/WPI/whoever would need to provide libraries anyway for a closed beta.

It also requires a CAN module for the cRIO.

I did email FIRST and ask about the possibility of doing a more open beta for the software. They said assuming there are enough qualified teams, they planned on the number of beta teams being larger then last year. I hope that other people with constructive feedback will take the time to email FIRST.

[Akash Rastogi]

It also requires a CAN module for the cRIO.

I did email FIRST and ask about the possibility of doing a more open beta for the software. They said assuming there are enough qualified teams, they planned on the number of beta teams being larger then last year. I hope that other people with constructive feedback will take the time to email FIRST.

I emailed before the summer started if the possibility of summer testing was something they could try out and then once school starts try to get any new hardware out to teams so they have at least a couple weeks to fool around with new/improved items...I unfortunately didn't get a response. :(

[Alan Anderson]

The issue I'm bringing up is that the teams with the most experiance, the most support, and the best chances at success were given an advantage (extra time with the cRIO to get an understanding of it's capabilities) while those who really needed the extra time and help got none.

That "advantage" was not limited to the Beta Test teams.

It sounds like you're forgetting the fact that the control system was available early for any team willing to pay for its shipment. It was shipped early to every recipient of a NASA grant, at no cost to the team, which ought to have included a fair number of rookies.

[Daniel_LaFleur]

That "advantage" was not limited to the Beta Test teams.

It sounds like you're forgetting the fact that the control system was available early for any team willing to pay for its shipment. It was shipped early to every recipient of a NASA grant, at no cost to the team, which ought to have included a fair number of rookies.

No, I'm acutely aware of that fact ... but:

Last year my team was new to FIRST (although I've had many years experiance as a mentor), but we were not considered a 'rookie' team because we had 5 members from a previous team.

We did get our control system early, but because we were new, we did not have a robot, and only a limited number of motors to test things with. We were not eligible for the NASA grant because we were not a rookie, hadn't had the grant the year before (we're new ;) ), and weren't going to a NASA regional.

While we put on our webpage, in the FIRST database, and elsewhere that we wwere looking for help we did not recieve any assistance from outside teams, even though we asked.

Now, I'm not complaining, because we have an excellent group of kids and a skilled set of our own mentors, so we faired well at our 1 competiton. My issues is for teams similar to mine but without mentors, teams that might only have a math or English teacher running the program.They are the ones that need the help the most, and usually end up with a box on wheels, because they didn't understand the needs of a program like FIRST.

But I digress...

My original post stands: IMHO the beta testers gain an advantage because of familiarity with the new systems (whether they use that advantage or not ... it's still there).

[JaneYoung]

My issues is for teams similar to mine but without mentors, teams that might only have a math or English teacher running the program.They are the ones that need the help the most, and usually end up with a box on wheels, because they didn't understand the needs of a program like FIRST.


I think that's ok. They will. There is nothing wrong with building a box on wheels, esp. a robust one that can be a welcome addition to an alliance because of other strengths, including attitude.

FRC isn't going to back away from the acronym it has created, inspiring young people and whole communities by partnering with engineers and other professional mentors. With initiatives such as this one, it becomes very evident, very fast, that the experience and educational background of the mentors on a team is valuable and applicable and an absolute plus. There's nothing wrong with that. Teams who are weak in technical mentors for whatever reason, or who are overwhelmed, can find ways to work with this initiative - by being mentored by the beta test teams. They can also work to build their technical strengths so that they can be considered in initiatives such as this in the future.

.02
Jane

[The Lucas]

One major thing I think would help teams with less programming expertise is:
There should be one and only one "all-the-basics" default code for each language. It should literally be named the "FRC 2010 DEFAULT CODE"
Last year, we were provided plenty of example programs and options. Labview had both Basic and Advanced mode. C++ had SimpleRobot and IterativeRobot. I loved the freedom to choose between these frame works and methods of programming. However, the less experienced don't want choices (at least initially), they want an obvious starting point that just works. It should be like the old IFI default code that mapped axis and buttons to all outputs so it could be used in competition without reprogramming. The BuiltinDefault code last year didnt do much, it was more of test of controller functionality then competition code. Also like the IFI controller, the map of default code functionality should be included in electrical white papers for the cRio, sidecars, CAN modules, etc.

I'm not saying WPILib should trim down the options, quite the contrary it like to see more. We should prop up one default code that we all can understand. All other example codes must be available in the same framework as the default (other frameworks can be provided too). Those teams with expertise still have the freedom to use/edit WPILib and port example code to fit their needs (ex: my team modified IterativeRobot to create MOERobot class). This year was much more difficult to help teams program in the pits because their code was based off of many different examples. In prior years most were based off of default and I could look right at the specific problem (the couple dozen modified lines) without figuring out their code structure first. I am very optimistic that this year's Beta Testers can help create a great default code building upon the work of previous Beta teams and other teams during the season.

[JDM]

All the arguments about disclosure/beta advantages/etc aside, has anyone seen any info on what is actually being beta tested (other than the "2010 control system")?

Other than that it will somehow include a Classmate PC in the driver station, I haven't.

[The Lucas]

All the arguments about disclosure/beta advantages/etc aside, has anyone seen any info on what is actually being beta tested (other than the "2010 control system")?

Other than that it will somehow include a Classmate PC in the driver station, I haven't.

CAN bus to control Jaguars is likely to be a major addition.

Some other functionality I would like opened up and added to the WPILib (mostly stuff that was technically possible last year):

Input from Classmate PC (keyboard, mouse, touch screen) to control the robot
Live video feedback on Classmate PC (lift the restrictions and standardize)
Network communication between allies.


As well as improving the current libraries and example codes.

[BornaE]

CAN bus to control Jaguars is likely to be a major addition.

Some other functionality I would like opened up and added to the WPILib (mostly stuff that was technically possible last year):
Input from Classmate PC (keyboard, mouse, touch screen) to control the robot
Live video feedback on Classmate PC (lift the restrictions and standardize)
Network communication between allies.
As well as improving the current libraries and example codes.

In addition to that, Java is added to the supported languages.
Also I believe both the C++ and The Labview Libraries will have new versions that needs to be tested.

I am also hoping for some access to the FPGA code that was locked in 2009.

[ttldomination]

Wow this forum got really controversial. :D. But an interesting topic none the less. Just to insert my opinion on the previous matter: I believe that the beta test teams did have aa advantage. But a good code can only take one so far. This year, the 67/1114 (a couple of the beta testers that i remember from last time) built AMAZING robots and they deserved to win. And the part that the programming would have came into play was the anti-slip code and possibly a trailer tracker, and considering that all teams had the same amount of time to solve this problem, I don't think anyone really had too much of an advantage.

As for the upcoming Beta Test, I'm not sure exactly what the Classmate PC will offer, but I'm sure it'll definitely be interesting. We've signed up for the Beta. With a bunch of teams within a 30 minute drive of us, it'll be a nice, central location. Although we surely wouldn't mind if 1746 or 1771 got the go for the test. Last yr, it was just a bit hard to make a trip down to midtown on tuesday afternoons to see the system.