Open scoring software?

[Kevin Sevcik]

Seeing various scoring spreadsheets fly around gaining features and fixes has gotten me thinking. Anybody want to speculate on the odds of FIRST just Open Sourcing the scoring software and throwing it at the teams and mentors to check over and debug?

There's gotta be a few dozen qualified programmers in the FIRST community, atleast. I'll grant you that game specific stuff would be tricky. But ranking algorithms and a general scoring database would be applicable from one year to the next, along with most display stuff and arena control stuff. FIRST might just keep one or two people on staff mostly updating the program for the next game, then release the source to teams the same time as they release the game. Then teams and other FIRSTers could check things over in their spare time. The FIRST staff programmer would then be in charge of integrating any fixes or changes. Yeah, I know that teams don't have that much free time, but still. I don't know that FIRST has anything to gain from keeping the code proprietary unless they're contractually bound to do so. Any other thoughts?

[David Kelly]

Seeing various scoring spreadsheets fly around gaining features and fixes has gotten me thinking. Anybody want to speculate on the odds of FIRST just Open Sourcing the scoring software and throwing it at the teams and mentors to check over and debug?

There's gotta be a few dozen qualified programmers in the FIRST community, atleast. I'll grant you that game specific stuff would be tricky. But ranking algorithms and a general scoring database would be applicable from one year to the next, along with most display stuff and arena control stuff. FIRST might just keep one or two people on staff mostly updating the program for the next game, then release the source to teams the same time as they release the game. Then teams and other FIRSTers could check things over in their spare time. The FIRST staff programmer would then be in charge of integrating any fixes or changes. Yeah, I know that teams don't have that much free time, but still. I don't know that FIRST has anything to gain from keeping the code proprietary unless they're contractually bound to do so. Any other thoughts?


That would never happen.

[Ryan Albright]

WEll the software is not written By FIRST its done by a third party

[Greg Marra]

WEll the software is not written By FIRST its done by a third party

This is kind of disappointing, because I'm sure some great things could come out of the community if it was available.

Not to mention it'd just be another outlet for people to learn.

[Kevin Sevcik]

This is kind of disappointing, because I'm sure some great things could come out of the community if it was available.

Not to mention it'd just be another outlet for people to learn.
I concur, which is why I'm mentioning. The community (mostly Joe Johnson) has developed and debugged a heck of an Excel scoring spreadsheet in under 48 hours. FIRST has no competitors to worry about. Indeed, I would think FIRST would welcome other organizations taking the software and using it for other robotics competitions....

[Justin]

Bravo!!! I think this is probably one of the best solutions to the scoring software "problem" I've heard yet. Let's face it there are several people already in the FIRST community that have already written their own version of the scoring software for one reason or another. I think this idea would have several key benefits. One it would open the algorithms to those in FIRST, no more wondering how thing X or Y is computed. If teams have issues they can go in and audit the code themselves. In addition if done right the software will be very modular and componentized and broken up into several teams. In theory things like rankings, GUI, shouldn't change much from year to year. Basically what will change is the way the score is calculated. Now this is something that would be "game specific" BUT I think that this could easily be handled by FIRST by recruiting Alumni and other "teamless" members of the community and getting that team under an non-disclosure-agreement.

I think this is an idea that could happen. To say "that would never happen" why? This is an idea that FIRST should seriously consider and that deserves some debate and honest thought. I think there are a lot of pros here. Just because there a pieces of the scoring system that need to remain private from teams doesn't mean that this isn't a feasible idea. Someone recently said that the route to power lies in giving some of it up, perhaps Dean and could learn from another Dean who made that observation. The FIRST community is full of rich resources and usally egar to help, they need only ask.

My 2 cents for what its worth,

Justin

[Joshua May]

I've got to say, this is a great idea. Hopefully, in the near future, FIRST will at least get the most experienced programmers and those that have experience with the excel-scoring programs and have them hack up a successful program. With the great minds working with many of the FIRST teams, it wouldn't be too bad. And that way, we could customize to what we need without any outsourcing and third parties, and open source would allow for pretty easy bug fixes. Certainly something to suggest to FIRST.

[seanwitte]

What do you mean by Open Source? I think a lot of people have different ideas about what that means. I have no strong feelings about OSS either way, but opening a product like that up to the community will create a "design by committee" atmosphere that is not really conducive to producing results. FIRST will want one person who is ultimately responsible and accountable for the software.

Instead of making it Open Source, why don't you recruit a team and submit a proposal to FIRST stating the objective. Your proposal should include information stating who is directly accountable for the end product. Your proposal should also include resumes and the method by which the project will be managed. Building a distributed realtime match management system should be possible assuming all regionals have internet connectivity.

If you choose to do it for free then they might just take you up on it. Sending them an email saying "make is Open Source" will get you nowhere.

[Kevin Sevcik]

I never said it had to be a design by committee. Go to Sourceforge and check out all the FOSS stuff there. There are tons of highly organized rapidly moving projects there. There's no reason that everyone has to vote on everything in the project. The sensible way to work it would be for FIRST to hire a Linux developer or someone experienced in managing FOSS projects. This person would be in charge of directing the volunteer programmers, organizing the project, and vetting all code to be incorporated into the system.

The whole point of FOSS is to have a really large team working on the project. More eyes on the code means more bugs found faster. I would say that FIRST could just post the source code for the current Hatch system, except that it's quite probably proprietary code that can't legally be opened.

[seanwitte]

I never said it had to be a design by committee. Go to Sourceforge and check out all the FOSS stuff there. There are tons of highly organized rapidly moving projects there. There's no reason that everyone has to vote on everything in the project. The sensible way to work it would be for FIRST to hire a Linux developer or someone experienced in managing FOSS projects. This person would be in charge of directing the volunteer programmers, organizing the project, and vetting all code to be incorporated into the system.

The whole point of FOSS is to have a really large team working on the project. More eyes on the code means more bugs found faster. I would say that FIRST could just post the source code for the current Hatch system, except that it's quite probably proprietary code that can't legally be opened.

I was trying to explain that simply telling FIRST "make it open source" is not going to cut it. "More eyes on the code means more bugs found faster" is not necessarily true. What if 90% of the extra people have no clue what they're looking at? You pull 100 well meaning individuals out of a crowd of volunteers and I would bet maybe 20% would be truly able to contribute. Having a large development team is great for a public domain project, but not for something that has been traditionally very guarded. These are things that I would question if I were FIRST.

It sounds like you have given it some thought and are headed in the right direction. You just need to be very specific in stated objectives, processes, and resources if you want to be taken seriously. Venting on CD is one thing, but is anyone willing to actually take the next step and float a proposal to FIRST?

[Justin]

Having a large development team is great for a public domain project, but not for something that has been traditionally very guarded. These are things that I would question if I were FIRST.

I think one of the good issues that this thread raises is...why is it so guarded?? I can see things like the game have to be guarded from people how have the desire/ability to transmit that information to teams. However we hear a lot about how FIRST is short on staff, etc. I think that if they decentralized more some good things could come from it. Teams have a lot to contribute and a lot of ideas on how FIRST is/should be/could be run. Yes there is the team forum and yes direct results come from that but why not give teams some power to truly effect things. It might serve the spread of FIRST better if FIRST were a more bottom up rather than top down organization. There are lot of alumni that are not on teams any longer and might be put to constructive uses. I haven't seen a lot of FIRST trying to do that.

It is also interesting to me that so many people think that FIRST is guarded/secretive/etc...perhaps because to some extent it is. However I also wonder if this isn't because of the oligarchical nature with which FIRST is run. I guess your opinion on whehter it should be this way depends of if you view FIRST as a service provider and teams a customers or whether you view FIRST as a community. Dean and Woodie to some extent are the co-rulers of the FIRST universe, I understand it is their creation and their vision and that naturally creates a desire for them to retain a good deal of the control over it. The question is, and I'm sure many will scoff, does this situation serve the overall goals of FIRST best??

Justin

[tux]

I agree that the scoring software should be open source (and Free, if possible) as should all of the code used for FIRST Robotics. This is all about learning, after all.

I can see some possible objection that the scoring system should be in place before the announcement of the competition, but at our regional competition it seemed that the scoring system was not complete even after the competition (no rankings during the competion, and no scores at all during the practice rounds).

Is Free Software the answer to every programming question? No (yes ;o) But at least at this level (students, learning about how systems work, and how people work together on systems) I see no reason not to use open code.

[Max Lobovsky]

In addition to improving the quality of the software, there are two advantages I have not seen mentioned in this thread. 1. It will save FIRST money on the software (do they get it donated to them already?) 2. Will be another engineering inspiration gracious professional <buzzword> <here> <here> <and here> thing that FIRST can do. Introducing people into open source, providing more interesting work that the students can do, etc etc.

[Josh Hambright]

After talking to Bob Hammond at length this season about the scoring software, the flaws in it, and the flaws in the entire process through which FIRST developes the scoring software, I can say pretty confidently that we will not be seeing open source scoring software any time soon.

FIRST will definately be looking at how they are doing things before next season, they may go with a new company they may stick with this one. One problem is that every year they goto a new company, or a new software package or a new system just as the bugs from the old system begin to be worked out. The system itself doesn't need to change much from year to year, but rather the gui does and maybe a few other minor tweaks.


I know that many people would love to develope a scoring system for FIRST, there are many people and teams just itching for the oppertunity, many of the people on here are more then qualified for the challenge, however I must say that most people completely underestimate the process.
If it was as easy as everyone thinks, then why has it been such an issue in the past? Why is it that almost every year there are problems with the system? Maybe its because its quit a task to take on. Most people dont realize how complicated the scoring system is. As it controls the field, the scoring, the alliance picking, the seeding, the rankings, everything other then IFI's monitoring stuff and the music and video feeds is controlled by that one system.


I think there will be more talk in the coming months about the scoring system, and I encourage people to try and develope ideas and share those ideas with FIRST, as that is how alot of the great things happen, by making an effort to make a change.

And as far as the multitude of homegrown scoring systems that poped up this year, we have been seeing these for years, however I can say from my own experience that they saved alot of peoples butts this year. both competitions i attended this year, i was involved in the scoring system at, and at both of them we used Dr. Joe's scoring system to keep a double check and a back up of the main scoring system. Which is great, redundancy is a very good thing when its something as vital as ranking and scoring. However i have yet to see any home grown system that has the gui, the control of the sounds, and most importantly control of the field boxes that the real scoring system does, and therein lies the challenge. there is so much more to this problem them people realize.

. Dean and Woodie to some extent are the co-rulers of the FIRST universe, I understand it is their creation and their vision and that naturally creates a desire for them to retain a good deal of the control over it. The question is, and I'm sure many will scoff, does this situation serve the overall goals of FIRST best??



Although Dean and Woodie are very powerful forces within FIRST, they do not make alot of decisions, nor do they do alot of the day to day operations. This task falls onto a small staff that manages to do an amazing job running this massive beast we call FIRST every year. This list (http://www.usfirst.org/about/brdirect.htm) has alot of the of the people who make the decisions as to what happens in FIRST, though there is also alot of input from respected people in the community, the staff at FIRST, and other forces that lead them to the decisions that they make.
The issue of secrecy isn't that big of a deal, the software that they have now is pretty much worthless without all the field hardware, i'm sure if you rent the field from FIRST they give you the software, I know that the scorekeepers all had access to the software, updates, backup systems, and doublecheck systems that FIRST released. Its not really a secret its more an issue of people not needing the software because there is nothing they can do with it.

[Dave Flowerday]

I know that many people would love to develope a scoring system for FIRST, there are many people and teams just itching for the oppertunity, many of the people on here are more then qualified for the challenge, however I must say that most people completely underestimate the process. If it was as easy as everyone thinks, then why has it been such an issue in the past? Why is it that almost every year there are problems with the system? Maybe its because its quit a task to take on. Most people dont realize how complicated the scoring system is. As it controls the field, the scoring, the alliance picking, the seeding, the rankings, everything other then IFI's monitoring stuff and the music and video feeds is controlled by that one system.
I do believe that a lot of folks underestimate the complexity of the software, but let's not overestimate it either. This stuff is not nearly as complicated as the things that many of the software engineers in FIRST work on during their day jobs. You ask "If it was as easy as everyone thinks, then why has it been such an issue in the past?" Perhaps the answer is because they've chosen the wrong people to do it. This year is a great example - the problems we saw at competition clearly illustrate that there was no formalized test process for the software that was written. Even a basic level of testing would have revealed defects such as half the teams being displayed as Team #0. Test processes that validate team rankings as calculated by the scoring software obviously didn't exist.

Software development groups that have to work on large scale projects that are "business-critical" (meaning failure simply is not tolerated by the customer, certainly applicable in this case) follow a very regimented process to do their development. Contrary to popular belief, software engineers do not just open up a text editor and start hacking code when they begin a project. (For details try reading this (http://www.fastcompany.com/online/06/writestuff.html) article about the way the software for the space shuttle is developed, which is very similar to what we do at Motorola and I'm sure many other companies.) From what we've seen this year we can conclude one of two things: either the company that developed this software did not have a good development process and just started hacking on it, or they did have a process in place but did not follow it. Either way it is inexcusable, and as such it's not something that just allowing them to get in a year of experience first is going to fix. I believe it would be a mistake for FIRST to reuse this company again next year.

I think the person who commented on this situation a few weeks ago had it right: we are customers of FIRST. Just because we believe strongly in its goals and want badly to see them succeed should not change the fact that we're paying customers. We should expect and demand better than they've given us this year. The fact that there are many qualified software engineers who regularly work on software that is orders of magnitude more complicated and are willing to help FIRST out in this area just makes this situation even more unacceptable.

To any teams out there that are unhappy about the scoring system this year: I encourage you to provide feedback to FIRST at the team forums this summer about this situation. There's no need to rehash all the problems or have a long discussion on it or anything, but make sure they take note of your opinion.

[tux]

> I can say pretty confidently that we will not be seeing open source scoring software any time soon.

> FIRST will definately be looking at how they are doing things before next season, they may go with a new company they may stick with this one. One problem is that every year they goto a new company, or a new software package or a new system just as the bugs from the old system begin to be worked out.


I do not see hiring a company to create the scoring software as diametrically ruling out open source software. Just make that one of the requirements of the contract.

If FIRST is purchasing software and not getting source code for it, I believe that is a terrible mistake. If they had the source code from the first time they bought scoring software, maybe they would not have had to buy it again (and again, and again). Software without source is just a black box. You will never fully understand it, and you will always need someone else to fix the (inevitable) problems.

> However i have yet to see any home grown system that has the gui, the control of the sounds, and most importantly control of the field boxes that the real scoring system does, and therein lies the challenge. there is so much more to this problem them people realize.

Sure. And Apache when it was first released did not have support for virtual hosts and modules and authentication and ssl and ... etc etc etc, but when people needed those things they were able to create them and add them in.

[lynca]

It seems that using screenshots and documentation on how to use the software would be a method of open source. Someone could easily replicate the interface and make it open source. After using the scoring package, it is actually quite simple. If this can be done with quite a bit of time before a competition than FIRST will most likely consider the best open source package available.

Also, the scoring animation/simulation white paper for 2005 was released and well tested a few days after the kickoff. I doubt FIRST had their scoring program complete at the same timeframe because real testing did not start until competition. FIRST should consider building off exisiting white papers and create their own official version.

~Andrew