Real-Time Scoring Not Working?
 

It appears that, at least from watching the Finger Lakes webcast, the real-time scoring system in the goals is not working well/at all.

Can anyone at an event corroborate?

FLR Match 12 saw the red alliance out 6+ discs into the 3 point goal in autonomous and they had a score of 0 coming into teleop. It was updated mid-match, but it worries me that they're tasking referees with the additional task of counting how many discs are scored.

Re: Real-Time Scoring Not Working?
 

Supposedly there is a camera taking a snapshot of the goals after auton. Presumably the scorekeeper or some kind of assistant reviews that snapshot, not a referee.

Re: Real-Time Scoring Not Working?
 

That's interesting. Do you know where/what angle the camera is placed at? Just wondering if they can correctly see if discs are stacked or not.

Re: Real-Time Scoring Not Working?
 

No idea. It was in Frank's Blog or something like that, a few days ago.

Re: Real-Time Scoring Not Working?
 

Is there actually a camera? I believe the blog post just used the word "snapshot," but didn't actually specify a camera. I interpreted that to mean they're just going to note how many discs were scored in the goals at the end of autonomous.

Re: Real-Time Scoring Not Working?
 

Good point. Someone is probably counting then...awesome.

Re: Real-Time Scoring Not Working?
 

They did say there wouldn't be a delay between autonomous and teleop:

Re: Real-Time Scoring Not Working?
 

I think what Cory is getting at is that once again, ensuring the auto score is correct is left to humans, which thanks to our fallibility, means that inevitably someone will make a mistake that affects the outcome of a match. Seriously; by now, FIRST should be able to design games that can be scored automatically, and accurately, with no room for human error.

Re: Real-Time Scoring Not Working?
 

I've seen scores flicker for entire matches.

Re: Real-Time Scoring Not Working?
 

Watching NYRO a minute ago, shortly after auto, the score was flickering 0-33 to 0-30, and then a few seconds later, it was 18-18.

So far, I'm totally unimpressed by the realtime scoring for this game.

Re: Real-Time Scoring Not Working?
 

There certainly seem to be problems. I too noticed the score flickering from 17 to 18 and back for one side in the last FLR match. Lots of trouble with autonomous scoring as well. Perhaps they could consider a manual/visual counting system while still taking the 'snapshot' into account.

Re: Real-Time Scoring Not Working?
 

aaaand FLR just had RTS fly to 147 for blue there before they took it off the screen.

Re: Real-Time Scoring Not Working?
 

FLR just flashed above 300. This is pretty bad.

Re: Real-Time Scoring Not Working?
 

Yup...

Re: Real-Time Scoring Not Working?
 

Here in Palmetto the scoring is awful. High score so far according to system was 1008 and not a single robot shot a disk. Like that in so many matches. Seems like even when it is corrected that it is still not right.

Re: Real-Time Scoring Not Working?
 

This is the main problem. Inaccurate real-time scores are worse than none at all, imo. It erodes people's confidence in the final scores.

Folks, trust that the final score is accurate. Even if the crew has to count the discs by hand, they'll be sure to get it right.

Re: Real-Time Scoring Not Working?
 

Call me cynical if you like, but I've been around this game long enough to know that I can't just trust that it will be right.

I've seen dozens of scoring errors over the years that affected the outcome of the match, and upon discovery, the teams were basically told "Oh well, crummy luck."

Never mind the hundreds of scoring errors that didn't affect the outcome of the match that I've seen.

I frequently hear people hide behind GP, as if expecting the event to be run in accordance with the rules and corrected accordingly when mistakes are made is too much to expect of volunteers. I wholeheartedly disagree. Event volunteers should be expected to be able to officiate the games in full compliance with the game manual. Period. Anything less is unprofessional.

Re: Real-Time Scoring Not Working?
 

When I read this, I think of Lunacy.

Re: Real-Time Scoring Not Working?
 

The Great Car Nack is right again.

Re: Real-Time Scoring Not Working?
 

The "snapshot" is virtual - there are humans that are counting and writing down the number of discs scored in autonomous. All the discs are counted as the goal bins are emptied at the end of the match.

I have seen scores "flickering"; presumably as the sensor picks up a weight that's nearly halfway between the weight of two discs and it's trying to figure out which discrete value to display.

One time the big screen showed a score of 404 until the FTA jostled the goal bin and the sensor went back on.

At Traverse City they had to replace the sensors or re-calibrate them 3 or 4 times today.

I don't believe there is a huge point swing once the manual count has been done. We had one match where the field crew counted correctly, but the numbers were entered into FMS for the wrong alliances - red and blue reversed. When that was pointed out several matches later, the scores were corrected.

It is unfortunate that the automated scoring doesn't work, and it does slow things down a little because it takes longer for field reset and scoring. But to FIRST's credit, they did admit that the system didn't work and came up with the manual backup. They didn't try to claim that "The sensors are always right."

Re: Real-Time Scoring Not Working?
 

Good choice of words Gary: Unfortunate. I am sure everyone had hoped it would work, but that's just not how it works in the real world.

I can't do better; anyone who can should do so before the end of the season and give one to FIRST for their use. Hey, maybe you'll end up on the GDC...

Re: Real-Time Scoring Not Working?
 

Problem to me seems to be the differentiation between auton and teleop points. Clearly just counting the ofrisbees in the goal at the end of the match does not tell you how many were scored in each category.

Palmetto score boards continuously posted auton points when no robots shot a single frisbee. What accounts for this? There should have been nothing to trigger a score.

Re: Real-Time Scoring Not Working?
 

At FLR, they put out a request early this week for extra volunteers (I would presume to fill the scoring roles. Our FTA said that FIRST realized the issue at the week 0 events, and it was in the plans for the events for this week to provide manual backup as Gary mentioned. The Field Crew at FLR was AMAZING, they were tirelessly and insanely counting and recounting frisbees... stacks and bins and everything, and doing it all often faster than the robots could be setup. Scoring tables were set up when we got there today, and at least 3 volunteers staffed each table at all times. Its a lot more work than we all hoped perhaps, but the volunteers have done a phenomenal job keeping up with it and keeping the day moving along very quickly.

Re: Real-Time Scoring Not Working?
 

On the subject of score flickering, part of it is (most likely) the discs still moving about inside the goal immediatly after landing inside the goal. From what I could tell today, the score slickering seemed to calm down as the discs 'calmed down' inside the box. Just my thoughts.

Re: Real-Time Scoring Not Working?
 

Hey maybe we're just that good? :p

Re: Real-Time Scoring Not Working?
 

Depends who you ask.:rolleyes:

Re: Real-Time Scoring Not Working?
 

The amount of variance in scores in the transition between auton and tele-op at Palmetto is stupid. This is getting ridiculous in most matches down there.

Re: Real-Time Scoring Not Working?
 

What is Manchester recommending to avoid these issues week 2?

Re: Real-Time Scoring Not Working?
 

I'm sure they will be working on it and a ruling will come out later this week. I don't think they would already have a fix.

Does anyone else seem to notice a large improvement in scoring today?
A lot of the matches at FLR are being scored correctly by the automatic system.

Re: Real-Time Scoring Not Working?
 

The sensors for real time scoring are insanely sensitive. Field reset was instructed to be as gentle as possible when emptying the goals at kettering so that the calibration wouldn't be thrown off. A feat in itself if you have the chance to see the rain of frisbees on the volunteer emptying them. A few teams' shooters were so violent that just the force of the discs hitting the back of the goal, this is after hitting the chains, threw off the weight sensors. I'm really shocked that FIRST didn't put a speed limit on discs in the rules like 06. It would be a good idea for many teams to tune their shooters down for simply field safety, let alone real time scoring.

There are multiple people counting discs entering the goals both during auton and during the teleop at kettering. From what I could see someone checks the auton scoring during the slight lull between aton and teleop to double check their figures, same with after the match. While inaccurate real-time scoring is disappointing, the FTA's, field crew, refs and everyone down at the field are trying to cover every possible way to make sure that the final score entered in for ranking is the right score. They should deserve a medal, in my opinion, this game is insane for them. One-hundred some odd game pieces to track, while dodging frisbees, making calls through a net, with a rather limited supply of extra discs. It is a miracle that matches don't take 10-15 mins to reset, and score.

Re: Real-Time Scoring Not Working?
 

My team has yet to go to competition, we're due there on March 14th, but we have been keeping an eye out for the scoring. A lot of people have been saying they're pulling in volunteers to fill in the spots for scorekeeping and such.
Can anyone validate these claims?

Re: Real-Time Scoring Not Working?
 

What kind of validation are you looking for? Isn't "a lot of people" pretty good validation? What about Frank's blog?

Re: Real-Time Scoring Not Working?
 

"A lot of people" is enough to support it, yes, but I was looking for a general number. (I should have specified.)
And could you please link me to Frank's blog?

Re: Real-Time Scoring Not Working?
 

Yes. Off by one to two discs.

Re: Real-Time Scoring Not Working?
 

I completely disagree with the attitude you are taking towards this. This is an issue that has some real on- and off-field consequences. Matches can be won or lost based on an error with the scoring system. Those matches affect who wins/loses events, who qualifies for championships, etc.

Sure, you might fall back on the whole "it's not about the robot" thing that people love to throw around. But I know that when talking to press, community members, potential sponsors, etc., it sure helps a lot to be able to show them some on-field success.

And what about when you explain to them that you missed out on winning an event because of an error with the scoring system? Doesn't that negatively influence their image of FIRST? Why would they want to sponsor a team that participates in a program that can't get its act together to figure out how to effectively score their own game?

This is another instance where FIRST should have anticipated this issue. How often have automated scoring systems worked out in games that have multiple game pieces being flung at high velocity into the goals? None that I can remember.

I also take issue with being told that I basically have no right to raise my concerns because I don't have a better solution. I'm a college student. My "job" is to study, learn, and pass my classes. The GDC and FIRST's job is to make sure their field works properly and that it collects accurate information about what's gong on.

Re: Real-Time Scoring Not Working?
 

I almost posted a picture earlier of a 300+ score coming out of Auton at Kettering. The blue alliance side was usually off by at least 50 points. Apparently they tried recalibrating it every few matches and then gave up.

At least with this system they realize that it really doesn't work, and have people manually scoring. Kettering had 4 people dedicated to counting discs during the match.

Everyone was basically ignoring the on-screen score since it changed wildly post-match.

Edit:
I think the best solution is to do something like what they did in many past games: Leave the scoring to the humans, but give them some sort of button box to deal with realtime scoring.

Re: Real-Time Scoring Not Working?
 

I was watching parts of the Palmetto and Finger Lakes regionals today, and what I didn't quite understand was how the automatic scoring system was generating odd scores in autonomous. You cannot have an odd score in autonomous since everything is multiplied by two...It looked like something was going on weird with the software.

Re: Real-Time Scoring Not Working?
 

Yes...many of the matches at Palmetto would start with a score of 30 or more on the scoreboard - before a disc was even shot.

It is particularly rough for those teams that developed a flexible endgame to apply for different needs...if you don't know how many points you need to win, how do you know what to do?

Re: Real-Time Scoring Not Working?
 

I worked as field reset and was told the weight sensors were for real-time scoring, and are highly sensitive. The reset volunteers count the discs and relay this to the front table, which overrules the real time scoring.

Re: Real-Time Scoring Not Working?
 

Darn. We had a 348 points for an alliance here at Hatboro.


Darn I thought we'd win.

Re: Real-Time Scoring Not Working?
 

I'm sure the real time sensors are weight sensitive and the refs were watching the auton scoring so they could subtract the number of discs from the total count. I suspect the photo had something mechanically stuck making the goal box seem heavy. If someone is counting the discs for each Q match, that's one hell of a job. There was more time being taken between elimination matches at Palmetto (including huddles at the scoring table), so they might have been counting them for eliminations.

If anyone here has a better suggestion for a real time scoring system for now or in the future, I'd suggest adding it to your feedback forms; but sometimes the real world is not a very precise place.

Re: Real-Time Scoring Not Working?
 

I was at the Kettering District competition in Michigan this weekend and the volunteer coordinator had me and a teammate counting the disks and reporting the scores to the scoring table to double check the automated scoring system. For the most part the scoring system seemed fairly accurate, however we did find that the one two point goal wasn't registering disks at the end of the day. Now keep in mind that me and my teammate are both certified referees and trusted to report the scores honestly and accurately. To just grab to random team members and have them reporting scores could result in opportunities/temptations to alter the scores but it keeps the load off referees and I expect to see it at the rest of the Michigan competitions.

Good luck to all teams

I'll see you at Gull Lake, St. Joes (as a Ref), Livonia, and Bedford

Re: Real-Time Scoring Not Working?
 

At the Hub City Regional in Texas they would count all of the discs at the end of autonomous and at the end of a match to decide the final scores. The live scoring system wasn't very accurate as the scores got higher, matches with fewer discs would be scored better. It is something that I hope to see calibrated better as the weeks move on.

Re: Real-Time Scoring Not Working?
 

I would not worry too much about this.
Humans, with our vastly superior computing power have found a solution and it is working fine.

I was sitting at the scoring table with the scorers counting the goals. It was actually kind of fun, especially when 1519, a cross court shooter, got in their calibrated zone. "1, 2, 3, 4, 5, 6, 7, 8 ,9 in the 2 point goal"

Here in New Hampshire, all of the FIRST horsepower was there analyzing the signals and checking the variance of each matches counts in each goal.
Data was taken and will be analyzed.

Between the visual counting of the frisbees going in in autonomous and tele-op, and the final volunteer counting verification, the scoring was sound. IMHO

A better solution may been to use RFID's on the frisbees. That way we could also catch the souvenir hunters. :)

Re: Real-Time Scoring Not Working?
 

Load cells used in weighing systems are generally accurate & robust. The problems come up with how they are placed into the system. The goals need to be total supported by the cells with no contact with anything else. I aspect this will get better as the weeks go on & lessons learned get incorporated into field set up.

Re: Real-Time Scoring Not Working?
 

They are totally supported by the cells.

Re: Real-Time Scoring Not Working?
 

i was a field volunteer at BAE.

From the start, we were counting discs as a verification of the score. When the counts were below about 15, it was easy to count from behind the player stations. When the counts were higher, or we were not confident, then we did a manual unload and count and the scores were not input until we submitted the human counted counts. We counted after Auton and at the end of the match.

For elims, there were two people at each end just counting and we had to agree.

All of the volunteers recognized the importance of accuracy, and how one disc can make an impact. I have a lot of confidence in the BAE scores.

I think FIRST will create an improved system, but it may take more than one week since the trucks are already on their way to week 2 events.

Re: Real-Time Scoring Not Working?
 

Chris, what's the protocol for verifying an Autonomous score? Can you elaborate on the "snapshot" concept?

Re: Real-Time Scoring Not Working?
 

After Einstein last year, I'm sure FIRST will be taking every measure to make sure the field works as designed.

Re: Real-Time Scoring Not Working?
 

In college, my Design/Build/Fly team tried to build a wind tunnel balance. We were unsuccessful at convincing any of the faculty let us use a real load cell, so we found one on the internet that was surprisingly cheap. We bought it, and implemented it in our design. Big surprise, it did not work at all. In back to back runs it would give us totally different answers. We ended up using a postal scale, which worked great for our purposes! Another guy on that team checked out the HH field, and apparently FIRST is using that same load cell. :ahh:

Re: Real-Time Scoring Not Working?
 

We counted and recorded the discs after auton was over. Those counts were easy because the most you could have was 9, and it was easy to count them coming in, and from looking up into the box. The person at the field table also counted as they went in (there was a team of people for red and for blue alliance ends).

At the end of the match, we then just did a total count. The person recording the scores (on paper) then did the math to write in the number scored in teleop. This written score was then handed to the official scorekeeper to input into the system.

Re: Real-Time Scoring Not Working?
 

This seems a little vague. Can you give more details? What internet sensor were you using, what circuit was the sensor in, what was the resolution and rate of the measurement?

Greg McKaskle

Re: Real-Time Scoring Not Working?
 

Probably because it is a little vague, and perhaps a little premature. I believe was a custom Arduino rig, but I am fuzzy on the details. Basically we took raw voltmeter readings -- clearly you could do a lot post processing to smooth it out. I've attached one of the resulting data files -- as I recall our issue was not with a given run giving weird data, it is that you just couldn't get the same thing twice back to back. Clearly we were sampling way faster than the sensor was updating. We were using it as a thrust test stand, which is not a great environment for a load cell because there is a bunch of vibration. However, we had similar issues even using bricks. Using an electronic postal scale to measure the force at the same location on the lever, these issues went away.

I don't think I have any of the documentation on my computer anymore, but I'll email around and see if someone else has the part number.

EDIT: FC-22, the 0-10 lb version. I can't find somewhere where FIRST lists the sensor that they used, but this family of sensors pops up quite a big when I google around for low cost load cells. Anyone "in the know" or take a look at the field to confirm?

Re: Real-Time Scoring Not Working?
 

Hopefully it can just be a software update to further refine and process the inputs, which can be applied at any time. As noted, weight counting is effective when used properly.

Re: Real-Time Scoring Not Working?
 

It seems like this is one of the few cases where high speed cameras might be useful. What did you guys do for goals that happened right at the buzzer?

Re: Real-Time Scoring Not Working?
 

I've got a better word than unfortunate. Unacceptable.
This is far from the first time that FIRST has screwed up real time scoring. It's a complete joke at this point. To has real time scores as far off as these are completely erodes the public and team's trust in FIRST and the GDC. Between this and the field delays, I actually had a student ask me something along the lines of "how can such smart people screw up so badly?" This isn't something that we should just shrug off. Just about everyone saw this coming from miles away, so why FIRST thought it would be an acceptable solution is beyond me.

And it goes beyond the load sensors. As someone already pointed out, the FMS is displaying odd scored in autonomous. Odd scores aren't even possible in autonomous. "61" was a frequent score for the blue alliance during autonomous at Hatboro-Horsham. You can score 60 points in auto. You can score 62 points in auto. You cannot score 61.

As for a better option, humans. That's what they're using now to get accurate scores, might as well use it for real-time as well. Aside of 2010 and 2012, the only accurate real time scoring FRC has ever used has been input by humans. 2004 was humans. 2005 was humans. 2007 was humans. 2008 was humans. 2011 was humans. The only time human RTS has been lacking was in 2003 and 2009, but automated scoring wouldn't have faired any better in those chaotic games.

The max in 9? Tell that to 365, 103, and 2559. They scored 11 in SF2-2 at Hatboro-Horsham.

Re: Real-Time Scoring Not Working?
 

I really wish the GDC would have anticipated the potential problems. I can't imagine, given how widespread they were, that this didn't crop up before. As Sean and some others have pointed out, humans could do the real time scoring. I think that pads like they had in 2011 would work for a good approximation for the RTS. Add in a manual count for verification at the end of each match and I think you would have a perfectly acceptable alternative.

Re: Real-Time Scoring Not Working?
 

Really? Our alliance scored 62 in auto in one of the finals in Hub City, and I'm pretty sure that wasn't the high auto score for week one. I'm not sure, but I think that was 7 6-pointers from us, 2 6-pointers from 3931, and 2 4-pointers from 1801. Still easy to count, but it might mean someone has to take off their shoes. :)

Re: Real-Time Scoring Not Working?
 

The highest I know of was either 66 or 72 points by 365, 103, and 2559. I can't remember if they got one autonomous match to work perfectly or not (I know they definitely had a match where they hit 11 shots and one fell out of 103 before shooting). 2559 would score 2 from the front right corner of the pyramid. 103 would score 3 from the back right corner, then drive underneath the pyramid to collect 2, and score those. 365 would score three from the rear center of the pyramid, drive backward to collect 2 from the centerline and score those. All of them at the three point goal. Both 103 and 365 would occasionally have some issues collecting their additional disks, but pretty much nobody on that alliance would miss. It was incredible.

Re: Real-Time Scoring Not Working?
 

While I agree that at this point human counting is probably the quickest, easiest and, by far, the cheapest solution to the inaccurate scores. How exactly are they going to be implemented into the existing field set up for a real time score? Not only software and hardware, but where would you put them? They can't be behind the goals; you can't see most of the discs enter up top or the pyramid. They cant be at mid field; you cant see the 1pt goals. It is also extremely hard to see the pyramid goals from field side. You would need at least 3 pads per alliance to even have a shot at being right.

When you add in the speed that discs will be scored during the later regionals/districts, let alone MSC, MAR champs, and St.Louis, and there is no way you will get an accurate real time score from a real time human count.

This game is simply not designed for human real time scoring. 100 some odd game pieces flying in multiple directions, being scored in 8 hard to see containers... I sure as heck don't want to be the one with that job. Its a similar situation to 2011, and that worked oh so well.:rolleyes:

Re: Real-Time Scoring Not Working?
 

2011 worked quite well...

Re: Real-Time Scoring Not Working?
 

2009... oops. I keep confusing them and I'm not sure why.

Re: Real-Time Scoring Not Working?
 

I disagree.

The rack was scored manually, that was fine.

The minibot tower sensors were quite bad in the first two weeks of gameplay. The initial fix was to heavily filter the data, which then changed the required time to hit the trigger (which, not being specified as a requirement in the manual, left many teams unable to trigger it because it didn't trigger as specified). A later fix was to redesign the sensors, and this helped somewhat.

However, the sensors still only had a ~95% success rate (based on our 2011 CMP scouting data), and the rules and team updates specifically prohibited the refs from awarding any points to a team which clearly hit the top of the tower. Basically, FIRST said numerous times that the sensors were infallible and there was no room for argument. FIM refs routinely ignored this and awarded some points anyway, but at CMP we recorded several incorrectly scored matches, some of which were close enough to possibly swing the results.

The problem in 2011 was not that the sensors weren't perfect (they eventually got quite good), but rather that the rules and team updates repeatedly asserted that the sensors were perfect and it was not possible to argue any other way.

2010 also had it's faults, specifically when two balls would go through the sensors together and it would read them as a single ball. Offsetting the sensors would have solved this. In addition, they would also give penalties based on a return timer, and an incorrectly detected ball on the return track could force the system to repeatedly give penalties until the end of the match.


I can't think of too many issues with the 2012 automated scoring.
I can't think of any issues with manual scoring. The realtime score is reasonably correct (not off by hundreds of points), and the final match score is always correct (after manually counting game pieces).

Re: Real-Time Scoring Not Working?
 

The manual aspects (racks scoring) did. The automated aspects (minibots), not so much. Why the GDC/FIRST doesn't see this trend yet, I have no clue.