A New Way to Scout

[Kimmeh]

Originally Posted by antimatter_john

Exactly what? There was no exactly to my question. My question is is why is IT SO IMPORTANT TO KEEP RAW DATA KEPT AWAY FOR ONLY CERTAIN TEAMS. (No yelling inteded just want to get my point across). There is no point. Yuo even said your self



If I am missing something please tell me.

My apologies. I was actually agreeing with you. I see no point in keeping raw data away from teams.



Now, I offer a more broad question to those not interested in waiting to make the information public.

As a community, we see nothing wrong with helping a team make their robot more competative at an event. It's considered a good thing. It helps lift the floor of the competition, right? We share all our "best practices" in hopes that it makes others better.

For those of you against this, or even just suggesting we wait until after the competition: why? I don't understand why you have a problem with it, and I'm trying to understand. In my mind, I see it the same way as helping a robot. We're trying to make teams better.

Perhaps I'm understanding this incorrectly (or others are), the goal would be to share the raw data, correct? We're not asking anyone to share how they manipulate the data. Just work together to get the data and let teams do whatever they choose with it. Correct?

[brennonbrimhall]

Originally Posted by Kimmeh

For those of you against this, or even just suggesting we wait until after the competition: why? I don't understand why you have a problem with it, and I'm trying to understand. In my mind, I see it the same way as helping a robot. We're trying to make teams better.

I agree; that's why we've scouted collaboratively before. Not only does it share data that would be hard for smaller, more challenged teams to get, but it also develops great inter-team relationships.

However, not all teams share that view, and that's perfectly fine.

Originally Posted by Kimmeh

Perhaps I'm understanding this incorrectly (or others are), the goal would be to share the raw data, correct? We're not asking anyone to share how they manipulate the data. Just work together to get the data and let teams do whatever they choose with it. Correct?

I think that's a topic that deserves more discussion. It seems we have three platforms that have been brought up, and if a unified system is to be achieved, we should discuss how that's implemented.

[Andrew Schreiber]

Originally Posted by yash101

Yep. That kind-of doesn't seem too accurate because this is about the 2006? control system!

No, all of Erich's information is about the current-ish control system. The 2006 control system was based on a 900mhz system. In other words, what are you talking about?


As for the questions earlier on why you can't run 2.4Ghz at low power - Simple, it's easier to say "No Wifi" than it is to say "No wifi except in this situation" because then you have to police that. It's more work for an already over worked group of folks.

Originally Posted by Kimmeh

Perhaps I'm understanding this incorrectly (or others are), the goal would be to share the raw data, correct? We're not asking anyone to share how they manipulate the data. Just work together to get the data and let teams do whatever they choose with it. Correct?

Nah, I think we should be sharing how we manipulate the data. Data manipulation and visualization is as much engineering as learning CAD even if it is relegated to the back rooms of scouting meetings in FRC. Course, I'm biased…

[Invictus3593]

This is great! How would you compile the list of each team competing and the regionals they are attending? Also, how would you get ranks, scores and match points for each event?

Our team programmers are developing an Android App to do scouting on tablets and also a PC program to sync up the data entered on the tablets with the PC via bluetooth, then compile all event data into a spreadsheet for review.

I foresee issues where teams don't agree on specific criteria to track.

To start, we got together and decided on which data to get and which data to omit, also there's a "notes" section where we can add things like "swerve drive", "defensive only", and things like that. There is also a spot for what we call "Blowouts", where a team may have gone into a match with a working robot and then gotten flipped or had electrical problems. This can give us an idea of how reliable their bot is or would be.

[1306scouting]

Team 1306 did this last year.

(As far as I know, I was the first to use the term CrowdScout to describe this paradigm of collaborative data gathering. I'm amused at the parallels between my announcement last year and the one at the beginning of this thread.)

Last year we worked with several teams using a paper/electronic hybrid system - scouters filled out laminated scouting sheets with wet erase marker, and we scanned them into the computer and then were able to distribute the data to other teams. It was quite successful - it cut the work for each time by a significant margin as well as providing more robust and useful data than having a single team scout. We can use our existing knowledge and infrastructure to help the initiative avoid some of the mistakes we made last year.

1306 is well into developing everything we'll need to do this on a large-scale environment, across all regionals, with teams contributing data and getting access to the raw data. We should be able to integrate data from pretty much any electronic source via the internet (we'll have an API as well as writing some interaction code for specific solutions), and share the data with all participants.

The more people who collaborate, the better this will work.

Please PM me if you are interested. I will start putting up more information in the coming few days.

[1306scouting]

As I mentioned (apologies for the double post, but this post covers different topics), 1306 has experience doing this, so I thought I should share to offer that viewpoint for anyone it can help:

1. It's really cool to watch teams work together on scouting.
   
2. We've designed most of the server-side application and are beginning to implement. Once I meet with the lead programmer, I'll see about open-sourcing it. We have dedicated server hardware for this, as well as some possible leads into getting redundancy and so forth.
   
   For the techies here, we're planning on using relational databases to store all the data so that we can efficiently pull the data we need while retaining large amounts of metadata (like the team that submitted and timestamps).
   
3. Mycroft Holmes method is fantastic, but it's not infallible - even if your person is amazing, they can be susceptible to biases and so forth. I like having a Mycroft Holmes-type person with data that we can tap into if we need. You are very lucky to have people that can do that.
   
4. In regards to privacy: 1306's general preference is to give all the raw data to everyone that participates and some "processed" data away for free (balanced by algorithms to create lists). 1306 has always given these lists away to rookies, but we think they're valuable to everyone without giving up any competitive advantage to participants.
   
   This is very open for discussion as we move into implementation and so forth. I personally think information should be free for everyone, but I can see why others would not agree.
   
5. We've designed the platform so that we have a CrowdScout API this year that allows us to take data from pretty much any electronic source. We use paper for our scouters (it's reliable and inexpensive, as well as not needing power and allowing the people scanning it in to see the data before it's uploaded to catch human errors immediately), but we should be able to support apps that implement the API as well as pretty much whatever else the community can dream up.
   
   The API is still young - I agree with brennonbrimhall about having different teams track different metrics, and agreeing on them on a per-tournament basis, but there are advantages to standardized metrics.
   
6. Bluetooth PANs work fairly well for localized internet sharing, and shouldn't have any interference issues with the field nor break any rules. That said, I highly recommend setting up ethernet networks if you can - bluetooth can have issues with many hosts; an old router and ethernet will almost always outperform bluetooth.
   
7. Qualitative data is incredibly important, and because every team functions differently, it's hard to share. CrowdScout could be the platform for doing so, if needed, but I've always thought a team-by-team basis was better for that.

[brennonbrimhall]

Originally Posted by 1306scouting

As I mentioned (apologies for the double post, but this post covers different topics), 1306 has experience doing this, so I thought I should share to offer that viewpoint for anyone it can help:

1. It's really cool to watch teams work together on scouting.
   
2. We've designed most of the server-side application and are beginning to implement. Once I meet with the lead programmer, I'll see about open-sourcing it. We have dedicated server hardware for this, as well as some possible leads into getting redundancy and so forth.
   
   For the techies here, we're planning on using relational databases to store all the data so that we can efficiently pull the data we need while retaining large amounts of metadata (like the team that submitted and timestamps).
   
3. Mycroft Holmes method is fantastic, but it's not infallible - even if your person is amazing, they can be susceptible to biases and so forth. I like having a Mycroft Holmes-type person with data that we can tap into if we need. You are very lucky to have people that can do that.
   
4. In regards to privacy: 1306's general preference is to give all the raw data to everyone that participates and some "processed" data away for free (balanced by algorithms to create lists). 1306 has always given these lists away to rookies, but we think they're valuable to everyone without giving up any competitive advantage to participants.
   
   This is very open for discussion as we move into implementation and so forth. I personally think information should be free for everyone, but I can see why others would not agree.
   
5. We've designed the platform so that we have a CrowdScout API this year that allows us to take data from pretty much any electronic source. We use paper for our scouters (it's reliable and inexpensive, as well as not needing power and allowing the people scanning it in to see the data before it's uploaded to catch human errors immediately), but we should be able to support apps that implement the API as well as pretty much whatever else the community can dream up.
   
   The API is still young - I agree with brennonbrimhall about having different teams track different metrics, and agreeing on them on a per-tournament basis, but there are advantages to standardized metrics.
   
6. Bluetooth PANs work fairly well for localized internet sharing, and shouldn't have any interference issues with the field nor break any rules. That said, I highly recommend setting up ethernet networks if you can - bluetooth can have issues with many hosts; an old router and ethernet will almost always outperform bluetooth.
   
7. Qualitative data is incredibly important, and because every team functions differently, it's hard to share. CrowdScout could be the platform for doing so, if needed, but I've always thought a team-by-team basis was better for that.

I'm very interested in your platform; 20 has been working on our own solution called Gemini. It too isn't designed to replace paper scouting, but to aggregate and calculate statistics. We've also scouted with other teams at WPI, Connecticut, Champs, and IRI, so our system was designed with that in mind. Once the finishing touches get finalized, I'll post the codebase here.

[Roger]

Originally Posted by khanh111

If anything, I think the "Mycroft Holmes" method should be supplemented by some form of collected hard data.

Well, if you can't believe Mycroft, who can you trust? (You're just lucky Mycroft is too lazy to prove he's right.)

It's also true that not every team has a "Mycroft", so alternatives have to be made anyway. Our team is also developing a "crowd" type scouting system, which does seem the way to do it -- the more data in, the better. Though in general two people can always have opposite conclusions for the same bit of data.

[khanh111]

That's always something to consider (synergy). And I definitely do think that those people are valuable. It's just that we are all human, and we make mistakes and/forget things. Scouting data is the best way to confirm observations made throughout the day.

Additionally, explaining the rationale behind decisions always leads to better ideas, and allows for proof when you make a statement.

[khanh111]

There are arguments to be made for both sides.

As of now, few enough teams are able to collect good data that good data is itself a competitive advantage. That's why I originally thought that data should be kept within teams of the scouting alliance.

In any case, data will eventually become so widespread that good data will be everywhere (think stock market data). When that comes, data analysis (which is already extremely important) will become even more important.

It's up to us as a community to decide when that will be though. I liked 1306scouting's idea of giving out separate lists to teams pause the scouting alliance.

[xXTHEGUARDIANxX]

i don't know if this method has been posted but what my team did at last regional was two people(we sent our driver and co-driver) would look at our next match and go scout the 5 teams (the opponents first) and on a sheet of paper ask them the statistics on their robot) after acquiring information we then go to our allies and ask for their information then we plan a strategy based on all information gathered. we keep all information so we can use it in future rounds/competitions

[EricH]

Originally Posted by xXTHEGUARDIANxX

two people(we sent our driver and co-driver) would look at our next match and go scout the 5 teams (the opponents first) and on a sheet of paper ask them the statistics on their robot)

Some folks call that pit scouting; others call it useless.

You're actually better off having someone in the stands look at how those teams do in the last match before the one in which they play you, and getting the stats data down to the drivers as soon as it's collected, than asking them their stats, because their stats will almost invariably be without defense and/or estimates--and therefore higher than actual.

The other question would be, why are you having the drivers scout? I can understand if you're short on people, but IMO, drivers should not be scouting. They should be working with data they've been handed to develop a strategy.

[xXTHEGUARDIANxX]

i am head driver/scout i offered the idea of pit scouting thanks for info on scouting while match going on. i came up with the idea of going to each pit because we weren't scouting at all our first matches.i see what you mean though last year we had only 10 (lack of members)...

[DampRobot]

Originally Posted by EricH

Some folks call that pit scouting; others call it useless.

You're actually better off having someone in the stands look at how those teams do in the last match before the one in which they play you, and getting the stats data down to the drivers as soon as it's collected, than asking them their stats, because their stats will almost invariably be without defense and/or estimates--and therefore higher than actual.

The other question would be, why are you having the drivers scout? I can understand if you're short on people, but IMO, drivers should not be scouting. They should be working with data they've been handed to develop a strategy.

I'm our team's coach, head of mechanics, and part of our pit crew. Arguably, I should have the least free time of almost anyone on the team. But before every match, I take an in depth look at the scouting data for each team we're playing with and against, and then take a look at their robots in the pits. If we're with them, I ask to talk to their drivers or coach and talk strategy for a good five-ten minutes (sometimes multiple times before a match if it's a critical match).

Actually seeing the robots we'll be in a match with with my own eyes is really important. You can get a good sense of how their robot and team is doing just looking for yourself. That robot with a 2 CIM Kitbot drive and ugly painted bumpers? Might not going to move. Neat looking bot with only the programmer working in the pits? Could be someone to look out for. Afterwards, I actually go and talk with them, to see if we can help them get their bot ready, or just to talk strategy. It's important that I'm the one doing this because I'm the one who's going to have to face them or work with them on the field. Also, because I know what it's like to play in a match firsthand, I know what I need to look for to gauge their robot's performance.

I do agree that just asking teams "how they've been doing" and basing your strategy on that isn't the best plan. Team's overestimate, have a proximity bias, exhibit a halo effect, yada yada yada. Like you said, it's critical to base your assumptions on how their robot will do on your own (well collected) scouting data. On the other hand, actually seeing the robots and talking to their people (as a driver) can really give a better perspective on what your strategy should be.