A New Way to Scout
 

As any scout knows, data is king. More data is better (as long as you can analyze it). What I wanted to discuss in this post were the methods we use for competitive analysis. (I actually gave a presentation on this recently - the slides are linked at the bottom).

As far as I have seen, there are five methods of scouting:
1. Paper
2. Spreadsheet
3. Tablet/Web/Phone Application
4. Databases (supplementary - The Blue Alliance, CheesyScout, CowScout, etc.)
5. Crowdscouting

Each of them has their own strengths and weaknesses. What I specifically wanted to talk about was crowdscouting. Crowdscouting would be a way for multiple teams that go to different events to contribute to the same dataset, which would be accessible by all members of the crowdscouting alliance (and publicly released after the season). It hasn't been used on any large level (that I know of), so there's a lot of potential for expansion here.




Why?
There are two problems that we will face in the upcoming years:

1. More data in less time - As FIRST grows, and we see more events (nationwide districts by 2017) and more teams, we'll be interacting with more teams at more events. We'll have less time to know each team's strengths and weaknesses, and to notice changes between each event (a team that does horribly at their first event could improve multifold by their second event).

2. Competition will increase - This was a major factor behind the change to districts. In the words of one of the planners of the district system, "Michigan is able to put out so many competitive robots because they have more events." Scouting capabilities will need to increase with competition.




Crowdscouting Discussion
I wanted to start a discussion on what guidelines and practices we could use to create crowdscouting systems. I've thought of a few myself:

1. Involved teams would need to decide what to keep track of early in the season. This could be done via email or a Hangout or Skype. They would need to decide on a method to collect data as well. Do they want an API to be able to collect realtime data?

2. Quantitative data only - this means things like types of drivetrains, autonomous points scored, etc. We don't want qualitative data because different teams want to track different things in different ways. There would be too much confusion based on judging data quality. Hard data can't be argued with. Teams would be free to collect their own qualitative data.

3. I'm having trouble imagining this with using an app. Data is only as helpful as how quickly you can gain access to it. On 1540, we use tablets with data connectivity to access the internet (because wifi can mess with the Field Management System). Our scout data is immediately accessible to the drive coach, who can then use data to plan our matches.

How many other teams do this? If there were enough, it would be simple to make an API to send data directly to a central server, and have any team able to access it immediately.




I'm already developing a crowdscouting API for this.

I look forward to the discussion.
-Hamzah

If you are interested in working with me on this, please contact me:
Email: khanh@catlin.edu
Twitter: @JSandrobots1540


Slides - http://static.squarespace.com/static...esentation.pdf

Re: A New Way to Scout
 

That is some great information!

Re: A New Way to Scout
 

At each of our regionals, our team has partnered with a few other teams for scouting. We called it our scouting "alliance". It works really well.

Currently, our awesome ginger programmer, Brennon (brennonbrimhall on cd) is developing this crazy effective scouting application. It's basically completed. It combines our paper scouting with a easier method of data entry (multiple computer data entry) among other awesome things.

I can't say I even close to understand it. But if you'd like, you can PM him. He is planning on probably releasing his program on CD sometime after it's been regional-tested.

You also might wanna talk to 610, who is making their scouting data at their regional available to all the attending teams on a monitor in their awesome scouting room thing.

Re: A New Way to Scout
 

I don't have anything technical to add, but I just want to say that I wish there were real statistics available to every team in every regional in real time. Then we could stop having arguments about OPR and have some really fun discussions about real scoring stats. Crowd-scouting would be a good step toward that.

Wasn't there a team that already developed something like this for the championship? I remember somebody had a system of paper forms that anybody from any team could fill out as a scout, then that team compiled everything electronically.

I remember that thread, but I don't think it was very active, so I don't know how successful that was.

Re: A New Way to Scout
 

I am thinking of designing my application to run on the internet, through my Pi server. I am aiming at using WebCache to save all the required files, reducing the data requirements. Everything will be synchronized to the server. I will try to create a database for every team where they can store their information. Other than that, there will probably be competition databases and public databases to make sure that a team doesn't have to share it's data if it doesn't want to. Also, another reason why I wan't to limit the network bandwidth is because this server runs on a 768kbps uplink! This would hog all the resources if I do not limit the usage!

Currently, I am working on the concept of it, and brushing my PHP, MySQL and AJAX skills so that I will be able to create a working program in no time!

Re: A New Way to Scout
 

Some students on 1306 have a good system for crowdscouting at events, which only requires paper, one laptop, and a scanner (so more teams can use it; no need for lots of tablets/laptops). Making an online database to store the info would be pretty easy.

http://www.chiefdelphi.com/forums/sh...d.php?t=116182

Re: A New Way to Scout
 

I know that this paper has a ton of statistical data and calculates OPR (and all of the other bells and whistles).

Re: A New Way to Scout
 

One simple way to get around the internet issues might be to have a "default" application for the hard data.

This would mean a mobile webpage that anyone can access from a smartphone. They would input hard data, which could be uploaded to the server at competitions without any internet issues.

The problem would be if a team wanted to have their own qualitative data. An option would be to publish the code, and allow changes so that the hard data go to the crowdscouting server, but the qualitative data goes to another server.

Re: A New Way to Scout
 

There is another method, which, for lack of a better term, I would refer as the "Mycroft Holmes" method (Wikipedia's description is incomplete). It is someone that somehow can keep statistics of all the teams in his/her head, and be able to balance off which team would be good, and which to watch out. We've had a couple like that, and they are much better than any other system. I told the team's teachers to look out for a sports fanatic student for this role.

Re: A New Way to Scout
 

I like your idea and I think it would work but I don't see the reason to keep the data private during the competition season. There is no hurt that I know of from the data being public. Also that would help the rookie teams or teams that don't have a proper scouting team in place to collect data. Also what is the point of the data after the season, it is more like a incentive that isn't really an incentive to the teams that aren't part of this but want the data or need the data. I don't know. Just seems improper that you aren't releasing the data to the public and keeping it private until the season is over.

Re: A New Way to Scout
 

I made this really advanced google docs scouting sheet

Here is how it worked:
1. The scouter filled out a little form on google
2. The form automatically sends the data to a google spreadsheet
a. One page in the spreadsheet is for the raw data
b. one is for the data that is sorted by teams and averaged
c. one is for the interface that we can use for graphs
3.The spreadsheet automatically sorts all the information and makes it easy for me or whoever is looking through it to see all the matches and trends in near real-time

The only downfall to this is you need internet (which was solved by writing down the matches and giving it to somebody that did)

and you need people who want to scout (which we had like nobody) (it also doesnt help that you dont do well)

If anyone wants to know how I did it, just ask, its pretty simple

Here is a picture of the interface, which has been improved with graphs and individual results after that competition
http://imageshack.com/a/img834/186/k4qi.png

Please note that all the boxes had info at it at one time. The code somehow broke when I was copying it for another competition

Re: A New Way to Scout
 

Team 696 used a similar Google Docs based scouting system for the 2012 and 2013 official seasons. Although simple and easy to use to create a system, it has MAJOR downsides.

1. Google's systems have crashed at every single competition. It seems like Google Spreadsheets can not handle the amount of formulas needed for any meaningful analysis. Because of this, we were not able to access our data for our Friday night scouting meetings.

2. Its SLOW. At the height of it's lag, it took around 30 minutes for data to be submitted. In addition, the sheets sometimes take forever to load.

I warned a team going to Champs not to use a Google Docs system, and they faced similar results (for post-day analysis).


If not for these, Google Docs would be the perfect scouting solution. This summer, we created a scouting system running off our website that would be able to input data, analyze the data, create rankings and team reports, and create a match strategy printout that people would be able to send to a printer in the pits. We successfully collaborated with 3476 and 3255 and used the system at IRI, Battle at the Border, and Fall Classic.

Re: A New Way to Scout
 

Our head scout for the last few years was this type of person. His skills were all enhanced by our data collection. It also made it easier for him to explain reasons to the rest of the team, which in turn allowed him to better develop strategies.

If anything, I think the "Mycroft Holmes" method should be supplemented by some form of collected hard data.

Re: A New Way to Scout
 

The reason I would make the data private during the season is because having data is currently a major competitive factor. Until we reach a point where there's some third party scout that makes all data public, data collection will be a huge playing card at any event.

To clarify, "private" as I used it meant accessible to all teams who contributed to the dataset.

There will be some point at which scouting data and stats will become widespread, and at that point, the challenge will change primarily to using the data effectively. (think stock market - there's a lot of data, and lots of opinions).

Re: A New Way to Scout
 

While our team does have it's own form of scouting, I kinda think that this method is what suits me the most. I like to look at regionals I didn't attend and try to work out what teams had the most synergy, and what the best possible alliances would be.

That's always something to consider (synergy). And I definitely do think that those people are valuable. It's just that we are all human, and we make mistakes and/forget things. Scouting data is the best way to confirm observations made throughout the day.

Additionally, explaining the rationale behind decisions always leads to better ideas, and allows for proof when you make a statement.

Re: A New Way to Scout
 

What advantage does it have over wether your teams lose or win though? Big teams already have their own scouting system and so does most medium sized teams. Also you could be helping rookie or new or small teams. I don't see the point of it being secret, it's not like it is a special robot strategy or robot function. It is merely harmless data.

Re: A New Way to Scout
 

Exactly. What makes scouting actually mean something is what you do with the data. You're sharing the raw data, not how it's manipulated to be effective for your team.

Re: A New Way to Scout
 

One of the ways we've debated doing scouting this season, as opposed to the hundreds of sheets of paper, is having a webpage which all of the scouters can connect to on their laptops, phones, tablets, etc. through a dedicated mobile hotspot. This data is collected, and organized on the scouting lead's Ipad, which allows for medium speed data review, depending on the internet speed.
We haven't thought about it fully, we've just been talking about it. The scanner method might work better though, as it seems simpler and more cost-efficient.

Re: A New Way to Scout
 

Check rule T04:

"Teams are not allowed to set up their own 802.11a/b/g/n/ac (2.4GHz or 5GHz) wireless communication (e.g. access points or ad-hoc networks) in the venue.

A wireless hot spot created by a cellular device would be considered an access point."

Re: A New Way to Scout
 

I foresee issues where teams don't agree on specific criteria to track. For example, as the only team that consistently used ground pickup at the Connecticut Regional, it made no sense for us to actively track teams who were picking up Frisbees off of the field. Constraining teams to a predetermined set of actions to track would be a major turn off to a lot of teams.

It would be far more feasible to have an effort where one or more teams at an event decide to publish their data in a common interchange format, to a common server. Teams could hash out collaborative details at events.

I see crowdscouting/teams publishing their scouting information as a potential solution to the downfalls of OPR for events that I can't attend, not a substitute for scouting well at your event. There isn't a replacement for solid data and being able to understand the picture those data are telling you.

I've been working on a scouting system that is based on node.js, which is awesomely cross-platform and renders as a webpage. With a few modifications, it could be a platform whereby teams can post their data to it -- like Blue Alliance, but with a much higher granularity.

Re: A New Way to Scout
 

This is great! How would you compile the list of each team competing and the regionals they are attending? Also, how would you get ranks, scores and match points for each event?

Our team programmers are developing an Android App to do scouting on tablets and also a PC program to sync up the data entered on the tablets with the PC via bluetooth, then compile all event data into a spreadsheet for review.

To start, we got together and decided on which data to get and which data to omit, also there's a "notes" section where we can add things like "swerve drive", "defensive only", and things like that. There is also a spot for what we call "Blowouts", where a team may have gone into a match with a working robot and then gotten flipped or had electrical problems. This can give us an idea of how reliable their bot is or would be.

Re: A New Way to Scout
 

Technically, you can make a bluetooth hotspot and achieve the same thing (802.15, separate IEEE standard). Of course it requires an ability to communicate via the desired bluetooth protocol, but it can be made to happen.

Re: A New Way to Scout
 

Exactly what? There was no exactly to my question. My question is is why is IT SO IMPORTANT TO KEEP RAW DATA KEPT AWAY FOR ONLY CERTAIN TEAMS. (No yelling inteded just want to get my point across). There is no point. Yuo even said your self

If I am missing something please tell me.

Re: A New Way to Scout
 

Because scouting is a whole other aspect to the competition. There are teams who devote themselves to having great scouting systems in order to gain a competitive advantage. And they absolutely deserve that advantage, in the same way that teams who devote themselves to building their robot deserve a competitive advantage, and the teams who devote themselves to outreach and the chairman's presentation deserve an advantage in the chairman's award. If the data was made public, then all of the sudden every team has access to this awesome resource, and all of the time and effort which some teams have put into developing their own systems is wasted. I'm definitely not on one of those great scouting teams, but I know that if I was then this happening would make me pretty frustrated.

Also, you've made the assumption that all big-medium teams have good scouting systems, but that is not at all true. Many teams, including those with fairly decent robots, still struggle to get any sort of meaningful scouting data. And I don't want it to be this way, but the solution is not just to hand them data that they didn't earn. That's one of the things that I like about crowd scouting the way the op is presenting it; it gives teams the opportunity to have good scouting, but they have to work for it. There is no reason why most* teams can't send 2 students to help the crowdscouting effort. So why make it optional?

*Of course there are some exceptions, but those should be handled on a case by case basis rather than as a general rule.

Re: A New Way to Scout
 

Exactly this. The point of scouting is to evaluate my opponents and determine who my best theoretical alliance is. If I'm in a picking position, other alliance captains' lack of scouting (for various reasons) allows me to potentially pick teams that put up exceptional numbers, yet may have flown under the radar. Often, these "steals of the draft" will all but guarantee my eventual victory. Why on earth would I want to make the data public, if it allows my opponents to recognize who I want to pick, and subsequently pick the teams I'm after?

Re: A New Way to Scout
 

Thanks Jim Wilks. I didn't catch that. I'll make sure to tell my team that, so we don't do anything stupid. We might just have to compile it online and offsite.

Re: A New Way to Scout
 

Do some research into tethering the devices via Bluetooth. It's under a different standard, and should be an effective workaround.

Re: A New Way to Scout
 

I'll have someone look into that, as I'm not very, shall we say, well-versed in the art of bluetooth/wireless connections. Thanks for the info!
*EDIT*
Would 3G connections (such as a wireless connection on a smartphone/tablet over a cellular network for the individual device) conflict with afore-mentioned rules?

Re: A New Way to Scout
 

True, but remember, bluetooth runs on the same frequency, 2.4GHz. So, that means, if you want the competition to go by fast and no robot breaking, limit on your usage. Though this may be hard to use, try something like ZigBee? Or if you want to get complicated:
Cell phone with 4G Data connection > Laptop (Bluetooth) > Ethernet ICS > Ethernet Switch > wired connection to node ethernet to usb > Tablet

Of course we don't want to get this complex, so let's stay simple...Just limit your bluetooth usage as it will cause interference. Also, with the above method, you can simplify things:
Rid bluetooth, cell phone and data connection (4G)
Run XAMPP on computer and serve your scouting application, at the spot.

Re: A New Way to Scout
 

Please, what you said doesn't make sense :)
Like I said earlier, I'll have someone who has some knowledge look into it. Thanks for your time.

Re: A New Way to Scout
 

What I mean is:
So, when you set up tethering on your phone, you create a Bluetooth Personal Area Network (PAN). Now, that is just one PAN. It won't do much of anything to the internet connection. Now, picture a stadium (or the stands at competition) with over a few hundred bluetooth PANs. When you have these many bluetooth PANs, the Wifi (the transmission the Field Management System (FMS) uses) will have more interference.

As a programmer, I would say this because the Ping time, back and forth, gets greatly increased. That means that if you are doing offboard processing like vision tracking, the lag will get so high, the driver would go home (or their hotel room) with a migraine.

I tried to elaborate. Please let us know if you have any problems, and remember, there is the old, handy-dandy google!
:D :D :D :D :D

Re: A New Way to Scout
 

There is a possibility that our team will be able to make a public real-score world ranking data base that all teams could contribute too. It may be a year or two but we're hoping to make it happen eventually.

Re: A New Way to Scout
 

Kind of the same here, though I am getting no team help. I'm just doing it for fun! :D

I am pretty sure many of you may be wondering what type of fun I have! :D

Re: A New Way to Scout
 

Regarding publishing data giving away a well-deserved competitive advantage, I have a unique perspective. 20 collaboratively scouted all three of our events, and IRI. Giving data away doesn't magically make teams better -- the analysis of data does, as well as the understanding of what the numbers mean.

That being said, I recognize the predicament. Perhaps teams should consider publishing data after the event on a unified server, in a common format. Thoughts?

Re: A New Way to Scout
 

This would be great.

Re: A New Way to Scout
 

It is my understanding that the field uses only the 5GHz band so it should be unaffected by bluetooth communication.

Re: A New Way to Scout
 

Team 1306 did this last year.

(As far as I know, I was the first to use the term CrowdScout to describe this paradigm of collaborative data gathering. I'm amused at the parallels between my announcement last year and the one at the beginning of this thread.)

Last year we worked with several teams using a paper/electronic hybrid system - scouters filled out laminated scouting sheets with wet erase marker, and we scanned them into the computer and then were able to distribute the data to other teams. It was quite successful - it cut the work for each time by a significant margin as well as providing more robust and useful data than having a single team scout. We can use our existing knowledge and infrastructure to help the initiative avoid some of the mistakes we made last year.

1306 is well into developing everything we'll need to do this on a large-scale environment, across all regionals, with teams contributing data and getting access to the raw data. We should be able to integrate data from pretty much any electronic source via the internet (we'll have an API as well as writing some interaction code for specific solutions), and share the data with all participants.

The more people who collaborate, the better this will work.

Please PM me if you are interested. I will start putting up more information in the coming few days.

Re: A New Way to Scout
 

It's a completely different standard, and it's used by many teams for a tablet-based scouting solution, RUSH being one of them. FIRST has never had any problems with it, nor have I ever heard of Bluetooth and wifi interfering with each other. So it's always an available solution.

Re: A New Way to Scout
 

Actually, after reading more into the control system, I found out that you, Nirvash, are right.
http://www.usfirst.org/uploadedFiles...n-Rev-0-4c.pdf -- Page 4

Re: A New Way to Scout
 

So, that makes me think...why is wifi 2.4GHz disallowed, especially at low transmission power? My DD-WRT router allows me to change the Tx power, and I can possibly change it to just 1mW, not enough to penetrate anything, or travel more than a few feet!

Re: A New Way to Scout
 

As I mentioned (apologies for the double post, but this post covers different topics), 1306 has experience doing this, so I thought I should share to offer that viewpoint for anyone it can help:

1. It's really cool to watch teams work together on scouting.
   
2. We've designed most of the server-side application and are beginning to implement. Once I meet with the lead programmer, I'll see about open-sourcing it. We have dedicated server hardware for this, as well as some possible leads into getting redundancy and so forth.
   
   For the techies here, we're planning on using relational databases to store all the data so that we can efficiently pull the data we need while retaining large amounts of metadata (like the team that submitted and timestamps).
   
3. Mycroft Holmes method is fantastic, but it's not infallible - even if your person is amazing, they can be susceptible to biases and so forth. I like having a Mycroft Holmes-type person with data that we can tap into if we need. You are very lucky to have people that can do that.
   
4. In regards to privacy: 1306's general preference is to give all the raw data to everyone that participates and some "processed" data away for free (balanced by algorithms to create lists). 1306 has always given these lists away to rookies, but we think they're valuable to everyone without giving up any competitive advantage to participants.
   
   This is very open for discussion as we move into implementation and so forth. I personally think information should be free for everyone, but I can see why others would not agree.
   
5. We've designed the platform so that we have a CrowdScout API this year that allows us to take data from pretty much any electronic source. We use paper for our scouters (it's reliable and inexpensive, as well as not needing power and allowing the people scanning it in to see the data before it's uploaded to catch human errors immediately), but we should be able to support apps that implement the API as well as pretty much whatever else the community can dream up.
   
   The API is still young - I agree with brennonbrimhall about having different teams track different metrics, and agreeing on them on a per-tournament basis, but there are advantages to standardized metrics.
   
6. Bluetooth PANs work fairly well for localized internet sharing, and shouldn't have any interference issues with the field nor break any rules. That said, I highly recommend setting up ethernet networks if you can - bluetooth can have issues with many hosts; an old router and ethernet will almost always outperform bluetooth.
   
7. Qualitative data is incredibly important, and because every team functions differently, it's hard to share. CrowdScout could be the platform for doing so, if needed, but I've always thought a team-by-team basis was better for that.

Re: A New Way to Scout
 

While we're at it with the bluetooth stuff, why not make your scouting system managed by a NXT? :D They can interface with up to 4 devices at once (or many if you chain them) Sure using Lego would be less-than Ideal, but it sure would be funny to see someone using a NXT as a practical device

Re: A New Way to Scout
 

I don't know how current this is (and, BTW, use of a 2009 Getting Started guide might not be the greatest idea, given that we're dealing with 2013/14 here--the system has changed), but at least at one time, the field COULD run on 2.4 GHz. That was the theory, at any rate. IIRC, it was actually tried once, partly due to restrictions on the 5 GHz band at one particular event. Suffice it to say that the odds of all 6 robots on the field actually working were pretty close to nil for the first day or so of that event, and not all that much better later (though there was improvement thanks to the field staff working very hard).



Tl;dr: While the field nominally runs on 5 GHz, it (and the robots) can run on 2.4 GHz if needed, and they don't need any more potential interference there than is already there.

Re: A New Way to Scout
 

My team has access to over 20 NXTs that is a great Idea.

Re: A New Way to Scout
 

I also think this would be great. It would be fun to play around with even if teams waited till after the season.

As for a format, one year we had a scouting program that stored everything in text files in the form:
<match#> <team#> <property name> <value>

So you'd get lines that said something like:

Re: A New Way to Scout
 

Yep. That kind-of doesn't seem too accurate because this is about the 2006? control system!

Re: A New Way to Scout
 

So, with this data storage, are you using text-based database? If so:

match/team#/climb code (0=no,1=10,2=20,3=30)
That would be a lot more readable to PHP and easier to implement. However, it would be important to note that text-based isn't so reliable. If this software is running for multiple teams, every time a match ends, the server will get a huge load. Multiple appends to the same file at the same time, many times, means a corrupt file. Instead, MySQL or some other database server built to store information may be the lifesaver!
The only time I used text databases is when I was learning PHP and using it on a site that maxes out at a view per day!

Re: A New Way to Scout
 

I'm very interested in your platform; 20 has been working on our own solution called Gemini. It too isn't designed to replace paper scouting, but to aggregate and calculate statistics. We've also scouted with other teams at WPI, Connecticut, Champs, and IRI, so our system was designed with that in mind. Once the finishing touches get finalized, I'll post the codebase here.

Re: A New Way to Scout
 

My apologies. I was actually agreeing with you. I see no point in keeping raw data away from teams.



Now, I offer a more broad question to those not interested in waiting to make the information public.

As a community, we see nothing wrong with helping a team make their robot more competative at an event. It's considered a good thing. It helps lift the floor of the competition, right? We share all our "best practices" in hopes that it makes others better.

For those of you against this, or even just suggesting we wait until after the competition: why? I don't understand why you have a problem with it, and I'm trying to understand. In my mind, I see it the same way as helping a robot. We're trying to make teams better.

Perhaps I'm understanding this incorrectly (or others are), the goal would be to share the raw data, correct? We're not asking anyone to share how they manipulate the data. Just work together to get the data and let teams do whatever they choose with it. Correct?

Re: A New Way to Scout
 

I agree; that's why we've scouted collaboratively before. Not only does it share data that would be hard for smaller, more challenged teams to get, but it also develops great inter-team relationships.

However, not all teams share that view, and that's perfectly fine.

I think that's a topic that deserves more discussion. It seems we have three platforms that have been brought up, and if a unified system is to be achieved, we should discuss how that's implemented.

There are arguments to be made for both sides.

As of now, few enough teams are able to collect good data that good data is itself a competitive advantage. That's why I originally thought that data should be kept within teams of the scouting alliance.

In any case, data will eventually become so widespread that good data will be everywhere (think stock market data). When that comes, data analysis (which is already extremely important) will become even more important.

It's up to us as a community to decide when that will be though. I liked 1306scouting's idea of giving out separate lists to teams pause the scouting alliance.

Re: A New Way to Scout
 

I agree 100%. There is no substitute for hard data, but having someone that actually remembers what the robots look like, how they play,what they do in a match and so on is very benefical. At events for my team, I watch one or two teams a match and write qualitative notes down. At the end of the event I know the basics of every team (and a lot about certain teams) and could make a pretty good pick list. But, I'm always surprised when going through the data to see how well or how poorly some teams actually do (consistency, especially in autonomous, is typically the biggest source for deviation). It simply comes down to not being able to watch every match for every team.

Re: A New Way to Scout
 

On the topic of keeping data private versus sharing it with the world:

It's a huge waste of time for every team to dedicate 7+ people to compiling the same exact set of statistics. If a community effort produced the same thing with far less total effort, the teams with the most effective scouting systems could instead put that effort into creating more advanced statistics and improving their qualitative scouting and their methods of collecting and using that qualitative data. Those teams would still have a competitive advantage to show for their efforts.

Re: A New Way to Scout
 

I agree that it can be a waste to have 7+ people constantly looking for the same set of statistics. However, some larger teams use that as a way to help teach newer members about scouting/strategy. Every experience helps a younger member.

Re: A New Way to Scout
 

That is what FRC is about, right? Anyway, I wouldn't see a problem if raw data was released, and not the analysis, interpretation, or anything similar. While we might think the game is the primary focus, we're all here to learn.
The only problem with releasing data, is that it'd be hard to get all the data world-wide in a real-time format, with enough time for choosing good alliance partners.
But those are the musings of a 1-year veteran. Shared data would help a lot of teams, esp. those with small or non-existant scouting teams.

Re: A New Way to Scout
 

Well, if you can't believe Mycroft, who can you trust? :) (You're just lucky Mycroft is too lazy to prove he's right.)

It's also true that not every team has a "Mycroft", so alternatives have to be made anyway. Our team is also developing a "crowd" type scouting system, which does seem the way to do it -- the more data in, the better. Though in general two people can always have opposite conclusions for the same bit of data.

Re: A New Way to Scout
 

It's a challenging task that students can learn from, true. But it's still duplicating the same work dozens of times at each event, and that carries an opportunity cost. Think of all of those hours of work put to use in other ways. I think the product on the field would improve.

I think we'll eventually have a system that makes stats from every event freely available, much like results and standings are available from FRC Spyder / TheBlueAlliance / usfirst.org / etc. When that happens, we'll still see people developing their own scouting systems to gather, aggregate, analyze, and report on additional types of scouting data.

Re: A New Way to Scout
 

No, all of Erich's information is about the current-ish control system. The 2006 control system was based on a 900mhz system. In other words, what are you talking about?


As for the questions earlier on why you can't run 2.4Ghz at low power - Simple, it's easier to say "No Wifi" than it is to say "No wifi except in this situation" because then you have to police that. It's more work for an already over worked group of folks.


Nah, I think we should be sharing how we manipulate the data. Data manipulation and visualization is as much engineering as learning CAD even if it is relegated to the back rooms of scouting meetings in FRC. Course, I'm biased…

Re: A New Way to Scout
 

Only 20? Try 50+!

Re: A New Way to Scout
 

I don't know if I would trust data that's gathered by "the community" for more than cursory overviews of robots. Our scouters all know the value of the data they collect, and see that it's of a real benefit to our team in qual matches and in alliance selection. That means that if they collect the data, it's good data. If an amorphous group of kids from different teams was working on the same data, they likely wouldn't have the same motivation to "get it right." You might end up with missing matches, people over counting their own teams, or just general slacking off. Of course, I agree having only one group of 7 per regional scout would be a ton less work, but potentially at the cost of data quality.

Re: A New Way to Scout
 

We have been considering this issue as well. Our team does not universally have smartphones or tablets...but in a one-to-one school they all have chromebooks. Is there any reason why we could not create a six-user WIRED network with a laptop as the "server" and write some html scouting form to use in an intra-net type setup?

And to the "data-sharing" issue...I definitely believe you should have to contribute to benefit. From my perspective, this benefits the rookie teams or smaller teams much more. Consider, if there was no investment required to obtain this data, the teams with the best scouting programs could hold all their own scouters for their own data, yet still get the larger group data. A smaller or less experienced team might not be able to generate good data themselves. On the other hand, if you need to contribute to the effort to gain access to the data the former team WILL participate because they won't want to risk missing something that everyone else will have access to. Their benefit will still be considerably less than the latter team, since the data they would have without cooperation would already be pretty good.

Re: A New Way to Scout
 

Data sharing will never replace good scouting. I would, however, trust data more than OPR for events I haven't attended in person, or scouted via webcast.

You pretty much described our scouting setup, with the caveat that we use paper sheets that are then entered via an HTML form, using node.js.

Let's just say that we found a new purpose for old robot D-Link bridges. ;)

Re: A New Way to Scout
 

Yea, you can do that with Wamp on a Windows Machine. EXTREMELY EFFECTIVE, because Wamp is simple to set up and configure. It is also reliable. We used a local Wamp server at 3 offseason events (IRI, Fall Classic, and Battle at the Border), and it never failed. PM me if you have any questions.

Re: A New Way to Scout
 

Sorry if anyone feels this is repetitive, but our team is looking for as many options to improve scouting, redundancy and such. Would the method described above have to be wired?

Re: A New Way to Scout
 

Yes, you need to have an ethernet cable running from each computer into a switch. You don't need to configure the switch in any fancy way though. Windows automatically assigns an IP (IPv6 I think?) that you can use to access the server from any computer wired to the switch.

Re: A New Way to Scout
 

You would be using IPv4, not v6, most likely. ;) . Also, my approach to getting this down would be to have an internet connection and to connect to my central database. Instead of a switch, a router with WiFi disabled would be the better approach because typically, routers are easier to manage because they are aimed at general purpose computing at home (Most standard users aren't geeks).

My database would allow every team to collaborate on their own database

Re: A New Way to Scout
 

I was just talking about a Wamp server. You CAN use a switch with a data connecting to connect to the internet in the same fashion. That is what we do for regionals.

Re: A New Way to Scout
 

Sorry for the double post. ;)
Here's what I want to accomplish in my scouting server:
Easy and intuitive interface so I don't have to teach everyone how to use it

Most typical data values (Goals, climb, etc)

Special features (Probably check buttons so PHP can sort it)

Global comments

Game comments (new eery game)

Ways to improve (new every game)

Graphs of the team's scores, improvement, highs, lows and where they get their points from (Probably using something like GraphViz)

Fast loading times (For the scouting stuff). At the end of scouting, where teams are deciding which teams to pick, the lag will get high because there will be large methods running, and my bandwidth will be hogged, sending all the images of the graphs

Every team's scoring graphs side by side, sorted by user criteria

Sorting between different teams. At competition, when over a hundred teams are competing, it is hard to decide which team shall get the crown. Why not let the computer do this for you?

A crapload of other features :D

I do not expect to get most of these features down for quite some time. I will start small and "grow" my program

:D :D :D

Re: A New Way to Scout
 

I feel like you could get all those graphs and data from an Excel spreadsheet on which the server could just draw from for live data on any given team at any point of time. Excel has a lot of power to it, and is probably the most common way of maintaining scouting data in FRC; if you want to make scouting information sharable with other teams, sticking with Excel may be a more universal choice compared with making your own interface.

Also, I don't think I understand the point of the graphs other than justa visual representation. Could you not just sort through data and get a ranking of teams based on different points of interest (maybe teleop, auto, endgame, consistencies) so then you wouldn't have to even worry about sending images? I think the benefit provided would be very little relative to how much effort and time it would take to get something like that working.

Re: A New Way to Scout
 

I was suggesting it as an interchange format. Think of it as an alternative to sending around the output of "mysqdump".

I feel like I'm missing something because I don't understand your suggested format at all:

• Is splitting text apart with a slash is easier than a space? It's been a while since I've written any PHP but why would slash be any different?
• Why have you chosen to omit the column name? Without it you have to add meta-data someplace else to describe what it is.
• How is representing climbs as 0-3 better than 0/10/20/30? They seem the same to me.

I also don't see why any of this would be a lot of work on the server.

Re: A New Way to Scout
 

Really? I have never experienced any lag or slowness from google drive, and it has been up 100% of the time...

Re: A New Way to Scout
 

Always worked in testing for us too. The problem is when it gets a large amount of data....
By the time any useful data can be extracted, there are about 120 lines of data. This is more than fine for Excel, but really causes problems on Drive. (Though, we had a few thousand formulas and a few sheets)

Re: A New Way to Scout
 

We run a VERY large gdocs based scouting system, that is doing a lot more than just averaging (such that when data is entered, a lot recalculates).

Even with this, it runs. I suggest you look into what runs fast/slow on gdoc, and thoroughly understand the process it uses when to update cells. Short answer is anytime cell A changes, every cell that references it upates, and that cycle repeats.

Don't use vlookups. At all. Even once. Just don't do it. Use index(match()) instead. Google it for good tutorials, this doesn't load the range per each cell called. Importrange/sortrange can also be used where vlookup would be used by some people with better results. More or less, don't use vlookup and don't use cell by cell calls to pass entire ranges if you can help it.

If you want me to take a look at your system and offer more specific advice I can. I'm not saying this to criticize you guys, but I'm 100% sure the issue here is implementation, not google docs.

Re: A New Way to Scout
 

Not to criticize, honest question - If you're already doing VLOOKUP and INDEX type operations why not invest a little bit of time to learn how to do this stuff with a full blown relational database? It'd be faster performing, more scalable, more reliable, and teach students useful skills. I guess the only real issue would be a simple front end but that's not that bad to do any more…


Disclaimer - my bread and butter is databases, Postgres, SQL-Server, Mongo, CouchDB… When all you've got is a hammer every problem looks like a nail, I'm genuinely curious why so many people seem to shoehorn excel into a job for an actual DB.

Re: A New Way to Scout
 

People/Resources. We can't dedicate programmers to this, and we're generally so short on programmers they are on the robot hte entire time.

Excel/Gdocs is a useful tool for ALL engineers, so in theory every student on the team should learn it.

A viable scouting system can be made in an hour easily if you're just doing averaging like most teams are. Less by an experienced user. You don't have to worry about potential errors/bugs you've introduced, etc... I don't buy the increased reliability argument.

Furthermore, the real big gain, is distribution. Every kid on the team has a google account, they have the scouting folder shared. Phones, tablets, laptops, etc... can all view it with no pain or hassle.

User interface as well, the UI is so much faster and easier to edit for people with no programming experience.

It has a lot of power for so little effort. It's the same reason why so many teams buy COTS shifters.

Re: A New Way to Scout
 

If you want good data, you can't force people to scout. I think we all agree on this point. Where this discussion leads from here depends on the goals of a collaborative scouting project. If the goal is for a few teams to work together to build some connections and gain a collective competitive advantage over the rest of the teams, then it might make sense to only allow access to the data to teams who contributed. That's not really the way I think of this. I think everyone would be better off if the data was freely available to anyone who wants it.

The way to do this would be to incorporate the data collecting into the event volunteer staff. Teams provide the volunteers, and those people take the requisite training just like they would if they want to become inspectors / referees / etc. Then they collect quality data in their roles as the official scorekeepers, and stats get released for public consumption. Just like they do in athletics.

The lack of this type of official scoring data is a deficiency in the FRC program. It's not like FRC doesn't have enough on its plate for the few people who actually get paid to run the operation, so that's not a criticism of them. I just think this is something that the community and FRC should address in the long term.

If we want FIRST to be loud, having scoring statistics readily available would help a bit. It gives people something to look at online to learn who's who, and that draws certain types of people in. It would make it possible to produce events with commentary and some statistics incorporated into the broadcast to add context and history to the matches to make them more interesting to casual observers and newbies. Without that stuff, it's harder for an observer to know or care about the difference between teams 6875 and 6587 and 6758.

Re: A New Way to Scout
 

Has anybody tried using IFTTT to text(SMS) data to a shared spreadsheet? Anyone with a cell phone could contribute to the data pool.

Scouters would need to open an IFTTT account and give it access to receive SMS/Texts (normal messaging rates apply), and Google account information to write to their Google Drive.

A Master Scout would need to designate a Spread Sheet file and share it with the other scouts. (All scouters would also automatically have access to the raw data in their Google Drive as well.)

If crowd scouting: On Thursday set up a scouting kiosk with a cellular hot spot for potential scouters to set up their account... or have the regional director e-mail all the teams ahead of time the set-up instructions.

PS. You can opt in the recipe to send/collect the phone number of the SMS sender. This would allow the master scout to text back anybody who is doing it wrong, or sort out all of their data and remove it from the pool. This would also release the phone numbers of the scouting participants to each other (no names attached). Not sure if that is a problem.

Re: A New Way to Scout
 

I use IFTTT for daily use, and I never realized it could do this. That's awesome.

I'm not going to grab a spreadsheet at the moment, but can you put data into hidden columns? Make the spreadsheet view only(And thus they can't directly open the hidden column) to everyone except the owner, and make a copy for the owner to check phone numbers in the hidden columns. From there, it's up to the owner to know when data is correct or faulty and let the users know how to improve their data input.

Re: A New Way to Scout
 

As far as I can tell, people who have a shared copy of the document cannot enter data unless they have the rights to edit the spreadsheet, including every column that IFTTT is trying to write to. I did verify that they can enter data while the column is hidden.

Even if the owner hides and protectes a column the shared users could still make a copy of the spreadsheet and then as the owner of the copy unhide and unprotect the information.

Re: A New Way to Scout
 

Sounds kind of like our old system.

That is probably it. We have a TON of vlookups, and that probably messed everything up. After finding the vlookups, we never tried to look for a more efficient method.
(Although I still am worried about losing the online data, after it consistently happened to us)


A few months ago, I would have loved for you to take a look at the system, but I invested quite literally my entire summer creating our own proprietary system, that somehow works. :yikes:

Thank you for the offer!

Re: A New Way to Scout
 

I talked with the lead programmer for CrowdScout today, and got some more details confirmed:

We'll use a database backend (MySQL or similar) to store the data. We'll have an API so that we can take data from many sources over in a standardized format; we'll publish this API for anyone to use with their software, though we may need to set up API keys to prevent server overloading from unintentional traffic floods blocking it for others.

We should have no issues accommodating all the data from every match in every tournament, with one database per year. It turns out FRC actually creates a fairly small amount of data in a season, somewhere in the millions of data points range (Relative to some of the data sets I've worked with, at least.)

The API will just provide a way to upload data; every year we'll have a suggested list of metrics to score by, as well as taking other inputs (we're still rolling ideas around for the custom inputs, so that's not final.)

Data will be distributed to teams by both raw data access (bulk exporting to some storage file) and data processing (running weighting algorithms and so forth). We envision teams as doing their own visualizations/processing beyond that, so we're not going to have a massively complicated GUI with a million different buttons; we could never match the ingenuity of teams in how they want to do their own processing.

(Someone could easily create a frontend and make it public, but we think that's better done locally and that our time and energy is better spent on making a rock-solid reliable backend and great API.)

Re: A New Way to Scout
 

Might I suggest taking a look at MongoDB. As it is schema less database you might have better luck, that way if teams don't collect some metric it won't be a complete mess for your DB and you don't have to have columns for everything. Simple clean JSON payloads :)

Re: A New Way to Scout
 

Our scouting software has been rebuilt annually since we started using software for scouting. This meant that one or two people spent over 300 collective hours during the season writing it. My plan for this year prioritizes maintenance, efficiency, and software design.

We are using:
node.js: Node is a tool for the modern web. It was created to be used for numerous client-server interactions in which small amounts of data were passed (think chatrooms). It is single-threaded, but extremely efficient. In all honesty, we could probably over a thousand FRC teams scouting concurrently with node (when used correctly). (Twitter did a study, and saw that using node for certain aspects of their system increased the amount of clients that a server could service from a few thousand to over a million).

express.js: This is a lightweight framework for node. Makes routing simple, among other small things.

mongodb w/ mongoose: mongo is the best option for use with node because it is nonblocking (unlike SQL). Both mongodb and node will be extremely efficient by servicing other requests while waiting for requests to be made and return. Mongoose is a nice wrapper library for mongo which makes it much easier, and adds schema support for mongo.

bootstrap: bootstrap is the most popular library on GitHub. We're using it as a frontend CSS framework to keep things simple.

angular.js: frontend JS framework made by Google. allows for two-way data binding, as well as easier maintenance of HTML. This is great, and eliminates some of the more boring code that deals with DOM manipulation.


In order to improve year on year, we have to use the most efficient system possible, while keeping it maintainable. I'm writing multiple wrapper APIs to further simply each of these parts of the frameworks.

Maybe we can work together to make a better overall crowdscouting system. I'd definitely want to see the details of how you currently do it.

Re: A New Way to Scout
 

Below is from last year before we went to wolds.

Re: A New Way to Scout
 

Khanh111: I've PMed you about a collaboration.

IPA is really cool, and I'd love to integrate it if 3138 is willing; not all teams would be able to submit all the data that IPA uses (paper based teams couldn't submit shot location, for instance), but we could definitely store that data on the platform we're developing, and the client would have access to both the specific data it needs as well as access the rest of the data.

Re: A New Way to Scout
 

With HTML5, you get drag n drop! So, record the position of robot shoot point(s) and store them in a database.

After thinking for a lot of time, I was thinking about which Database I should use. My hosting platform is a Raspberry Pi, so I do not have too much power. What database should I use for such an intensive application?

Single MySQL Database
Multiple MySQL Databases
Single Text File
Multiple Text Files
Something else?

Re: A New Way to Scout
 

The issue is for teams like mine that scout on paper sheets - there's no reasonable way to encode where the shots are coming from on a piece of paper (though I think OMAR, our scan-reading software, could probably do it if we wanted it to).

(Warning: speculation ahead, I could be wrong about this: )

Honestly, a Raspberry Pi doesn't have nearly enough power to run this scale an application. You need a lot more RAM for DBs; the 512MB on the Raspberry Pi probably won't be enough, I'd want at least 2-4GB for this size of database with the amount of data that we could be working with. Part of that is me wanting plenty extra for comfort, since the database may be able to run in 512MB, but part is the fact that I use more than 512MB while my web servers are idle on my personal server (admittedly not heavily optimized, but also not nearly as intensive as the sort of database we'll be using.)

Re: A New Way to Scout
 

Maaannn!!! What do you run on that poor server? My server uses just 192MB on standby, running all the crapware I keep on it!
See the screenshot. BTW, I was using SSH Display forwarding, so I was hogging up CPU resources and RAM by that. I do not keep LXDE running.

Lol. I was writing this post while taking the screenshot!

Re: A New Way to Scout
 

Caching. My personal server (which runs primarily team stuff) is borrowed from a friend and getting near 10 years old, so the bottleneck is the CPU with modern content; as such, I have as much cache as I can on everything.

I'm running several PHP-heavy sites (in Apache, nonetheless); both internal documentation for the team and a Moodle for our safety training. Neither are lightweight.

I could probably cut it down a bit (I haven't banished XFCE yet because I've had other priorities), but I have significantly more productive things to do with my time than optimize my RAM usage when there's still plenty free. I optimize the CPU usage and make sure nothing ever gets swapped first.

Re: A New Way to Scout
 

I forgot to show the details running as root. This has a greater view of what is sipping up the RAM. Here are some more details (in the screenshot). I never work on optimizing the RAM! If you want, I can give you a list of all the packages installed on the Pi! That might give you a bit of info on where everything is going! Also, what processor do you have? My Windows 2000 machine is about as powerful as my Pi, maybe even more. The only problem is that it has only 256 MB of RAM! Do you use SWAP? That might help ease the RAM usage of the system!

I, myself, dislike caching because it is hard to make sure it is updated! For the good loading speeds, I use the accelerator, APC. It doesn't cache the HTML. It caches the code to make PHP execute faster. I use the WordPress plugin, QuickCache to cache the HTML. Since it is based on WP, it will automatically update when I add/change/remove content!

Re: A New Way to Scout
 

That looks quite nice. Since I have a reasonable amount of RAM (1.5GB, low for a server but it works) I don't worry about it too much. It's running a 2.4 GHz Pentium 4 from about 2002 or 2003, which is why the CPU is the bottleneck.

Caching shouldn't be problematic; I'd recommend Varnish (which I would use if I served more static resources) if you haven't looked at it (though if QuickCache works well you may as well stick with it for WP). It can be annoying to set up but works really well. I also use APC, though I'll switch to the built-in opcache when I start using PHP 5.5.


To return to your earlier question, I think you'd best be served by a single MySQL database if you're on a Pi (though I'd recommend you check out the alternatives to MySQL too in case there are ones with better optimization.) I don't see a good reason to use multiple MySQL-style databases, and file-based databases have few advantages except simplicity.

Re: A New Way to Scout
 

Dude, I am running quite good specs on my Pi! I can have as many databases as I want because there is 500GB of storage! The storage is a hard drive (I boot off this too). I have a partition for the OS, a partition for the home server and a partition for web content. The web content partition is what holds everything served, and it is sized with 64GB (Though I will increase it to 128GB once I start running out of space)!

Also, I dislike Varnish because it is crap when you are running dynamic HTML! I doubt that wp-admin would even if varnish was installed properly! That is why I like PHP-based caching. It is a lot slower, but still gives a decent speed. It wouldn't be as fast as Varnish, but it would be fast enough. Otherwise, I was thinking that is it were possible, to serve the static parts like images, etc. from Nginx, but the dynamic content from Apache! However, that wouldn't help too much because my internet connection is 768kbps up!

Re: A New Way to Scout
 

i don't know if this method has been posted but what my team did at last regional was two people(we sent our driver and co-driver) would look at our next match and go scout the 5 teams (the opponents first) and on a sheet of paper ask them the statistics on their robot) after acquiring information we then go to our allies and ask for their information then we plan a strategy based on all information gathered. we keep all information so we can use it in future rounds/competitions :)

Re: A New Way to Scout
 

Some folks call that pit scouting; others call it useless.

You're actually better off having someone in the stands look at how those teams do in the last match before the one in which they play you, and getting the stats data down to the drivers as soon as it's collected, than asking them their stats, because their stats will almost invariably be without defense and/or estimates--and therefore higher than actual.

The other question would be, why are you having the drivers scout? I can understand if you're short on people, but IMO, drivers should not be scouting. They should be working with data they've been handed to develop a strategy.

Re: A New Way to Scout
 

i am head driver/scout i offered the idea of pit scouting thanks for info on scouting while match going on. i came up with the idea of going to each pit because we weren't scouting at all our first matches.i see what you mean though last year we had only 10 (lack of members)...

Re: A New Way to Scout
 

I'm our team's coach, head of mechanics, and part of our pit crew. Arguably, I should have the least free time of almost anyone on the team. But before every match, I take an in depth look at the scouting data for each team we're playing with and against, and then take a look at their robots in the pits. If we're with them, I ask to talk to their drivers or coach and talk strategy for a good five-ten minutes (sometimes multiple times before a match if it's a critical match).

Actually seeing the robots we'll be in a match with with my own eyes is really important. You can get a good sense of how their robot and team is doing just looking for yourself. That robot with a 2 CIM Kitbot drive and ugly painted bumpers? Might not going to move. Neat looking bot with only the programmer working in the pits? Could be someone to look out for. Afterwards, I actually go and talk with them, to see if we can help them get their bot ready, or just to talk strategy. It's important that I'm the one doing this because I'm the one who's going to have to face them or work with them on the field. Also, because I know what it's like to play in a match firsthand, I know what I need to look for to gauge their robot's performance.

I do agree that just asking teams "how they've been doing" and basing your strategy on that isn't the best plan. Team's overestimate, have a proximity bias, exhibit a halo effect, yada yada yada. Like you said, it's critical to base your assumptions on how their robot will do on your own (well collected) scouting data. On the other hand, actually seeing the robots and talking to their people (as a driver) can really give a better perspective on what your strategy should be.