Quote:
Originally Posted by 1306scouting
I talked with the lead programmer for CrowdScout today, and got some more details confirmed:
We'll use a database backend (MySQL or similar) to store the data. We'll have an API so that we can take data from many sources over in a standardized format; we'll publish this API for anyone to use with their software, though we may need to set up API keys to prevent server overloading from unintentional traffic floods blocking it for others.
We should have no issues accommodating all the data from every match in every tournament, with one database per year. It turns out FRC actually creates a fairly small amount of data in a season, somewhere in the millions of data points range (Relative to some of the data sets I've worked with, at least.)
The API will just provide a way to upload data; every year we'll have a suggested list of metrics to score by, as well as taking other inputs (we're still rolling ideas around for the custom inputs, so that's not final.)
Data will be distributed to teams by both raw data access (bulk exporting to some storage file) and data processing (running weighting algorithms and so forth). We envision teams as doing their own visualizations/processing beyond that, so we're not going to have a massively complicated GUI with a million different buttons; we could never match the ingenuity of teams in how they want to do their own processing.
(Someone could easily create a frontend and make it public, but we think that's better done locally and that our time and energy is better spent on making a rock-solid reliable backend and great API.)
|
Our scouting software has been rebuilt annually since we started using software for scouting. This meant that one or two people spent over 300 collective hours during the season writing it. My plan for this year prioritizes maintenance, efficiency, and software design.
We are using:
node.js: Node is a tool for the modern web. It was created to be used for numerous client-server interactions in which small amounts of data were passed (think chatrooms). It is single-threaded, but extremely efficient. In all honesty, we could probably over a thousand FRC teams scouting concurrently with node (when used correctly). (Twitter did a study, and saw that using node for certain aspects of their system increased the amount of clients that a server could service from a few thousand to over a million).
express.js: This is a lightweight framework for node. Makes routing simple, among other small things.
mongodb w/ mongoose: mongo is the best option for use with node because it is nonblocking (unlike SQL). Both mongodb and node will be extremely efficient by servicing other requests while waiting for requests to be made and return. Mongoose is a nice wrapper library for mongo which makes it much easier, and adds schema support for mongo.
bootstrap: bootstrap is the most popular library on GitHub. We're using it as a frontend CSS framework to keep things simple.
angular.js: frontend JS framework made by Google. allows for two-way data binding, as well as easier maintenance of HTML. This is great, and eliminates some of the more boring code that deals with DOM manipulation.
In order to improve year on year, we have to use the most efficient system possible, while keeping it maintainable. I'm writing multiple wrapper APIs to further simply each of these parts of the frameworks.
Maybe we can work together to make a better overall crowdscouting system. I'd definitely want to see the details of how you currently do it.