The Blue Alliance Data Scraper


#1

My team is looking for a data scraper for the Blue Alliance website so we can efficiently collect information on the other teams attending Detroit champs.

If anyone has any info. or code that we could use as a sample, please let me know! Team 2265 thanks you!


#2

No need to scrape, the blue alliance offers a full rest api
https://www.thebluealliance.com/apidocs


#3

The Blue Alliance has a very full featured API that you can read about here.

If you want to use Python, (https://github.com/saikiranra/frcstat) that can do some specific stuff, but might not be as useful as the raw API.


#4

Alternatively, you can interface with TBA data in Excel using the BigQuery dataset (details about this are on the API documentation page)


#5

Thanks for all of the responses! We looked into the Blue Alliance’s API but we don’t have any experience with it. If anyone has sample code we could view, I would appreciate it!


#6

**How about a downloadable Excel spreadsheet file with all the raw data in it for weeks 1 thru 7 ?

See the 3rd link at this post.**


#7

Here’s a small sample in Python that will get all the matches at a given event.


request_url = tba_base_url + "event/" + event_key + "/matches"
payload = {"X-TBA-Auth-Key": your_api_key_here}
request = requests.get(request_url, params=payload).json()

Then you can just get specifics with something like this:

red_score = match"score_breakdown"]"red"]"totalPoints"]

For other endpoints (there are lots and lots of them), see the documentation here. It’s super duper awesomely useful. Also if you have any other questions, feel free to pm me or just ask here!


#8

Allow me to present, the TBA API, in library form! You have your choice of language (currently 5 to pick from):

They’re auto-generated from the TBA API Swagger spec, so they’re not fully tested and I can’t guarantee they’ll work flawlessly, but they work. I know a few people that have used them.

And obviously, if you have any issues with them, please let me know!


#9

If you write in perl, it also works well. I used libraries HTTP::Response, JSON:PP, and LWP::UserAgent to call the APIs & process the data. If you want details a pm will work.

TBA has pretty extensive help, with examples in a number of languages.


#10

I realize that the previous post is nearly a year old but this seemed to most relevant place to ask this. I want to perform some data analysis on team data from last year and looking for a source of the data. When I go to the TBA and look at what they have, the BigQuery dataset seems to no longer be there. When I look at the CSVs they have, I am not seeing the level of detail I want.

Here is an example of the type of data I am looking for. To be honest, I wouldn’t mind even more detailed information that looks at the types of data as in the link below but down at the individual match level but I am not too greedy. If needed I can probably do some “cut and paste”.

https://www.thebluealliance.com/event/2018inmis#rankings


#11

The API is the best way to access data.