Go to Post These folks have bigger hearts than most. I think they grow them that way in Indiana. - JaneYoung [more]
Home
Go Back   Chief Delphi > Competition > Rules/Strategy
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 04-16-2018, 03:48 PM
girlinstem girlinstem is offline
Registered User
AKA: Aviva S.
FRC #2265 (Fe Maidens)
Team Role: Programmer
 
Join Date: Mar 2018
Rookie Year: 2017
Location: New York CIty
Posts: 2
girlinstem is an unknown quantity at this point
The Blue Alliance Data Scraper

My team is looking for a data scraper for the Blue Alliance website so we can efficiently collect information on the other teams attending Detroit champs.

If anyone has any info. or code that we could use as a sample, please let me know! Team 2265 thanks you!
Reply With Quote
  #2   Spotlight this post!  
Unread 04-16-2018, 03:53 PM
pjt0620 pjt0620 is offline
Registered User
FRC #2832
 
Join Date: Sep 2016
Rookie Year: 2007
Location: Livonia MI,
Posts: 8
pjt0620 is an unknown quantity at this point
Re: The Blue Alliance Data Scraper

No need to scrape, the blue alliance offers a full rest api
https://www.thebluealliance.com/apidocs
Reply With Quote
  #3   Spotlight this post!  
Unread 04-16-2018, 03:55 PM
saikiranra's Avatar
saikiranra saikiranra is offline
Zot Zot
AKA: Saikiran Ramanan
FRC #3476 (Code Orange)
Team Role: Mentor
 
Join Date: Oct 2012
Rookie Year: 2011
Location: Irvine, CA
Posts: 269
saikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond reputesaikiranra has a reputation beyond repute
Re: The Blue Alliance Data Scraper

The Blue Alliance has a very full featured API that you can read about here.

If you want to use Python, I have a library that can do some specific stuff, but might not be as useful as the raw API.
__________________
2014 - Current: Team 3476 Mentor
2011 - 2014: Team 696 Student
Reply With Quote
  #4   Spotlight this post!  
Unread 04-16-2018, 04:24 PM
Zach O's Avatar
Zach O Zach O is offline
FRC Emcee
AKA: @FRCZach
no team
 
Join Date: Jan 2009
Rookie Year: 2009
Location: Atlanta, GA
Posts: 514
Zach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond reputeZach O has a reputation beyond repute
Re: The Blue Alliance Data Scraper

Alternatively, you can interface with TBA data in Excel using the BigQuery dataset (details about this are on the API documentation page)
__________________
Follow me @FRCZach
Reply With Quote
  #5   Spotlight this post!  
Unread 04-16-2018, 04:29 PM
girlinstem girlinstem is offline
Registered User
AKA: Aviva S.
FRC #2265 (Fe Maidens)
Team Role: Programmer
 
Join Date: Mar 2018
Rookie Year: 2017
Location: New York CIty
Posts: 2
girlinstem is an unknown quantity at this point
Re: The Blue Alliance Data Scraper

Thanks for all of the responses! We looked into the Blue Alliance's API but we don't have any experience with it. If anyone has sample code we could view, I would appreciate it!
Reply With Quote
  #6   Spotlight this post!  
Unread 04-16-2018, 04:33 PM
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 9,109
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
Re: The Blue Alliance Data Scraper

Quote:
Originally Posted by girlinstem View Post
Thanks for all of the responses! We looked into the Blue Alliance's API but we don't have any experience with it.
How about a downloadable Excel spreadsheet file with all the raw data in it for weeks 1 thru 7 ?

See the 3rd link at this post.




Last edited by Ether : 04-16-2018 at 04:50 PM.
Reply With Quote
  #7   Spotlight this post!  
Unread 04-16-2018, 04:47 PM
Hitchhiker 42's Avatar
Hitchhiker 42 Hitchhiker 42 is offline
Roboter
AKA: Mark Lavrentyev
FRC #4557 (FullMetal Falcons)
Team Role: Programmer
 
Join Date: Oct 2015
Rookie Year: 2015
Location: Connecticut
Posts: 925
Hitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond reputeHitchhiker 42 has a reputation beyond repute
Re: The Blue Alliance Data Scraper

Quote:
Originally Posted by girlinstem View Post
Thanks for all of the responses! We looked into the Blue Alliance's API but we don't have any experience with it. If anyone has sample code we could view, I would appreciate it!
Here's a small sample in Python that will get all the matches at a given event.

Code:
request_url = tba_base_url + "event/" + event_key + "/matches"
payload = {"X-TBA-Auth-Key": your_api_key_here}
request = requests.get(request_url, params=payload).json()
Then you can just get specifics with something like this:
Code:
red_score = match["score_breakdown"]["red"]["totalPoints"]
For other endpoints (there are lots and lots of them), see the documentation here. It's super duper awesomely useful. Also if you have any other questions, feel free to pm me or just ask here!
__________________
Don't forget your towel!

Reply With Quote
  #8   Spotlight this post!  
Unread 04-16-2018, 05:28 PM
Fletch1373's Avatar
Fletch1373 Fletch1373 is offline
Registered User
AKA: Fletch
FRC #1126 (SPARX(1126) and Panthers(3181))
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2003
Location: Webster, NY
Posts: 288
Fletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant futureFletch1373 has a brilliant future
Re: The Blue Alliance Data Scraper

Quote:
Originally Posted by girlinstem View Post
Thanks for all of the responses! We looked into the Blue Alliance's API but we don't have any experience with it. If anyone has sample code we could view, I would appreciate it!
Allow me to present, the TBA API, in library form! You have your choice of language (currently 5 to pick from):

They're auto-generated from the TBA API Swagger spec, so they're not fully tested and I can't guarantee they'll work flawlessly, but they work. I know a few people that have used them.

And obviously, if you have any issues with them, please let me know!
__________________
Student:
<05-08: FRC1373> <04: FRC0213>
Mentor:
<18-??: FRC1126> <15-??: FRC3181> <12-14: FRC0073> <11-12: FRC3555> <09-10: FRC0809>
Volunteer:
<FTAA> <CSA> <Scorekeeper> <Robot Inspector> <and many more...>
2018 Tour:
NYUT[CSA] > CTWAT[FTAA] > NYRO[CSA] > DetCMP[maybe...?]
Reply With Quote
  #9   Spotlight this post!  
Unread 04-16-2018, 06:23 PM
Jaxom Jaxom is offline
Registered User
AKA: Michael Hartwig
FRC #1986 (Team Titanium)
Team Role: Mentor
 
Join Date: Apr 2010
Rookie Year: 2008
Location: Lee's Summit, MO
Posts: 414
Jaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant future
Re: The Blue Alliance Data Scraper

If you write in perl, it also works well. I used libraries HTTP::Response, JSONP, and LWP::UserAgent to call the APIs & process the data. If you want details a pm will work.

TBA has pretty extensive help, with examples in a number of languages.
__________________

RI 2010-present, GKC LRI 2018

Mentor 2008-present http://www.teamtitanium.org/

Last edited by Jaxom : 04-16-2018 at 06:23 PM. Reason: Mention TBA examples
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 05:49 AM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi