OCCRA
Go to Post Life is unfair. Learn to deal with it or be unhappy. - Joe Johnson [more]
Home
Go Back   Chief Delphi > Competition > Rules/Strategy > Scouting
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 01-22-2018, 03:23 PM
danieltcalkin's Avatar
danieltcalkin danieltcalkin is offline
Registered User
AKA: Celebrimbor
FRC #4910 (East Cobb Robotics)
Team Role: CAD
 
Join Date: Feb 2016
Rookie Year: 2015
Location: Marietta, GA
Posts: 54
danieltcalkin has a spectacular aura aboutdanieltcalkin has a spectacular aura aboutdanieltcalkin has a spectacular aura about
Question Compiling raw data from FIRST

Hi, so I (not my team! I'm still in the collect info before the pitch phase) am thinking of doing a scouting project using the raw data from recent games. The raw data can be confusing, and I was wondering if anyone else has ever done a similar project. I'm thinking of making a massive spreadsheet of the past 5 years or so. Has anyone ever done anything similar? I'm not asking you to give me your final results if you don't want to, but just anything you could give would be great. I'm not a programmer, so I can't read Python or Java sadly. If all you can say is it can't be done without knowing some sort of programming language then I still need to hear it before I attempt to compile all this raw data alone lol.
__________________
Building Good Humans since 2013




Reply With Quote
  #2   Spotlight this post!  
Unread 01-22-2018, 03:37 PM
jaredhk's Avatar
jaredhk jaredhk is offline
No longer old enough to volunteer
AKA: Jared Hasen-Klein
no team (Volunteer, 1836 Alum, and extreme instigator)
 
Join Date: Feb 2015
Rookie Year: 2009
Location: 🥑Southern California
Posts: 478
jaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond reputejaredhk has a reputation beyond repute
Re: Compiling raw data from FIRST

Consider taking a look at BigQuery from The Blue Alliance (blog about it here).

If that's a little too complicated, some more raw data can be found on Github/the-blue-alliance/the-blue-alliance-data - not sure if this is any better than the FIRST data.
__________________
JARED HASEN-KLEIN
w: jaredhk.me // Facebook @jhasenklein // Instagram @rationalskeptic
―――
Dean's List Finalist // The Blue Alliance developer // Los Angeles FLL VOY 2017 // CPP 2022 // *52 FIRST Events and Counting! (16,812 miles) * // Team 1836: The MilkenKnights Class of 2018

―――
FREE ROBOTICS RESOURCES // Admin of: FB's FIRST ROBOTICS NETWORK // How to: FIRST DIVERSITY, EQUITY, & INCLUSION
―――
"I am, as I've said, merely competent. But in an age of incompetence, that makes me extraordinary." -Billy Joel
Reply With Quote
  #3   Spotlight this post!  
Unread 01-22-2018, 03:40 PM
bigbeezy's Avatar
bigbeezy bigbeezy is offline
Registered User
AKA: Bryan Rickards
no team (1592 Bionic Tigers / 2338 Gear it Forward)
Team Role: Mentor
 
Join Date: Dec 2006
Rookie Year: 2005
Location: Peoria, IL
Posts: 380
bigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud of
Re: Compiling raw data from FIRST

I started looking at this in the fall. I'm certainly not a programmer but can understand the logic and am getting better with VBA within Excel. Through online guides and searches I was able to figure out how to pull the data from the API site into an excel doc and then use that data to create a scouting program. Hope to modify it for 2018 once the FMS data variables are published.

I know that I learned a TON doing this fun little project. I'll try to post part of my code tonight.
__________________
Team 1592 Bionic Tigers -- Driver 2005-2008
Winner - Florida Regional 2005
Finalist - Newton Division 2007
Winner - Colorado Regional 2008
Florida State University - BS Mechanical Engineering
Team 2338 Gear it Forward -- Mentor/Drive Coach 2013-2017
Winner - Wisconsin Regional 2013
Finalist - Midwest Regional 2014, 2015, 2016
Winner - Archimedes Sub-Division 2015
Chairman's Award - Midwest Regional 2015, 2016, 2017
Reply With Quote
  #4   Spotlight this post!  
Unread 01-22-2018, 10:00 PM
bigbeezy's Avatar
bigbeezy bigbeezy is offline
Registered User
AKA: Bryan Rickards
no team (1592 Bionic Tigers / 2338 Gear it Forward)
Team Role: Mentor
 
Join Date: Dec 2006
Rookie Year: 2005
Location: Peoria, IL
Posts: 380
bigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud of
Re: Compiling raw data from FIRST

As promised, here is the VBA code I have for pulling in the data from API into Excel. You will need your own account and token, instructions for that are on the site you linked. I'm sure there are cleaner, more efficient ways to code this, but it works.

Code:
Sub GetMatchData()

Dim UI As Worksheet: Set UI = Sheets("User Interface")
Dim MD As Worksheet: Set MD = Sheets("MatchData")
Dim MI As Worksheet: Set MI = Sheets("MatchInfo")
Dim DB As Worksheet: Set DB = Sheets("Database")
Dim Back As Worksheet: Set Back = Sheets("Background")

'clear match data from previous search
MD.Range(MD.Cells(3, 1), MD.Cells(10000, 36)).ClearContents

Dim strURL As String, year As Integer, eventcode As String, eventname As String, matchtype As String
Dim authUser As String, authPass As String

'example: https://frc-api.firstinspires.org/v2.0/2017/scores/ILCH/qual

strURL = "https://frc-api.firstinspires.org/v2.0/"
    year = 2017
    strURL = strURL & year & "/scores/"
    eventname = UI.Range("EventName") 'event name entered in User Interface
    eventcode = WorksheetFunction.VLookup(eventname, Back.Range("A2:B164"), 2, False) 'Event code lookup
    strURL = strURL & eventcode & "/"
    matchtype = "qual"
    strURL = strURL & matchtype

MD.Cells(1, 2).Value = eventcode

authUser = "xxxxxxxxxx" 'enter your user ID
authPass = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx" 'enter your token

'YouTube Tutorial: https://www.youtube.com/watch?v=mmT4qxOnEZk
Dim hreq As Object
Set hreq = CreateObject("Microsoft.XMLHTTP")
hreq.Open "GET", strURL, False
hreq.SetRequestHeader "Content-Type", "application/xml"
hreq.SetRequestHeader "Accept", "application/xml"
hreq.SetRequestHeader "Authorization", "Basic " + Base64Encode(authUser + ":" + authPass)
hreq.Send

Dim strResp As String
strResp = hreq.ResponseText

Dim xmlDoc As New MSXML2.DOMDocument
If Not xmlDoc.LoadXML(strResp) Then
    MsgBox "Load Error"
End If

Dim Match As MSXML2.IXMLDOMNode
Dim Alliance As MSXML2.IXMLDOMNode

Dim intRow As Integer
intRow = 3

For Each Match In xmlDoc.getElementsByTagName("Score_2017")
    MD.Cells(intRow, 1).Value = Match.SelectNodes("matchLevel")(0).Text
    MD.Cells(intRow, 2).Value = Match.SelectNodes("matchNumber")(0).Text
    MD.Cells(intRow + 1, 1).Value = Match.SelectNodes("matchLevel")(0).Text
    MD.Cells(intRow + 1, 2).Value = Match.SelectNodes("matchNumber")(0).Text
    intRow = intRow + 2
Next

intRow = 3
For Each Alliance In xmlDoc.getElementsByTagName("Alliance")
    MD.Cells(intRow, 3) = Alliance.SelectNodes("alliance")(0).Text
    MD.Cells(intRow, 4) = Alliance.SelectNodes("robot1Auto")(0).Text
    MD.Cells(intRow, 5) = Alliance.SelectNodes("robot2Auto")(0).Text
    MD.Cells(intRow, 6) = Alliance.SelectNodes("robot3Auto")(0).Text
    MD.Cells(intRow, 7) = Alliance.SelectNodes("autoFuelLow")(0).Text
    MD.Cells(intRow, 8) = Alliance.SelectNodes("autoFuelHigh")(0).Text
    MD.Cells(intRow, 9) = Alliance.SelectNodes("rotor1Auto")(0).Text
    MD.Cells(intRow, 10) = Alliance.SelectNodes("rotor2Auto")(0).Text
    MD.Cells(intRow, 11) = Alliance.SelectNodes("rotor1Engaged")(0).Text
    MD.Cells(intRow, 12) = Alliance.SelectNodes("rotor2Engaged")(0).Text
    MD.Cells(intRow, 13) = Alliance.SelectNodes("rotor3Engaged")(0).Text
    MD.Cells(intRow, 14) = Alliance.SelectNodes("rotor4Engaged")(0).Text
    MD.Cells(intRow, 15) = Alliance.SelectNodes("teleopFuelLow")(0).Text
    MD.Cells(intRow, 16) = Alliance.SelectNodes("teleopFuelHigh")(0).Text
    MD.Cells(intRow, 17) = Alliance.SelectNodes("touchpadNear")(0).Text
    MD.Cells(intRow, 18) = Alliance.SelectNodes("touchpadMiddle")(0).Text
    MD.Cells(intRow, 19) = Alliance.SelectNodes("touchpadFar")(0).Text
    MD.Cells(intRow, 20) = Alliance.SelectNodes("kPaRankingPointAchieved")(0).Text
    MD.Cells(intRow, 21) = Alliance.SelectNodes("rotorRankingPointAchieved")(0).Text
    MD.Cells(intRow, 22) = Alliance.SelectNodes("foulCount")(0).Text
    MD.Cells(intRow, 23) = Alliance.SelectNodes("techFoulCount")(0).Text
    MD.Cells(intRow, 24) = Alliance.SelectNodes("autoPoints")(0).Text
    MD.Cells(intRow, 25) = Alliance.SelectNodes("autoMobilityPoints")(0).Text
    MD.Cells(intRow, 26) = Alliance.SelectNodes("autoRotorPoints")(0).Text
    MD.Cells(intRow, 27) = Alliance.SelectNodes("autoFuelPoints")(0).Text
    MD.Cells(intRow, 28) = Alliance.SelectNodes("teleopPoints")(0).Text
    MD.Cells(intRow, 29) = Alliance.SelectNodes("teleopFuelPoints")(0).Text
    MD.Cells(intRow, 30) = Alliance.SelectNodes("teleopRotorPoints")(0).Text
    MD.Cells(intRow, 31) = Alliance.SelectNodes("teleopTakeoffPoints")(0).Text
    MD.Cells(intRow, 32) = Alliance.SelectNodes("kPaBonusPoints")(0).Text
    MD.Cells(intRow, 33) = Alliance.SelectNodes("rotorBonusPoints")(0).Text
    MD.Cells(intRow, 34) = Alliance.SelectNodes("adjustPoints")(0).Text
    MD.Cells(intRow, 35) = Alliance.SelectNodes("foulPoints")(0).Text
    MD.Cells(intRow, 36) = Alliance.SelectNodes("totalPoints")(0).Text
    
    intRow = intRow + 1
        
Next

Set hreq = Nothing
Set xmlDoc = Nothing

End Sub
__________________
Team 1592 Bionic Tigers -- Driver 2005-2008
Winner - Florida Regional 2005
Finalist - Newton Division 2007
Winner - Colorado Regional 2008
Florida State University - BS Mechanical Engineering
Team 2338 Gear it Forward -- Mentor/Drive Coach 2013-2017
Winner - Wisconsin Regional 2013
Finalist - Midwest Regional 2014, 2015, 2016
Winner - Archimedes Sub-Division 2015
Chairman's Award - Midwest Regional 2015, 2016, 2017
Reply With Quote
  #5   Spotlight this post!  
Unread 01-23-2018, 09:27 AM
danieltcalkin's Avatar
danieltcalkin danieltcalkin is offline
Registered User
AKA: Celebrimbor
FRC #4910 (East Cobb Robotics)
Team Role: CAD
 
Join Date: Feb 2016
Rookie Year: 2015
Location: Marietta, GA
Posts: 54
danieltcalkin has a spectacular aura aboutdanieltcalkin has a spectacular aura aboutdanieltcalkin has a spectacular aura about
Talking Re: Compiling raw data from FIRST

Quote:
Originally Posted by bigbeezy View Post
As promised, here is the VBA code I have for pulling in the data from API into Excel. You will need your own account and token, instructions for that are on the site you linked. I'm sure there are cleaner, more efficient ways to code this, but it works.

Code:
Sub GetMatchData()

Dim UI As Worksheet: Set UI = Sheets("User Interface")
Dim MD As Worksheet: Set MD = Sheets("MatchData")
Dim MI As Worksheet: Set MI = Sheets("MatchInfo")
Dim DB As Worksheet: Set DB = Sheets("Database")
Dim Back As Worksheet: Set Back = Sheets("Background")

'clear match data from previous search
MD.Range(MD.Cells(3, 1), MD.Cells(10000, 36)).ClearContents

Dim strURL As String, year As Integer, eventcode As String, eventname As String, matchtype As String
Dim authUser As String, authPass As String

'example: https://frc-api.firstinspires.org/v2.0/2017/scores/ILCH/qual

strURL = "https://frc-api.firstinspires.org/v2.0/"
    year = 2017
    strURL = strURL & year & "/scores/"
    eventname = UI.Range("EventName") 'event name entered in User Interface
    eventcode = WorksheetFunction.VLookup(eventname, Back.Range("A2:B164"), 2, False) 'Event code lookup
    strURL = strURL & eventcode & "/"
    matchtype = "qual"
    strURL = strURL & matchtype

MD.Cells(1, 2).Value = eventcode

authUser = "xxxxxxxxxx" 'enter your user ID
authPass = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx" 'enter your token

'YouTube Tutorial: https://www.youtube.com/watch?v=mmT4qxOnEZk
Dim hreq As Object
Set hreq = CreateObject("Microsoft.XMLHTTP")
hreq.Open "GET", strURL, False
hreq.SetRequestHeader "Content-Type", "application/xml"
hreq.SetRequestHeader "Accept", "application/xml"
hreq.SetRequestHeader "Authorization", "Basic " + Base64Encode(authUser + ":" + authPass)
hreq.Send

Dim strResp As String
strResp = hreq.ResponseText

Dim xmlDoc As New MSXML2.DOMDocument
If Not xmlDoc.LoadXML(strResp) Then
    MsgBox "Load Error"
End If

Dim Match As MSXML2.IXMLDOMNode
Dim Alliance As MSXML2.IXMLDOMNode

Dim intRow As Integer
intRow = 3

For Each Match In xmlDoc.getElementsByTagName("Score_2017")
    MD.Cells(intRow, 1).Value = Match.SelectNodes("matchLevel")(0).Text
    MD.Cells(intRow, 2).Value = Match.SelectNodes("matchNumber")(0).Text
    MD.Cells(intRow + 1, 1).Value = Match.SelectNodes("matchLevel")(0).Text
    MD.Cells(intRow + 1, 2).Value = Match.SelectNodes("matchNumber")(0).Text
    intRow = intRow + 2
Next

intRow = 3
For Each Alliance In xmlDoc.getElementsByTagName("Alliance")
    MD.Cells(intRow, 3) = Alliance.SelectNodes("alliance")(0).Text
    MD.Cells(intRow, 4) = Alliance.SelectNodes("robot1Auto")(0).Text
    MD.Cells(intRow, 5) = Alliance.SelectNodes("robot2Auto")(0).Text
    MD.Cells(intRow, 6) = Alliance.SelectNodes("robot3Auto")(0).Text
    MD.Cells(intRow, 7) = Alliance.SelectNodes("autoFuelLow")(0).Text
    MD.Cells(intRow, 8) = Alliance.SelectNodes("autoFuelHigh")(0).Text
    MD.Cells(intRow, 9) = Alliance.SelectNodes("rotor1Auto")(0).Text
    MD.Cells(intRow, 10) = Alliance.SelectNodes("rotor2Auto")(0).Text
    MD.Cells(intRow, 11) = Alliance.SelectNodes("rotor1Engaged")(0).Text
    MD.Cells(intRow, 12) = Alliance.SelectNodes("rotor2Engaged")(0).Text
    MD.Cells(intRow, 13) = Alliance.SelectNodes("rotor3Engaged")(0).Text
    MD.Cells(intRow, 14) = Alliance.SelectNodes("rotor4Engaged")(0).Text
    MD.Cells(intRow, 15) = Alliance.SelectNodes("teleopFuelLow")(0).Text
    MD.Cells(intRow, 16) = Alliance.SelectNodes("teleopFuelHigh")(0).Text
    MD.Cells(intRow, 17) = Alliance.SelectNodes("touchpadNear")(0).Text
    MD.Cells(intRow, 18) = Alliance.SelectNodes("touchpadMiddle")(0).Text
    MD.Cells(intRow, 19) = Alliance.SelectNodes("touchpadFar")(0).Text
    MD.Cells(intRow, 20) = Alliance.SelectNodes("kPaRankingPointAchieved")(0).Text
    MD.Cells(intRow, 21) = Alliance.SelectNodes("rotorRankingPointAchieved")(0).Text
    MD.Cells(intRow, 22) = Alliance.SelectNodes("foulCount")(0).Text
    MD.Cells(intRow, 23) = Alliance.SelectNodes("techFoulCount")(0).Text
    MD.Cells(intRow, 24) = Alliance.SelectNodes("autoPoints")(0).Text
    MD.Cells(intRow, 25) = Alliance.SelectNodes("autoMobilityPoints")(0).Text
    MD.Cells(intRow, 26) = Alliance.SelectNodes("autoRotorPoints")(0).Text
    MD.Cells(intRow, 27) = Alliance.SelectNodes("autoFuelPoints")(0).Text
    MD.Cells(intRow, 28) = Alliance.SelectNodes("teleopPoints")(0).Text
    MD.Cells(intRow, 29) = Alliance.SelectNodes("teleopFuelPoints")(0).Text
    MD.Cells(intRow, 30) = Alliance.SelectNodes("teleopRotorPoints")(0).Text
    MD.Cells(intRow, 31) = Alliance.SelectNodes("teleopTakeoffPoints")(0).Text
    MD.Cells(intRow, 32) = Alliance.SelectNodes("kPaBonusPoints")(0).Text
    MD.Cells(intRow, 33) = Alliance.SelectNodes("rotorBonusPoints")(0).Text
    MD.Cells(intRow, 34) = Alliance.SelectNodes("adjustPoints")(0).Text
    MD.Cells(intRow, 35) = Alliance.SelectNodes("foulPoints")(0).Text
    MD.Cells(intRow, 36) = Alliance.SelectNodes("totalPoints")(0).Text
    
    intRow = intRow + 1
        
Next

Set hreq = Nothing
Set xmlDoc = Nothing

End Sub

Thanks! This is exactly what I need. Now for the reading to start.
__________________
Building Good Humans since 2013




Reply With Quote
  #6   Spotlight this post!  
Unread 01-23-2018, 09:28 AM
danieltcalkin's Avatar
danieltcalkin danieltcalkin is offline
Registered User
AKA: Celebrimbor
FRC #4910 (East Cobb Robotics)
Team Role: CAD
 
Join Date: Feb 2016
Rookie Year: 2015
Location: Marietta, GA
Posts: 54
danieltcalkin has a spectacular aura aboutdanieltcalkin has a spectacular aura aboutdanieltcalkin has a spectacular aura about
Talking Re: Compiling raw data from FIRST

Quote:
Originally Posted by jaredhk View Post
Consider taking a look at BigQuery from The Blue Alliance (blog about it here).

If that's a little too complicated, some more raw data can be found on Github/the-blue-alliance/the-blue-alliance-data - not sure if this is any better than the FIRST data.
These resources are perfect! Thank you so much.
__________________
Building Good Humans since 2013




Reply With Quote
  #7   Spotlight this post!  
Unread 02-16-2018, 11:10 PM
Bucephalus Bucephalus is offline
Registered User
FRC #5985 (Project Bucephalus)
Team Role: Mentor
 
Join Date: Dec 2015
Rookie Year: 2010
Location: Wollongong
Posts: 5
Bucephalus is an unknown quantity at this point
Re: Compiling raw data from FIRST

Thanks for sharing!

One question: We're attempting something similar with VBA. However, we keep getting asked for authentication from https://frc-api.firstinspires.org/. The username/authentication token don't seem to be accepted. We suspect this may be an issue with our base64 encoding.

Is this an issue you encountered?

Andrew
Reply With Quote
  #8   Spotlight this post!  
Unread 02-17-2018, 12:21 AM
bigbeezy's Avatar
bigbeezy bigbeezy is offline
Registered User
AKA: Bryan Rickards
no team (1592 Bionic Tigers / 2338 Gear it Forward)
Team Role: Mentor
 
Join Date: Dec 2006
Rookie Year: 2005
Location: Peoria, IL
Posts: 380
bigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud ofbigbeezy has much to be proud of
Re: Compiling raw data from FIRST

Have you created a login and recieved a token? Takes a couple days to get via email.
__________________
Team 1592 Bionic Tigers -- Driver 2005-2008
Winner - Florida Regional 2005
Finalist - Newton Division 2007
Winner - Colorado Regional 2008
Florida State University - BS Mechanical Engineering
Team 2338 Gear it Forward -- Mentor/Drive Coach 2013-2017
Winner - Wisconsin Regional 2013
Finalist - Midwest Regional 2014, 2015, 2016
Winner - Archimedes Sub-Division 2015
Chairman's Award - Midwest Regional 2015, 2016, 2017
Reply With Quote
  #9   Spotlight this post!  
Unread 05-16-2018, 11:05 AM
cvbriggler cvbriggler is offline
Registered User
FRC #1425 (Error Code Xero)
Team Role: Mentor
 
Join Date: May 2018
Rookie Year: 2012
Location: Wilsonville, OR
Posts: 3
cvbriggler is an unknown quantity at this point
Re: Compiling raw data from FIRST

I'm not able to get to the BigQuery data set. Getting an error message that says "Unable to find dataset".

Anybody know if this resource is off line or no longer available?
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 04:53 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi