Go to Post I love how the people of Chief Delphi can turn a spam post into a somewhat non-timewasting thread. Either way, congrats CD and thanks for this ;) - logank013 [more]
Home
Go Back   Chief Delphi > FIRST > General Forum
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
 
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 23-12-2009, 23:35
Unsung FIRST Hero
Greg Marra Greg Marra is offline
[automate(a) for a in tasks_to_do]
FRC #5507 (Robotic Eagles)
Team Role: Mentor
 
Join Date: Oct 2004
Rookie Year: 2005
Location: San Francisco, CA
Posts: 2,030
Greg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond repute
Re: FIRST Website Team Pages?

Quote:
Originally Posted by Joe Ross View Post
Pat posted how he did it in your previous thread: http://www.chiefdelphi.com/forums/sh...ad.php?t=78368
Thanks Joe! I hadn't seen these posts.

Quote:
Originally Posted by Alan Anderson View Post
Why not just use frclinks itself and not worry about reinventing that particular wheel?
FRCLinks uses a Javascript redirect. I am pointing at FRCLinks for links to team pages right now on TBA, but I need to do a full scrape of FIRST's pages to update Team Names to be accurate now. Wget doesn't follow Javascript redirects - I may need to bake up something a bit fancier to either parse these out of FRCLinks, or parse them out of FIRST's team data when I am scraping event attendance.

I wish this were easier.

I was able to get this URL for listing Teams, but it's hardcoded to max out at 250 teams listed. I was hoping to get every single team on the page at once, and then scrape out all the team URLs in one go. This approach won't work on its own.

https://my.usfirst.org/myarea/index....asons_frc=2010

Last edited by Greg Marra : 23-12-2009 at 23:41.
Reply With Quote
  #2   Spotlight this post!  
Unread 24-12-2009, 13:25
Unsung FIRST Hero
Greg Marra Greg Marra is offline
[automate(a) for a in tasks_to_do]
FRC #5507 (Robotic Eagles)
Team Role: Mentor
 
Join Date: Oct 2004
Rookie Year: 2005
Location: San Francisco, CA
Posts: 2,030
Greg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond repute
Re: FIRST Website Team Pages?

Attached is a CSV with every team competing in 2010's team number and tpid, which is the number FIRST uses to refer to the team. Hopefully this will be useful to someone in the future.
Attached Files
File Type: txt tpids.txt (17.8 KB, 46 views)
Reply With Quote
  #3   Spotlight this post!  
Unread 24-12-2009, 13:50
Andrew Schreiber Andrew Schreiber is offline
Joining the 900 Meme Team
FRC #0079
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 4,059
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: FIRST Website Team Pages?

Quote:
Originally Posted by Greg Marra View Post
FRCLinks uses a Javascript redirect. I am pointing at FRCLinks for links to team pages right now on TBA, but I need to do a full scrape of FIRST's pages to update Team Names to be accurate now. Wget doesn't follow Javascript redirects - I may need to bake up something a bit fancier to either parse these out of FRCLinks, or parse them out of FIRST's team data when I am scraping event attendance.
"window.location.*?=.*?\"(.*)\"" as a regex on the content of the frclinks is a pretty simple way of grabbing Pat's redirect. That is how frcfeed is doing it. Just grab the content of group 1.

In python:
Code:
URL = re.search("window.location.*?=.*?\"(.*)\"",Content).group(1)
__________________




.
Reply With Quote
  #4   Spotlight this post!  
Unread 24-12-2009, 14:25
Unsung FIRST Hero
Greg Marra Greg Marra is offline
[automate(a) for a in tasks_to_do]
FRC #5507 (Robotic Eagles)
Team Role: Mentor
 
Join Date: Oct 2004
Rookie Year: 2005
Location: San Francisco, CA
Posts: 2,030
Greg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond repute
Re: FIRST Website Team Pages?

Quote:
Originally Posted by Andrew Schreiber View Post
"window.location.*?=.*?\"(.*)\"" as a regex on the content of the frclinks is a pretty simple way of grabbing Pat's redirect. That is how frcfeed is doing it. Just grab the content of group 1[/code]
I agree. I talked with Pat and decided it was easiest to just re-scrape the data from FIRST myself, since it required minimal modifications to existing TBA scraping scripts. Pat's service is great, and I'm going to keep using it on TBA where it makes sense.
Reply With Quote
  #5   Spotlight this post!  
Unread 25-12-2009, 23:30
Unsung FIRST Hero
Greg Marra Greg Marra is offline
[automate(a) for a in tasks_to_do]
FRC #5507 (Robotic Eagles)
Team Role: Mentor
 
Join Date: Oct 2004
Rookie Year: 2005
Location: San Francisco, CA
Posts: 2,030
Greg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond reputeGreg Marra has a reputation beyond repute
Re: FIRST Website Team Pages?

Done and done.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
**FIRST EMAIL**/Team Yearbook Pages Information Deadline Reminder KathieK FIRST E-Mail Blast Archive 0 20-02-2006 07:42
New Teams/FIRST Info Pages DarkJedi613 General Forum 3 22-10-2005 20:48
**IMPORTANT FIRST EMAIL**/Team Yearbook Pages Now Open! miketwalker FIRST E-Mail Blast Archive 1 27-02-2004 15:52
Team Pages Back UP!! archiver 2001 0 24-06-2002 01:30
team pages David Kelly General Forum 3 09-09-2001 20:05


All times are GMT -5. The time now is 22:15.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi