Go to Post We all change, and we need to make sure that we can grow and change together in a positive way. Keep close the people who are capable of doing this with you, and you will surround yourself with the tools for success. - Jaine Perotti [more]
Home
Go Back   Chief Delphi > FIRST > General Forum
CD-Events   CD-Media   CD-Spy   FRC-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 10-25-2009, 07:58 PM
Andrew Schreiber's Avatar
Andrew Schreiber Andrew Schreiber is offline
6 events this season?
FRC #0125 (EWCP)
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 2,928
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
FRC Event Data in XML Format

I would like to announce yet another way to browse the FIRST website. I have often wished that I could have a Windows Gadget or a Dashboard widget that I could configure to scroll the schedule, or the rankings, or the match results. The problem has always been getting the data. I assume that Gameday has the data, and I know that TBA also has a wonderful API but I am not sure how often their data is updated. As a result I decided to pull data directly from the FIRST site and give it back out as XML. Enter http://frcfeed.appspot.com/ The goal of this site (also built on the Google App Engine) is to enable us to develop all sorts of applications without first having to deal with the hassle that is FIRST's website (anyone who has ever looked at the website's html will agree that a lot of it is ugly)

This site will return XML. For example the Awards for the 2009 Greater Toronto regional can be accessed at, http://frcfeed.appspot.com/2009/Grea...Toronto/awards upon viewing source you will see:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<Award>
	<Home Town>&nbsp;Port Perry, ON, Canada</Home Town>
	<Name>&nbsp;Bill Holtby</Name>
	<Award>Outstanding Volunteer Award</Award>
	<Team>&nbsp;1006</Team>
</Award>
<Award>
	<Home Town>&nbsp;Mississauga, ON, Canada</Home Town>
	<Name>&nbsp;General Motors of Canada & Rick Hansen Secondary School</Name>
	<Award>Imagery Award</Award>
	<Team>&nbsp;1241</Team>
</Award>
<Award>
	<Home Town>&nbsp;Whitby, ON, Canada</Home Town>
	<Name>&nbsp;MTC/Ontario Power Generation</Name>
	<Award>Rockwell Automation Innovation in Control Award</Award>
	<Team>&nbsp;1075</Team>
</Award>
<Award>
	<Home Town>&nbsp;Webster, NY, USA</Home Town>
	<Name>&nbsp;Xerox Corporation & Webster High Schools</Name>
	<Award>Xerox Creativity Award</Award>
	<Team>&nbsp;1126</Team>
</Award>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;TDSB & Martingrove C. I.</Name>
	<Award>Delphi "Driving Tomorrow's Technology" Award</Award>
	<Team>&nbsp;854</Team>
</Award>
<Award>
	<Home Town>&nbsp;Oakville, ON, Canada</Home Town>
	<Name>&nbsp;St. Mildred's Lightbourn Parent Association / Ford Motor Co. of Canada, Limited & St. Mildred's-Lightbourn School</Name>
	<Award>Kleiner Perkins Caufield & Byers Entrepreneurship Award</Award>
	<Team>&nbsp;771</Team>
</Award>
<Award>
	<Home Town>&nbsp;Sarnia, ON, Canada</Home Town>
	<Name>&nbsp;Mellon/Ontario Power Generation/Research In Motion & Conseil scolaire des écoles catholiques du Sud-Ouest & Saint-Francois-Xavier High School</Name>
	<Award>Judges Award</Award>
	<Team>&nbsp;1846</Team>
</Award>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;General Motors - St. Catharines Powertrain & Governor Simcoe Secondary School</Name>
	<Award>Website Award</Award>
	<Team>&nbsp;1114</Team>
</Award>
<Award>
	<Home Town>&nbsp;Calgary, AB, Canada</Home Town>
	<Name>&nbsp;Dean Weisensel</Name>
	<Award>Woodie Flowers Award</Award>
	<Team>&nbsp;1482</Team>
</Award>
<Award>
	<Home Town>&nbsp;Mississauga, ON, Canada</Home Town>
	<Name>&nbsp;BAE Systems/Promation Engineering Ltd. & Gordon Graydon Memorial SS</Name>
	<Award>Autodesk Visualization Award</Award>
	<Team>&nbsp;1325</Team>
</Award>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;Scotiabank/Bell Canada/Toronto District School Board & Woburn Collegiate Institute</Name>
	<Award>General Motors Industrial Design Award</Award>
	<Team>&nbsp;188</Team>
</Award>
<Award>
	<Home Town>&nbsp;Stoney Creek, ON, Canada</Home Town>
	<Name>&nbsp;CNC Woodcraft & HWDSB & Orchard Park Secondary School</Name>
	<Award>Motorola Quality Award</Award>
	<Team>&nbsp;2056</Team>
</Award>
<Award>
	<Home Town>&nbsp;Kincardine, ON, Canada</Home Town>
	<Name>&nbsp;Bruce Power/Power Workers' Union & Kincardine District Secondary School</Name>
	<Award>Johnson & Johnson Gracious Professionalism</Award>
	<Team>&nbsp;781</Team>
</Award>
<Award>
	<Home Town>&nbsp;Whitby, ON, Canada</Home Town>
	<Name>&nbsp;MTC/Ontario Power Generation</Name>
	<Award>Chrysler Team Spirit Award</Award>
	<Team>&nbsp;1075</Team>
</Award>
<Award>
	<Home Town>&nbsp;Sherbrooke, QC, Canada</Home Town>
	<Name>&nbsp;Bombardier Aéronautique/Uiversité de sherbrooke & &permil;cole du Triolet High School Centre 24 Juin</Name>
	<Award>Judge's Award</Award>
	<Team>&nbsp;3117</Team>
</Award>
<Award>
	<Home Town>&nbsp;</Home Town>
	<Name>&nbsp;</Name>
	<Award>Judge's Award</Award>
	<Team>&nbsp;</Team>
</Award>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;ASECO Integrated Systems / Sherrard Kuzz LLP / St. Catharines Corrugated Containers / GM Canada & Denis Morris High School</Name>
	<Award>Rookie Inspiration Award</Award>
	<Team>&nbsp;2852</Team>
</Award>
<Award>
	<Home Town>&nbsp;Kingston, ON, Canada</Home Town>
	<Name>&nbsp;Queen's University/Science '82 and '78/Queen's Engineering Society/Transformix Engineering & Kingston Collegiate Vocational Institute High School</Name>
	<Award>Rookie All Star Award</Award>
	<Team>&nbsp;2809</Team>
</Award>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;ASECO Integrated Systems / Sherrard Kuzz LLP / St. Catharines Corrugated Containers / GM Canada & Denis Morris High School</Name>
	<Award>Highest Rookie Seed Award</Award>
	<Team>&nbsp;2852</Team>
</Award>
<Award>
	<Home Town>&nbsp;Peterborough, ON, Canada</Home Town>
	<Name>&nbsp;G.E. Volunteers & Thomas A. Stewart High School</Name>
	<Award>Underwriters Laboratories Industrial Safety Award</Award>
	<Team>&nbsp;1141</Team>
</Award>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;Scotiabank/Bell Canada/Toronto District School Board & Woburn Collegiate Institute</Name>
	<Award>Regional Finalist #1</Award>
	<Team>&nbsp;188</Team>
</Award>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;Corporate Pharmacy/Bangor Metals/Diteba Research Laboratories Inc./Extrude-A-Trim Aluminum Warehouse & Crescent School</Name>
	<Award>Regional Finalist #2</Award>
	<Team>&nbsp;610</Team>
</Award>
<Award>
	<Home Town>&nbsp;North Bay, ON, Canada</Home Town>
	<Name>&nbsp;Near North Student Robotics Initiative</Name>
	<Award>Regional Finalist #3</Award>
	<Team>&nbsp;1305</Team>
</Award>
<Award>
	<Home Town>&nbsp;</Home Town>
	<Name>&nbsp;</Name>
	<Award>Regional Finalist #4</Award>
	<Team>&nbsp;</Team>
</Award>
<Award>
	<Home Town>&nbsp;Stoney Creek, ON, Canada</Home Town>
	<Name>&nbsp;CNC Woodcraft & HWDSB & Orchard Park Secondary School</Name>
	<Award>Regional Winner #1</Award>
	<Team>&nbsp;2056</Team>
</Award>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;General Motors - St. Catharines Powertrain & Governor Simcoe Secondary School</Name>
	<Award>Regional Winner #2</Award>
	<Team>&nbsp;1114</Team>
</Award>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;Etobicoke Collegiate Institute & TDSB</Name>
	<Award>Regional Winner #3</Award>
	<Team>&nbsp;2185</Team>
</Award>
<Award>
	<Home Town>&nbsp;</Home Town>
	<Name>&nbsp;</Name>
	<Award>Regional Winner #4</Award>
	<Team>&nbsp;</Team>
</Award>
<Award>
	<Home Town>&nbsp;Sault Ste Marie, ON, Canada</Home Town>
	<Name>&nbsp;Tenaris Algoma Tubes / EPOH & Algoma District School Board</Name>
	<Award>Regional Engineering Inspiration Award</Award>
	<Team>&nbsp;1535</Team>
</Award>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;General Motors - St. Catharines Powertrain & Governor Simcoe Secondary School</Name>
	<Award>Regional Chairman's Award</Award>
	<Team>&nbsp;1114</Team>
</Award>
Accessing this for different things is very simple, use the base url http://frcfeed.appspot.com/[year optional]/[Event name OR code]/[qualification OR elimination if applicable]/[results OR standings OR schedule OR awards]

Results and Schedule require either qualification or elimination be specified but the other two don't need it, adding it won't hurt anything though.

A word on the event name or code, FIRST specifies a bunch of codes for events, (ex: dt for Detroit) I hate remembering those, I assume that everyone hates remembering those. I decided that I would rather specify an event by its official name (removing the words regional, mi district, division, and field to save typing) However, I realize that some people would prefer to use a code, this supports BOTH, if you specify something it doesn't recognize as an event name it will assume you know what you are doing and treat it as an event code. It will return nothing if you get the wrong name or code.

Currently it defaults to 2010 for the year if you don't specify a year, there is no data for these events so you should probably specify 2009 as the year.

Enjoy!

ps: This is not to step on the feet of any of the other great services out there, TBA is amazing and I have heard great things about Gameday. If they offer a similar services consider this an alternative.
__________________

Planning to top a 4 banner year...
Reply With Quote
  #2   Spotlight this post!  
Unread 10-25-2009, 08:14 PM
rtfgnow rtfgnow is offline
Registered User
AKA: Alex Tannahill
no team
 
Join Date: Feb 2008
Rookie Year: 2008
Location: Here and There
Posts: 2,685
rtfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond repute
Re: FRC Event Data in XML Format

Good job, Andrew. I am a novice at parsing xml but would it be helpful to put each award on a new line?

It seems that Pat and Andrew have too great pieces of code out. There would be a sizeable explosion of awesomeness if theses two combined.
Reply With Quote
  #3   Spotlight this post!  
Unread 10-25-2009, 08:18 PM
Andrew Schreiber's Avatar
Andrew Schreiber Andrew Schreiber is offline
6 events this season?
FRC #0125 (EWCP)
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 2,928
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: FRC Event Data in XML Format

Quote:
Originally Posted by rtfgnow View Post
Good job, Andrew. I am a novice at parsing xml but would it be helpful to put each award on a new line?

It seems that Pat and Andrew have too great pieces of code out. There would be a sizeable explosion of awesomeness if theses two combined.
I must confess, I did indeed take a peek at the insides of FRCLinks when making this.

What do you mean put each award on a new line?

I am also looking into (as soon as I get a bit more spare time) adding the option to have the data come back as csv, xml, or json. The nice thing about json would be that you would only have to make a request to the page and then eval the results in Javascript.
__________________

Planning to top a 4 banner year...
Reply With Quote
  #4   Spotlight this post!  
Unread 10-25-2009, 08:25 PM
rtfgnow rtfgnow is offline
Registered User
AKA: Alex Tannahill
no team
 
Join Date: Feb 2008
Rookie Year: 2008
Location: Here and There
Posts: 2,685
rtfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond reputertfgnow has a reputation beyond repute
Re: FRC Event Data in XML Format

I mean put a break between awards.
Code:
<?xml version="1.0" encoding="UTF-8"?>
<Award>
	<Home Town>&nbsp;Port Perry, ON, Canada</Home Town>
	<Name>&nbsp;Bill Holtby</Name>
	<Award>Outstanding Volunteer Award</Award>
	<Team>&nbsp;1006</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Mississauga, ON, Canada</Home Town>
	<Name>&nbsp;General Motors of Canada & Rick Hansen Secondary School</Name>
	<Award>Imagery Award</Award>
	<Team>&nbsp;1241</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Whitby, ON, Canada</Home Town>
	<Name>&nbsp;MTC/Ontario Power Generation</Name>
	<Award>Rockwell Automation Innovation in Control Award</Award>
	<Team>&nbsp;1075</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Webster, NY, USA</Home Town>
	<Name>&nbsp;Xerox Corporation & Webster High Schools</Name>
	<Award>Xerox Creativity Award</Award>
	<Team>&nbsp;1126</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;TDSB & Martingrove C. I.</Name>
	<Award>Delphi "Driving Tomorrow's Technology" Award</Award>
	<Team>&nbsp;854</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Oakville, ON, Canada</Home Town>
	<Name>&nbsp;St. Mildred's Lightbourn Parent Association / Ford Motor Co. of Canada, Limited & St. Mildred's-Lightbourn School</Name>
	<Award>Kleiner Perkins Caufield & Byers Entrepreneurship Award</Award>
	<Team>&nbsp;771</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Sarnia, ON, Canada</Home Town>
	<Name>&nbsp;Mellon/Ontario Power Generation/Research In Motion & Conseil scolaire des écoles catholiques du Sud-Ouest & Saint-Francois-Xavier High School</Name>
	<Award>Judges Award</Award>
	<Team>&nbsp;1846</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;General Motors - St. Catharines Powertrain & Governor Simcoe Secondary School</Name>
	<Award>Website Award</Award>
	<Team>&nbsp;1114</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Calgary, AB, Canada</Home Town>
	<Name>&nbsp;Dean Weisensel</Name>
	<Award>Woodie Flowers Award</Award>
	<Team>&nbsp;1482</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Mississauga, ON, Canada</Home Town>
	<Name>&nbsp;BAE Systems/Promation Engineering Ltd. & Gordon Graydon Memorial SS</Name>
	<Award>Autodesk Visualization Award</Award>
	<Team>&nbsp;1325</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;Scotiabank/Bell Canada/Toronto District School Board & Woburn Collegiate Institute</Name>
	<Award>General Motors Industrial Design Award</Award>
	<Team>&nbsp;188</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Stoney Creek, ON, Canada</Home Town>
	<Name>&nbsp;CNC Woodcraft & HWDSB & Orchard Park Secondary School</Name>
	<Award>Motorola Quality Award</Award>
	<Team>&nbsp;2056</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Kincardine, ON, Canada</Home Town>
	<Name>&nbsp;Bruce Power/Power Workers' Union & Kincardine District Secondary School</Name>
	<Award>Johnson & Johnson Gracious Professionalism</Award>
	<Team>&nbsp;781</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Whitby, ON, Canada</Home Town>
	<Name>&nbsp;MTC/Ontario Power Generation</Name>
	<Award>Chrysler Team Spirit Award</Award>
	<Team>&nbsp;1075</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Sherbrooke, QC, Canada</Home Town>
	<Name>&nbsp;Bombardier Aéronautique/Uiversité de sherbrooke & &permil;cole du Triolet High School Centre 24 Juin</Name>
	<Award>Judge's Award</Award>
	<Team>&nbsp;3117</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;</Home Town>
	<Name>&nbsp;</Name>
	<Award>Judge's Award</Award>
	<Team>&nbsp;</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;ASECO Integrated Systems / Sherrard Kuzz LLP / St. Catharines Corrugated Containers / GM Canada & Denis Morris High School</Name>
	<Award>Rookie Inspiration Award</Award>
	<Team>&nbsp;2852</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Kingston, ON, Canada</Home Town>
	<Name>&nbsp;Queen's University/Science '82 and '78/Queen's Engineering Society/Transformix Engineering & Kingston Collegiate Vocational Institute High School</Name>
	<Award>Rookie All Star Award</Award>
	<Team>&nbsp;2809</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;ASECO Integrated Systems / Sherrard Kuzz LLP / St. Catharines Corrugated Containers / GM Canada & Denis Morris High School</Name>
	<Award>Highest Rookie Seed Award</Award>
	<Team>&nbsp;2852</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Peterborough, ON, Canada</Home Town>
	<Name>&nbsp;G.E. Volunteers & Thomas A. Stewart High School</Name>
	<Award>Underwriters Laboratories Industrial Safety Award</Award>
	<Team>&nbsp;1141</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;Scotiabank/Bell Canada/Toronto District School Board & Woburn Collegiate Institute</Name>
	<Award>Regional Finalist #1</Award>
	<Team>&nbsp;188</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;Corporate Pharmacy/Bangor Metals/Diteba Research Laboratories Inc./Extrude-A-Trim Aluminum Warehouse & Crescent School</Name>
	<Award>Regional Finalist #2</Award>
	<Team>&nbsp;610</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;North Bay, ON, Canada</Home Town>
	<Name>&nbsp;Near North Student Robotics Initiative</Name>
	<Award>Regional Finalist #3</Award>
	<Team>&nbsp;1305</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;</Home Town>
	<Name>&nbsp;</Name>
	<Award>Regional Finalist #4</Award>
	<Team>&nbsp;</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Stoney Creek, ON, Canada</Home Town>
	<Name>&nbsp;CNC Woodcraft & HWDSB & Orchard Park Secondary School</Name>
	<Award>Regional Winner #1</Award>
	<Team>&nbsp;2056</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;General Motors - St. Catharines Powertrain & Governor Simcoe Secondary School</Name>
	<Award>Regional Winner #2</Award>
	<Team>&nbsp;1114</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Toronto, ON, Canada</Home Town>
	<Name>&nbsp;Etobicoke Collegiate Institute & TDSB</Name>
	<Award>Regional Winner #3</Award>
	<Team>&nbsp;2185</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;</Home Town>
	<Name>&nbsp;</Name>
	<Award>Regional Winner #4</Award>
	<Team>&nbsp;</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;Sault Ste Marie, ON, Canada</Home Town>
	<Name>&nbsp;Tenaris Algoma Tubes / EPOH & Algoma District School Board</Name>
	<Award>Regional Engineering Inspiration Award</Award>
	<Team>&nbsp;1535</Team>
</Award>
<br>
<Award>
	<Home Town>&nbsp;St. Catharines, ON, Canada</Home Town>
	<Name>&nbsp;General Motors - St. Catharines Powertrain & Governor Simcoe Secondary School</Name>
	<Award>Regional Chairman's Award</Award>
	<Team>&nbsp;1114</Team>
</Award>
Reply With Quote
  #5   Spotlight this post!  
Unread 10-25-2009, 08:51 PM
Andrew Schreiber's Avatar
Andrew Schreiber Andrew Schreiber is offline
6 events this season?
FRC #0125 (EWCP)
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 2,928
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: FRC Event Data in XML Format

It would probably aid in human reading it but for an automated parser it shouldn't make a difference.
__________________

Planning to top a 4 banner year...
Reply With Quote
  #6   Spotlight this post!  
Unread 10-26-2009, 09:19 AM
rsisk's Avatar
rsisk rsisk is offline
The GURU Channel
AKA: Richard Sisk
FRC #2493 (Robokong)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2007
Location: Riverside, CA
Posts: 2,390
rsisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond repute
Send a message via MSN to rsisk
Re: FRC Event Data in XML Format

I agree with Andrew on leaving out the <br>. This is all data, no formatting allowed.

Thanks for putting this together Andrew, can't wait to use it as a data feed for the Regional Twitter Accounts this year. Now I can spend my time adding new features rather than battling First's crazy formatting for their match scores.
Reply With Quote
  #7   Spotlight this post!  
Unread 10-27-2009, 12:40 AM
Andrew Schreiber's Avatar
Andrew Schreiber Andrew Schreiber is offline
6 events this season?
FRC #0125 (EWCP)
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 2,928
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: FRC Event Data in XML Format

There have been a couple changes to this tonight (since I am slowly becoming nocturnal in an attempt to thwart kids on my lawn on Devil's Night)

First, you will probably not notice but I am caching results now, from your point of view it means faster page loads. For me it means less worrying about how long it takes to parse the page because I am no longer trying to do it multiple times at once (which is just plain stupid) The downside, results are no longer IMMEDIATE, they are potentially delayed by 5 minutes. I can't see this being a problem for most applications, in all honesty that is only 1 match delayed and I wouldn't doubt that FIRST caches there website anyway so updates probably only come out every N minutes. Besides, id rather have a slightly delayed service than one that doesn't work. Just as a point of reference, previously each time the page loaded it was taking just over a second of cpu time, now it is taking a tenth of a second or less. Much more usable.

Second, you will probably notice that the results have been sorted now (except Awards) Up until now they have been seemingly random (yeah, I know they aren't but they look like they were arranged by a monkey and it bothered me) Now, Schedule and Results are sorted by Match Number and Rankings are sorted by Rank. Hope these work for you.

Upcoming changes (read as, after I carve a pumpkin using a dremel)
  1. More Robust Request Handler
  2. JSON Format
  3. CSV Format

Any other requests?
__________________

Planning to top a 4 banner year...

Last edited by Andrew Schreiber : 10-27-2009 at 08:36 AM.
Reply With Quote
  #8   Spotlight this post!  
Unread 10-27-2009, 09:15 AM
rsisk's Avatar
rsisk rsisk is offline
The GURU Channel
AKA: Richard Sisk
FRC #2493 (Robokong)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2007
Location: Riverside, CA
Posts: 2,390
rsisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond repute
Send a message via MSN to rsisk
Re: FRC Event Data in XML Format

Any chance of getting an option to bypass the cache? For some applications a 5 minute delay is a lifetime
Reply With Quote
  #9   Spotlight this post!  
Unread 10-27-2009, 09:39 AM
Andrew Schreiber's Avatar
Andrew Schreiber Andrew Schreiber is offline
6 events this season?
FRC #0125 (EWCP)
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 2,928
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: FRC Event Data in XML Format

Really? I can lower the cache time to maybe a match length but would prefer to still use the cache due to the performance increase.

I suppose in the worst case at 5 minutes you are 2 matches behind, if we go with a match length the absolute worst you can do is one match behind. Would a 2 minute cache be more acceptable? I may also try making the caching system smarter, obviously for regionals that are already completed the cached version is fine for a lot longer than for regionals that are running at the moment. Either that or take a peek at the cached version (if it exists) and if a match should have occurred (based on the original schedule) discard the cached version... I don't know. I will try to lower that number as much as I can. Who knows, this whole thing could crash and burn 1st week since that is the first time it will really be tested under real load.
__________________

Planning to top a 4 banner year...
Reply With Quote
  #10   Spotlight this post!  
Unread 10-27-2009, 09:45 AM
rsisk's Avatar
rsisk rsisk is offline
The GURU Channel
AKA: Richard Sisk
FRC #2493 (Robokong)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2007
Location: Riverside, CA
Posts: 2,390
rsisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond repute
Send a message via MSN to rsisk
Re: FRC Event Data in XML Format

Last year I was able to get twitter messages out on the Regional Twitter Accounts within a couple of minutes of posting on the First site. Adding a 5 minute delay to that would make the decision to use a consistent XML interface versus the headache of parsing the First site directly a bit tougher.

How about distributing the parsing load as a way to improve performance?
Reply With Quote
  #11   Spotlight this post!  
Unread 10-27-2009, 10:08 AM
Andrew Schreiber's Avatar
Andrew Schreiber Andrew Schreiber is offline
6 events this season?
FRC #0125 (EWCP)
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 2,928
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: FRC Event Data in XML Format

Looking at actual match data though, Michigan events were running on a 6-8 minute cycle time last year, some quick glances at other events shows this is pretty much normal. For now I will set the cache to expire every minute, this should mean we get match results quickly and without hammering too hard on anyone's machines. If I am consistently near my limits I will adjust the cache accordingly but it will NOT go to more than 50% of the average cycle time. (for last year this would be roughly 4 minutes) Would this be an acceptable solution?

Also, this is a heads up, the XML will change slightly, spaces in the tags shouldn't be there, I was really tired when I wrote that. I will also be putting an enclosing <Event> tag around everything. (if anyone has suggestions on what the XML should look like please tell me now)
__________________

Planning to top a 4 banner year...
Reply With Quote
  #12   Spotlight this post!  
Unread 10-27-2009, 10:12 AM
rsisk's Avatar
rsisk rsisk is offline
The GURU Channel
AKA: Richard Sisk
FRC #2493 (Robokong)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2007
Location: Riverside, CA
Posts: 2,390
rsisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond repute
Send a message via MSN to rsisk
Re: FRC Event Data in XML Format

Seems like a reasonable solution.

So, if I understand correctly, you will parse a particular page once per minute?

I assume you can also specify a time range when the page should be parsed? That way you only parse the pages where change is expected.
Reply With Quote
  #13   Spotlight this post!  
Unread 10-27-2009, 10:22 AM
Andrew Schreiber's Avatar
Andrew Schreiber Andrew Schreiber is offline
6 events this season?
FRC #0125 (EWCP)
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 2,928
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: FRC Event Data in XML Format

Currently, no I can't specify a time range, I could probably set up a way of flagging events as completed and set the cache on those regionals to infinity (ie, as long as google keeps it in cache)

Currently the process the page takes is:

Figure out what they want.
Check to see if the data is in the cache.
If it is serve the cached data
If it isn't go grab the html, parse the data (this request gets to be the lucky one that takes a longer time)
Put the new data in the cache and tell the cache to dump it in N seconds
Serve the new data.

Data is only cached when someone requests it, I am assuming that if someone REALLY wants to display data from 2007 they can wait a little bit for it. The theory I am operating on is that if one person wants the data there are probably other people who will request it shortly (ie the event is running) Obviously someone could just be curious about old match results but I am not too worried about that.
__________________

Planning to top a 4 banner year...
Reply With Quote
  #14   Spotlight this post!  
Unread 10-27-2009, 10:30 AM
rsisk's Avatar
rsisk rsisk is offline
The GURU Channel
AKA: Richard Sisk
FRC #2493 (Robokong)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2007
Location: Riverside, CA
Posts: 2,390
rsisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond reputersisk has a reputation beyond repute
Send a message via MSN to rsisk
Re: FRC Event Data in XML Format

OK, I see what you are doing, I was thinking in my poll driven parsing model. How about different cache rates for different request types? Match results should be the most real time as possible, standings less real time, and awards the longest cache time.

It's almost like you need an application to manage the cache times to get the best trade off between performance and responsiveness. For example, crank down the cache time when matches are running for an event, then crank them up to infinity after the event.
Reply With Quote
  #15   Spotlight this post!  
Unread 10-27-2009, 10:43 AM
Dave Flowerday Dave Flowerday is offline
Software Engineer
VRC #0111 (Wildstang)
Team Role: Engineer
 
Join Date: Feb 2002
Rookie Year: 1995
Location: North Barrington, IL
Posts: 1,351
Dave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond reputeDave Flowerday has a reputation beyond repute
Re: FRC Event Data in XML Format

Andrew, just a suggestion: it looks like the Last-Modified HTTP header is set correctly for the usfirst.org pages you're looking at. Thus, if your script can handle it, you should be able to do a HTTP HEAD request, look at the Last-Modified field, and check that against the last time you parsed the page to determine if the data has changed and needs to be re-parsed.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
2009 FRC Kit of Parts (Pro/ENGINEER format) - Supported by PTC PTC_FRC CAD 18 11-18-2009 12:02 AM
FIRST Event Data in XML format Andrew Schreiber Technical Discussion 11 05-03-2009 03:18 AM
XML Interchange format Nibbles Scouting 25 10-03-2008 01:09 AM
How to write image data to binary PGM file format(P5)? tommy_chai Programming 0 10-08-2007 07:22 AM
Scouting Data Interchange Format proegssilb Scouting 7 06-06-2007 06:29 PM


All times are GMT -5. The time now is 07:32 AM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi