Re: FRC Event Data in XML Format
 

Was that run for 2009?

The lack of data for UT makes sense, but MD doesn't.
MD might have some leading garbage data that throws the code off.

Some of them are year dependent too, like EIN vs CMP.

Re: FRC Event Data in XML Format
 

If one of everything uses up 3% of your bandwidth, you may have to consider what will happen when more than a few people are systematically pulling everything. Is this a per-24-hour cycle? Is the JSON just totally fat?

If I were to switch The Blue Alliance over to pulling from FRCFeeds, I would pull down every current-week event's match results every five minutes. In a day, this makes for (8 events * 12 requests per hour * 24 hours per day = 2300 requests) Is that OK? [I intend to run my own clone of the FRCFeeds source if I do, but I am just posing a thought question]

Re: FRC Event Data in XML Format
 

Make that two of us that would do what TBA does. I do the same for Regional Twitter Accounts.

Re: FRC Event Data in XML Format
 

Yeah, it was a run for 2009, I am going to take a look at the html to see what sort of garbage is in there in a bit.

That was my daily upload bandwidth, I had to put the templates in so was sending a lot of POST data. I am going to rerun that test to see what sort of impact it had.

App Engine cites that it should handle 5 million page views per day for free realistically I should be able to see about half that. This was a run of ~200 and it used ~3% and 1% of my daily quotas. Meaning with both TBA and Twitter Feeds it should be able to handle it. (That being said, without doing a full test I make no promises)

As for running a local version. I will put the code up within the next few days. I have some minor changes that have to be made to run in a non app engine environment.

Re: FRC Event Data in XML Format
 

Now I see why you are not worried about bandwidth :D

Re: FRC Event Data in XML Format
 

Time for the hard numbers, I reran the test. With 749 calls to the page it used:
Basic math will tell me that 100/(.04/749) = 1.8 million requests for XML data per day. If that number isn't high enough I will either try to figure out how to force compression of the data or will fork out the couple bucks to up that cap.

Currently there is a 16% error rate, I hope to get that number much lower in the upcoming week. About 12 of the errors are due to the fact that some of the characters on the pages "can't be decoded by the ascii codec" (That is the error I see in the logs) No idea how to fix that but I have a hunch it will be really simple. As for the rest, I think it is just garbage HTML I have to get rid of before the parser gets a hold of it. Really FIRST, is it too hard to make code that is valid?

Re: FRC Event Data in XML Format
 

It would be great to maintain both app-engine and non-app-engine versions in the Google Code project. That way it's easiest for people who want to bring their own versions online.

I am surprised you used 3% bandwidth so quickly if it's supposed to handle 5 million pageviews. You only requests a hundred or so pages, right?

Re: FRC Event Data in XML Format
 

I've encountered this problem before. You will need to tell Python to treat the strings as Unicode instead of ASCII.

Check these links for more info:
http://www.amk.ca/python/howto/unicode
http://eric.themoritzfamily.com/2008...s-and-unicode/

Good luck!

Re: FRC Event Data in XML Format
 

See above for the number.

My calls to the FIRST website take incoming bandwidth (app engine makes a distinction) so for every call I get I am using that miniscule amount of bandwidth to get their http request and then using a decent amount to grab the html from the FIRST website.

Thanks for the tips on handling those strings.

Re: FRC Event Data in XML Format
 

4 months later...

Just a heads up the new version went live ~20 minutes ago. It does work with 2010 stuff. I tweaked the request stuff slightly.

Standard query string stuff ?event=[event code]&what=[qm,em,qs,awards,rankings,standings,teams]&format[json,xml]

I also added a filters option, you can use the following syntax:
&filters=([filter-name]-[team])

You can gang multiple filters together within one filters arg or gang multiple filters args together. For example:

http://frcfeed.appspot.com/?event=nj...5)&format=json will return any matches that 25 and 11 played together at New Jersey.

Current options are:
won
lost
played
isRed
isBlue

Filters within a filters arg are ANDed together and multiple filters args are ORed together. The filters for any of 11's matches or any of 25's matches would be &filters=(played-11)&filters(played-25).

Hope it is useful.

Re: FRC Event Data in XML Format
 

Andrew have you updated this for 2011?

I tried the event code TN (for a rookie regional) and it came up with an error.

Are you only supporting xml/json or do other formats work?

Also can you add a what that is only team numbers and not the names?

Re: FRC Event Data in XML Format
 

I'm working on a project that could really, really benefit from this feed. Any word on if this will be working this season, or should I start hashing out a solution from the Twitter feed? (Pun intended)

Re: FRC Event Data in XML Format
 

I didn't know anyone was using it. I've been working on some new stuff to put in it but am sort of delayed because of job concerns (and my partner on it is really busy)

My suggestion, if you really need to parse the pages you can download the source code for the parser (http://github.com/bryanculver/FRC-Fe...feed/Parser.py) and the helper stuff (http://github.com/bryanculver/FRC-Fe.../FRCHelper.py). You will also need BeautifulSoup (http://github.com/bryanculver/FRC-Fe...autifulSoup.py) .

Those links should always contain the latest working versions of those functions but none of those have changed since I initially created them. (FRCHelper.py occasionally gets a new filter or two but chances are you won't need those). If you need help using any of the stuff or have features you really would like added just let me know.

Re: FRC Event Data in XML Format
 

<sarcasm>Oh great. I get to learn Python.</sarcasm> :P

I have until the first week of regionals to finish the web app, but I'll probably go ahead and download the source to use on my server. Also, thank you for creating this. It's a huge time-saver for me. :)