I put in some more work this afternoon, and the Datafeed system is to the point that we can get all Events, Teams, and Matches from 2010 regionals posted on the FIRST website. We'll be importing this data from the TBAv3 SQL rather than re-scraping from FIRST, but this system is important moving forward for 2011.
There are still issues with how long datastore actions take. Some of these will be improved by using
Model.get_or_insert() instead of separately looking things up before trying to modify them. This will particularly improve Match insertion, as we need to create MatchTeam and MatchScore objects to accompany each match.
Memcached will play a large role in reducing CPU usage as well. By memcaching certain requests like "give me the HTML for Connecticut 2010's matches", we can dodge hitting the datastore at all, instead making a single memcached call. We'll expire these memcached objects when we make updates to the underlying data they hold (such as getting new Match results), but this should be particularly useful for page views (fast!) and API calls from other apps (repetitive!).
I'm still not 100% certain that the (Match, MatchTeam, MatchScore) object model is what we will ultimately go with. It creates
nine objects per Match, which is a bit expensive. It's also expensive to make sure that we don't have straggling objects if Teams were erroneously attached to matches. It really does provide nice flexibility going forward though.
Moving forward, the next tasks are to build out mock pages for Events, Teams, and Matches. This will make sure that our object model contains everything we think will go in the final pages. We're hoping to do a redesign of the site, so we're not going to put a lot of effort into the visual design at this point. Bare HTML will do.
As we get things more towards "works without elaborate setup" we'll push the code into either a Google Code or GitHub repository (any opinions here?). We hope the community will help suggest performance improvements or even commit patches!