[TBA]: TBATV v4 Development Log

[Andrew Schreiber]

Originally Posted by Greg Marra

Memcached will play a large role in reducing CPU usage as well. By memcaching certain requests like "give me the HTML for Connecticut 2010's matches", we can dodge hitting the datastore at all, instead making a single memcached call. We'll expire these memcached objects when we make updates to the underlying data they hold (such as getting new Match results), but this should be particularly useful for page views (fast!) and API calls from other apps (repetitive!).

Instead of memcaching the html you could memcache the data and fill in a django template with it. To me that seems like the cleanest way of putting data into HTML files without mixing python code in with html code.

[Greg Marra]

Originally Posted by Andrew Schreiber

Instead of memcaching the html you could memcache the data and fill in a django template with it. To me that seems like the cleanest way of putting data into HTML files without mixing python code in with html code.

This is probably what we'll end up doing. I don't think rendering templates is very expensive compared to everything else, so that should be clean and easy.

[Greg Marra]

Here is a screenshot of AppStats showing the problem with the Match, MatchTeam, MatchScore object model. All of the database gets and puts to insert a Match add up to a lot of CPU time.

I wonder if storing everything in Match using a red1, red2, red3, blue1, blue2, blue3 system or a red_teams, blue_teams list reference property is a better idea. I'd really love for these relation objects to be faster, but when we render a full event worth of matches, it takes almost 20 seconds.

Does anyone know if it is possible to non-lazily query a bunch of objects ReferenceProperties all at once? Like say, "I want all the Matches, and their MatchTeam objects, and the Team objects on the other end of those"?

[Greg Marra]

I am going to change the Match model to be less flexible so we can speed up datastore performance.

I am thinking we will have a Match have Match.teams, which will be a ListProperty that stores the Teams in the match. Separately, we will store Match.alliances, which will contain a dictionary shaped like {"red": ["frc177", "frc195", "frc125"], "blue": ["frc433", "frc190", "frc222"]}. The teams property will basically be an index to let us quickly search by team, and the alliances property will store the actual structure of the alliances.

We'll add another property to Matches called "game", where we write down which FRC game was being played. This way, if they change the game structure in the future (or we get data on past games), we'll easily be able to adjust Controllers to handle it without having to muck with the model.

New concept:

class Match(db.Model):
    """
    Matches represent individual matches at Events.
    Matches have many Videos.
    key_name is like 2010ct_qm10 or 2010ct_sf1m2
    """
    event = db.ReferenceProperty(Event,
                                 required=True)
    time = db.DateTimeProperty()
    comp_level = db.StringProperty(required=True,choices=set(["Qualifications", "Quarterfinals", "Semifinals", "Finals"])) # This choices set should probably become a global Constant somewhere. How do you do that in Python properly? -greg 5/20/2010
    set_number = db.IntegerProperty(required=True)
    match_number = db.IntegerProperty(required=True)
    teams = db.ListProperty(Team) #Primarily for indexing and searching
    alliances = db.StringProperty #Store a Dictionary as a JSON string
    scores = db.StringProperty #Store a Dictionary as a JSON string

This will reduce the number of Datastore lookups to display a Match (assuming we're not interested in more than the team number) by just under an order of magnitude. I think that's a good thing!

Similarly, by changing from having a bunch of EventTeam objects, we could make teams a ListProperty of an Event. Then to get all of the Teams at an Event would require just finding the Event, instead of finding all of the EventTeams. This is switching away from a many-to-many relationship to many one-to-many relationships.

I haven't had much time to work on development, but I think these new ideas will remove some of the major roadblocks that existed.