The Blue Alliance - Data Loss

[Greg Marra]

UPDATE: We experienced some data loss on The Blue Alliance. I'm working to restore the situation, and it should be restored entirely by the end of April 4th, 2012.

Those curious about technical details can read more in this Google Doc, where I am recording what I am doing to fix it.

Most of 2012's data is recovered already. Not all events have been, so some pages will continue to generate errors. Whatever doesn't heal itself with cronjobs will get fixed tomorrow night.

Thanks,
Greg

[Greg Marra]

my.usfirst.org appears to be down, which is preventing The Blue Alliance from rescraping data from FIRST.

http://www.usfirst.org/whatsgoingon fails to load its iframe, for example.

[Greg Marra]

At this point, nearly all data has been recovered. 2012 Events and Matches are partially missing. 2011 Events and Matches are entirely missing. This data will be recovered when FIRST's pages come back online.

Some teams who have not competed in 2012 have lost their details like nickname. These will be restored when FIRST's pages come back online.

Will write more about backup measures we should take in the future in the document, but not tonight.

[Greg Marra]

FIRST's servers are rejecting our scraping attempts from our production server. I've emailed frcteams@usfirst.org to attempt to resolve the issue. Does anyone know anyone else I can get in touch with?

"Google access to this page has been blocked due to repeated failure to respect robots.txt"

[Mark McLeod]

What are you violating? The 10 sec page request delay or something else?
You'll want to assure them you'll respect current settings and future changes in robots.txt.

I've talked to their IT dept before, but normal changeover almost assures that it's different people by now.
New employees are publicized in the FIRST Newsletter. I've attached a list, but beware, the earlier people may be gone or re-positioned within the organization as time passed.

[Greg Marra]

Originally Posted by Mark McLeod

What are you violating? The 10 sec page request delay or something else?
You'll want to assure them you'll respect current settings and future changes in robots.txt.

I got in touch with the FIRST web team, and we're going to make some changes to how we request pages. Things should be back up and running later tonight :-)

[Tetraman]

Thank you for keeping such a website up and running. It's a great tool to use and FIRST wouldn't be the same without it.

[Barry Bonzack]

Thanks Greg for all your hard work. The Blue Alliance is a true asset to the community.

[Greg Marra]

I believe all data is restored now. We lost some metadata that had been manually edited, but the Events and Matches should be back. We'll monitor our logs for errors in the next few days, and fix anything else that crops up.

We're now investigating backup options :-)

[Akash Rastogi]

Thanks for your hard work Greg! I needed this back up ASAP to do some scouting.

[Greg Marra]

Originally Posted by Akash Rastogi

Thanks for your hard work Greg! I needed this back up ASAP to do some scouting.

Glad we managed to fix everything midweek instead of over a weekend.

What event is 2012oj? We don't seem to have it, and I can't figure out what it is. People are trying to get to it though, and it's throwing errors.

[Billfred]

Originally Posted by Greg Marra

What event is 2012oj? We don't seem to have it, and I can't figure out what it is. People are trying to get to it though, and it's throwing errors.

Considering this throws an error, I suspect there is no such event.

[Greg Marra]

Originally Posted by Billfred

Considering this throws an error, I suspect there is no such event.

I'll just put another thing in the list of "why we should make proper 404 pages". Thanks!

[Littleboy]

Many team names have been lost and replaced with just the number.

[~Cory~]

3 Apr 2012, 21:27 - Reporting user replies that methods were unknowingly called during probe of security hole. Thus, data was lost in prod.

This sounds very unethical, especially with TBAv4 being open source. Said user should have run tests on a local machine.

Can you expand a little more on this portion?