The Blue Alliance - Data Loss

[Greg Marra]

Originally Posted by Akash Rastogi

Thanks for your hard work Greg! I needed this back up ASAP to do some scouting.

Glad we managed to fix everything midweek instead of over a weekend.

What event is 2012oj? We don't seem to have it, and I can't figure out what it is. People are trying to get to it though, and it's throwing errors.

[Billfred]

Originally Posted by Greg Marra

What event is 2012oj? We don't seem to have it, and I can't figure out what it is. People are trying to get to it though, and it's throwing errors.

Considering this throws an error, I suspect there is no such event.

[Greg Marra]

Originally Posted by Billfred

Considering this throws an error, I suspect there is no such event.

I'll just put another thing in the list of "why we should make proper 404 pages". Thanks!

[Littleboy]

Many team names have been lost and replaced with just the number.

[~Cory~]

3 Apr 2012, 21:27 - Reporting user replies that methods were unknowingly called during probe of security hole. Thus, data was lost in prod.

This sounds very unethical, especially with TBAv4 being open source. Said user should have run tests on a local machine.

Can you expand a little more on this portion?

[Greg Marra]

Originally Posted by Littleboy

Many team names have been lost and replaced with just the number.

Yea, we don't have fully up to date team names for teams that didn't compete in 2012. If you find any missing for 2012 teams, let me know.

For pre-2012 teams, I've opened an issue to fix this (we've never done it right): https://github.com/gregmarra/the-blu...nce/issues/108

Originally Posted by ~Cory~

Said user should have run tests on a local machine.

Can you expand a little more on this portion?

There was no malicious intent. This was a combination of curiosity and us having a big bug in our configurations. Everyone learned something.

[dodar]

Team 801 is missing name and sponsors.

[Greg Marra]

Originally Posted by dodar

Team 801 is missing name and sponsors.

Hmm, trying to figure out why we're not re-grabbing these right. Thanks for the pointers. FIRST's page structure makes scraping this information tricky (you can't just look up a team with it's team number), so I think something may have changed in the mean time.

Will dig in later this week, thanks.

[kmehta]

Originally Posted by Greg Marra

FIRST's page structure makes scraping this information tricky (you can't just look up a team with it's team number), so I think something may have changed in the mean time.

You could use frclinks.com/t/####.

[The Lucas]

Oddly enough, it looks like you only have team info for defunct team numbers like 40, 47 & 65. Wonder if that is stale backup data because it is not overloading them in failed scrapes.


All the team info TBA uses is available in one (easy to parse) tab deliminated page
https://my.usfirst.org/frc/scoring/i...?page=teamlist
It would be easier to just scrape that page. Plus, that is only 1 page request instead of thousands.

Alternatively, you could ask 358 for its database (especially if you want to fill in defunct team data)

Great job getting it back up, Greg!

[Eugene Fang]

Originally Posted by The Lucas

Oddly enough, it looks like you only have team info for defunct team numbers like 40, 47 & 65. Wonder if that is stale backup data because it is not overloading them in failed scrapes.


All the team info TBA uses is available in one (easy to parse) tab deliminated page
https://my.usfirst.org/frc/scoring/i...?page=teamlist
It would be easier to just scrape that page. Plus, that is only 1 page request instead of thousands.

Alternatively, you could ask 358 for its database (especially if you want to fill in defunct team data)

Great job getting it back up, Greg!

Just found https://my.usfirst.org/frc/scoring/i...page=eventlist too!
This is amazing...

[qzrrbz]

Originally Posted by EugeneF

Just found https://my.usfirst.org/frc/scoring/i...page=eventlist too!
This is amazing...

Don't believe the headers on this one though! Seems the "num_teams" field isn't there.

[Joseph Bisch]

Originally Posted by The Lucas

All the team info TBA uses is available in one (easy to parse) tab deliminated page
https://my.usfirst.org/frc/scoring/i...?page=teamlist
It would be easier to just scrape that page. Plus, that is only 1 page request instead of thousands.

Originally Posted by EugeneF

Just found https://my.usfirst.org/frc/scoring/i...page=eventlist too!
This is amazing...

While those would reduce the page requests, those seem to only provide data for 2012. I tried tacking on "&year=2011" and "&season_FRC=2011" to both URLs, but neither provided 2011 data. Is there something I am missing?

I think we should go ahead with implementing it and use it when we need data for the current season.

[The Lucas]

Originally Posted by Joseph Bisch

While those would reduce the page requests, those seem to only provide data for 2012. I tried tacking on "&year=2011" and "&season_FRC=2011" to both URLs, but neither provided 2011 data. Is there something I am missing?

I think we should go ahead with implementing it and use it when we need data for the current season.

You are not missing anything. That page is for the FMS so it only needs this year's data. Again, I recommend talking to 358 for any historical team data.

Other FMS pages you might find useful
2012 Event List
Sample Event Team List ('12 CMP)
You can get to the event team list by using index.lasso?page=event_teamlist&ID_event=<ID on the Event List Page>
I have been using these pages for a for scouting/stat purposes since they parse easier in Excel than the /myarea/ ones linked from the FRC regional page (copypasta a team list from there each team will have their town will show up in its own row, and it may be over multiple pages).

[Greg Marra]

Originally Posted by The Lucas

All the team info TBA uses is available in one (easy to parse) tab deliminated page
https://my.usfirst.org/frc/scoring/i...?page=teamlist
It would be easier to just scrape that page. Plus, that is only 1 page request instead of thousands.

!!

This page is amazing!!! Issue opened to switch our scrapers to this - it's so much simpler!

I made a FIRST Wiki page to trade notes on how to scrape FIRST pages. Borrowing Pat Fairbank's TPID scraper made scraping Team pages possible before, but I don't think that's a widely known technique. It needs more love, but an OK start.