[FRC BLOG] The Great Registration System Crash of 2016
 

The Great Registration System Crash of 2016

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

I remember where I was...

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

Pepperidge Farm remembers

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

It was said that this event caused periods known as the Great Compression and STEAM Bowl to follow...

In all seriousness, I personally appreciate the continued transparency initiative from Frank and the rest of FIRST HQ. It's nice to be in the loop about these things! :)

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

I can't wait to find out exactly what the problem was. I'm really curious what caused it to crash before any teams where even registered.

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

So when is round 2???

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

We probably won't hear a time for that until they've remedied the problem. No sense in setting a deadline you can't meet. Thank you Frank for addressing the crash before the close of the day.

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

I suspected it was more than just load. If teams are anything like me, they didn't just log on at precisely 12pm- they were refreshing the page a few minutes beforehand.

The system crashed right when it should have gone live, so I am thinking something with activating registration revealed a different problem.

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

I am really looking forward to this. I expect nginx and IIS logs!

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

I appreciate that FIRST is honest and open about it, and they made the right call to reschedule to next week. However, many of us waited an hour at our computers, in an ambiguous state not knowing if we were going to register today or not. I'm not sure they even realize how high the stakes are to get into certain events. Certain events literally fill within 120 seconds, or even less.

What FIRST really should have done was had a twitter announcement ready to fire out in the event that something like this happened, and a staff member assigned to do it. They should have seen that they had a problem that they could not resolve by 9:05 AM. They should have been ready to abort the process by 9:10 AM, and a Twitter announcement should have gone out no later than 9:15 AM at the latest, whether they had a statement prepared or not. It's not okay to make that many people wait as long as we did, when it's in the middle of the work day. The second they saw they had an issue that would take more than 1 minute to fix, they should have aborted the whole plan, and gone to the plan to release a statement regarding rescheduling of the registration date.

I agree, load was not the issue. I was getting consistent 5-second page refresh times from 8:45 AM PDT right up to 8:59:56 PDT when I did my last refresh. At 9:00:02 the page immediately loaded with a run-time error.

I appreciate them calling it off, but I just wish it had happened (via twitter, facebook, e-mail, etc) long before 10AM.

Let this be a lesson for FIRST not only in registration, but in the fact that they need to have plans and procedures in place to widely distribute information out to their teams in a more timely fashion than a 1-hour delay.

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

If the issue had something to do with volume, FIRST could have a different day for district teams to register.
FIRST could even separate the districts further, by picking a different day for each district to register.

Dave

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

To split hairs, the abort message hit their Twitter account at 12:45 PM Manchester time. I imagine they were making efforts to get things back up before realizing how borked everything was.

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

Having been in situations like this before (in a professional capacity), I can tell you that a 45 minute response time isn't actually that bad. Even if you have everyone needed to analyse the problem standing by, it does take some amount of time to pull the logs, find something relevant in them, and figure out what's going on, or even to figure out if it's a 5 minute or 5 hour fix. Add on top of that having to communicate the details and expectations from the engineers to management, get a decision, and get the communication out.

And then you have to ask what FIRST would consider acceptable. Sure, they want everything to go smoothly, but if there is a hiccup is starting registration 5 or 10 minutes late acceptable? Where do they decide to draw the line? It's all good to sit back as an armchair quarterback and say that 5 minutes is too late, but that really doesn't take into account the realities of business. This isn't life or death, waiting a few minutes, or even an hour, isn't going to be the end of the world.

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

So is there a pool on the reason?

If there is I want: "A team set up automatic registration routine to make sure they got their first choice and that routine went nuts bringing down the server."

Re: [FRC BLOG] The Great Registration System Crash of 2016
 

My bet is for a failed load balancer OR a failed interaction between the login system and the registration system, though they could be one system, I don't think they are.