[FRC BLOG] The Great Registration System Crash of 2016

[bdaroz]

The Great Registration System Crash of 2016

The Great Registration System Crash of 2016
Written by Frank Merrick
I am so sorry for the trouble we experienced with initial event registration today. We at FIRST HQ really do understand the investment of resources teams make to participate in our programs, and teams have the right to expect that we will be holding up our end of the deal by making things happen like we say we will. We failed to do that in this instance, and once again, I apologize.

While we did perform extensive load testing on the servers in preparation for this event, something still went wrong. Our Information Technology Department has been working feverishly since the crash to puzzle this out and come up with a plan to prevent recurrence. It’s not yet clear that it was the registration load per se that actually caused the problem. The system crashed immediately, before even a single team was registered, and this is strange behavior.

As you hopefully know from our tweets, Facebook posts, and emails, we are postponing initial registration until next week. Our goal is to be able to announce the specific day and time for registration before the weekend begins, so you can make appropriate plans.

Another goal we have is to be open and honest about what caused this issue and the steps we are taking to correct it. Our IT department has said they will do a guest blog explaining the situation in detail as soon as we have the facts.

Once again, I am sorry. Despite this issue, I do believe we have a great season ahead of us!

Frank

[SenorZ]

I remember where I was...

[dodar]

Originally Posted by SenorZ

I remember where I was...

Pepperidge Farm remembers

[Jardanium]

It was said that this event caused periods known as the Great Compression and STEAM Bowl to follow...

In all seriousness, I personally appreciate the continued transparency initiative from Frank and the rest of FIRST HQ. It's nice to be in the loop about these things!

[BenDSterling]

I can't wait to find out exactly what the problem was. I'm really curious what caused it to crash before any teams where even registered.

[JB987]

So when is round 2???

[BrendanB]

Originally Posted by JB987

So when is round 2???

We probably won't hear a time for that until they've remedied the problem. No sense in setting a deadline you can't meet. Thank you Frank for addressing the crash before the close of the day.

[Bryan Herbst]

I suspected it was more than just load. If teams are anything like me, they didn't just log on at precisely 12pm- they were refreshing the page a few minutes beforehand.

The system crashed right when it should have gone live, so I am thinking something with activating registration revealed a different problem.

[Trying to Help]

Originally Posted by Jardanium

In all seriousness, I personally appreciate the continued transparency initiative from Frank and the rest of FIRST HQ. It's nice to be in the loop about these things!

Yes, thank you Frank for speaking to this directly.

[marshall]

Originally Posted by Frank's Blog

Another goal we have is to be open and honest about what caused this issue and the steps we are taking to correct it. Our IT department has said they will do a guest blog explaining the situation in detail as soon as we have the facts.

I am really looking forward to this. I expect nginx and IIS logs!

[sanddrag]

I appreciate that FIRST is honest and open about it, and they made the right call to reschedule to next week. However, many of us waited an hour at our computers, in an ambiguous state not knowing if we were going to register today or not. I'm not sure they even realize how high the stakes are to get into certain events. Certain events literally fill within 120 seconds, or even less.

What FIRST really should have done was had a twitter announcement ready to fire out in the event that something like this happened, and a staff member assigned to do it. They should have seen that they had a problem that they could not resolve by 9:05 AM. They should have been ready to abort the process by 9:10 AM, and a Twitter announcement should have gone out no later than 9:15 AM at the latest, whether they had a statement prepared or not. It's not okay to make that many people wait as long as we did, when it's in the middle of the work day. The second they saw they had an issue that would take more than 1 minute to fix, they should have aborted the whole plan, and gone to the plan to release a statement regarding rescheduling of the registration date.

I agree, load was not the issue. I was getting consistent 5-second page refresh times from 8:45 AM PDT right up to 8:59:56 PDT when I did my last refresh. At 9:00:02 the page immediately loaded with a run-time error.

I appreciate them calling it off, but I just wish it had happened (via twitter, facebook, e-mail, etc) long before 10AM.

Let this be a lesson for FIRST not only in registration, but in the fact that they need to have plans and procedures in place to widely distribute information out to their teams in a more timely fashion than a 1-hour delay.

[DaveL]

If the issue had something to do with volume, FIRST could have a different day for district teams to register.
FIRST could even separate the districts further, by picking a different day for each district to register.

Dave

[runneals]

Originally Posted by DaveL

If the issue had something to do with volume, FIRST could have a different day for district teams to register.
FIRST could even separate the districts further, by picking a different day for each district to register.

Originally Posted by Whatever

If there is I want: "A team set up automatic registration routine to make sure they got their first choice and that routine went nuts bringing down the server."

Both of these are very valid ideas that are both worth attempting (although I think I read the second one differently the first time). Why don't they do something like what they do in December with FIRST Choice, where it allows a team to put in events and rank them which they would want 1-5(10). This would allow teams to come up with a list of events that they would want to attend, allow mentors to input them in their free time before the specified date, and then the slotting program be ran at a specified time to slot teams into events based on their choices that they made. For quite a few people, registration is not during their lunch hour so it could be hard for them to do it at work. The "slotting program" could also account for starting at 0 and working it's way up so older teams get first pick, but it could also include some spatial algorithm to allow teams to attend their closest event.
Allowing district teams to register on their own day would offer a way for IT to ensure that their systems would work for regional registration.

Just some random ramblings after reading this

[nuclearnerd]

Originally Posted by runneals

Why don't they do something like what they do in December with FIRST Choice, where it allows a team to put in events and rank them which they would want 1-5(10). This would allow teams to come up with a list of events that they would want to attend, allow mentors to input them in their free time before the specified date, and then the slotting program be ran at a specified time to slot teams into events based on their choices that they made. For quite a few people, registration is not during their lunch hour so it could be hard for them to do it at work. The "slotting program" could also account for starting at 0 and working it's way up so older teams get first pick, but it could also include some spatial algorithm to allow teams to attend their closest event.

+1
https://www.chiefdelphi.com/forums/s...&postcount=148

[Chris is me]

Originally Posted by runneals

Both of these are very valid ideas that are both worth attempting (although I think I read the second one differently the first time). Why don't they do something like what they do in December with FIRST Choice, where it allows a team to put in events and rank them which they would want 1-5(10). This would allow teams to come up with a list of events that they would want to attend, allow mentors to input them in their free time before the specified date, and then the slotting program be ran at a specified time to slot teams into events based on their choices that they made. For quite a few people, registration is not during their lunch hour so it could be hard for them to do it at work. The "slotting program" could also account for starting at 0 and working it's way up so older teams get first pick, but it could also include some spatial algorithm to allow teams to attend their closest event.
Allowing district teams to register on their own day would offer a way for IT to ensure that their systems would work for regional registration.

Just some random ramblings after reading this

I don't like this nearly as much as I like the FIRST Choice system.

If you don't get what you want on FIRST Choice, you don't get some free stuff you weren't totally expecting to get. If a regional is full, I want to know as soon as I sign up for it that I'm not on the confirmed list so I can look at my other options and decide.

How can an algorithm decide if I'd rather be on the waitlist for one event rather than confirmed for an event 7 hours away that I know won't be filled? There's too many human decisions in this process for me to want to automate my decision making in an algorithm.

FIRST should be able to handle a few thousand page requests at once. If they can't, any number of outside firms I'm sure would love to have a contract to do this. This isn't an insurmountable challenge.