FIRST is really looking into the Einstein problems

[Jon Stratis]

Originally Posted by JohnBoucher

If I am not mistaken this weekend is the Manchester test of the Einstein robots and the Einstein field. After some initial announcements, FIRST has been very quiet about this.

Attn FIRST: The CD community and the FIRST community are thankful that you are working hard to understand what happened on Einstein at champs. In order to restore the confidence of the FIRST community you need to communicate to us what is being done. We initially received a few communications, but there has been none leading up to this weekends test sessions. Please make this process more transparent. We need information. We do not expect that you will publish final results after this weekend, but we need to know what is being done. Please keep us in the loop.

Probably the most damaging thing FIRST could do right now would be to release any information that ends up not being 100% accurate. What more can they tell us other than they're investigating and having the teams come out? What would you and everyone else here think if they said "We've determined the issue is not with the wifi routers", and a month form now they change their minds?

There's a lot they can do to investigate the issue and try to reproduce it themselves... but until they can reproduce it with team robots and setup, they can't be 100% sure they've found the issue. They need to reproduce it with the teams, fix it with the teams, AND prove that the fix actually worked before they can be sure of anything. And with an intermittent issue like this, proving it can be quite a chore.

Half of my job is to triage issues in our production system... Sometimes, it's easy. Sometimes, it's an intermittent issue that's almost impossible to reproduce reliably. Lets say you run through a test for the issue, and you encounter it only 3 times out of 10 runs. You think you understand it and apply a fix. To prove that it's actually fixed, how many times do you need to run your test? How reliable is your test if it isn't run on a production-equivalent environment? Debugging issues is difficult enough when your users interact with your system through a clearly defined interface (like a website). Debugging them when your users are writing their own custom code that interfaces with your system has to be a nightmare.