Lessons to be learned from Einstein reports

[Todd]

Focusing on some of the software specific lessons that we might be able to take away from the technical report on the failures encountered on Einstein, I know from personal experience and aiding and reading over the shoulder of very many a student and mentor from another team that the following issues are something that we should probably all be careful to take note of in the future. Feel free to add any other topics that we should all remember.

Excessive ethernet communications:
Transmitting too large images, at too high a framerate, as well as sending too much data far faster than it can possibly actually be required at.

Excessive 'useless' CPU utilization:
Some tasks have to run fast, all tasks are not equal, and they do not all have to run fast. Particularly, many teams put time consuming I/O instructions into code segments that are expected to run very often, resulting in full CPU utilization just in the processor trying to keep up.

Ignorance of the potential for 'partial' failures:
Things happen, wires get cut, parts of your CAN bus fail, that gorgeous IC2 chip you have gets smashed by a basketball. Software should be architected so that the failure of one system does not mean the failure of the whole system. This is particularly dangerous in the case where a failure with an I/O device causes excessive timeout delays, which lead to the 'useless' CPU utilization problem mentioned above.

Segregation of 'I/O' jobs, vs 'Processing' jobs:
This one isn't entirely necessary, per say, but something that in my opinion is good practice. Too many a PID loop running in threads driven by instruction packet receipt, alongside I/O tasks that could throttle the control system loop into running in a totally unexpected timestep. The more that your system can seperate I/O tasks from Processing tasks, and from each other, the likely better off you are. (Certainly not in all cases, and only to a certain degree, but in general, in my opinion)



Some things you can do to work toward these goals:

Have your I/O jobs work independently from your processing jobs, storing aside and fetching requisite data.

Work to ensure that I/O jobs can detect when there is a problem with their communications, and work to free resources for other subsystems when they do, rather than consume them.

Throttle processing and I/O jobs to say, double the rate they have to be performed at to generate the performance you require, rather than 1000 times that rate. (Faster is not -always- better) (I know sometimes those control loops do actually have to run quite fast, but they're a special exception and should be handled separately)

I know there's always a lot of talk on here about using different software problems to alleviate performance needs, but any software language can run software that leaves itself open to performance concerns. All of the languages that are currently (popularly) utilized by FIRST teams can perform the task put before us, but not if we get in their way.