Comparing performance of two sets of code

[Jon Stratis]

Be careful on how much you're logging... only log what you absolutely need to! I have two real-world stories about logging for you... one recent (in the past 2 weeks), and the other from back in college.

First, the college one. I was in a class called "Autonomous Robotics". We had 3 person teams, built a small robot, and controlled it with a HandyBoard from MIT. We spent 3 weeks debugging it trying to figure out why the robot would randomly go haywire... and the more logging/display output we did, the worse the behavior got! We eventually found out that there was a bug in the firmware for the board. If you sent an odd-length message to the display, it would corrupt your entire program. 3 weeks of testing to find out that our only issue was the debugging statements we were printing out!

Now, the recent one. We recently did a fairly intensive process to reclaim some space on our database. We had it all tested out and everything looked great - it would take about a full day, based on testing in an equivalent, non-production environment, but what can you do? We went to do it on the production environment, and had to stop in the middle because it was taking too long - we projected 3 days, based on what it had finished when we stopped it! It turns out that additional logging we had turned on in the production environment was what was slowing down the process. We were able to turn off the logging and get through it in the anticipated 24 hours.

So, be very careful about logging. It's entirely possible to log too much and have it seriously affect your performance!