I've always found square and elliptical gears, and similar mechanisms amazing. I'm pretty good with geometry but I don't know how people figure out the shape of the teeth on those types of things.
One of the great (way under-known and under-rated) things that Cornell University has is the Reuleaux Collection of Mechanisms and Machines. I lived in the mechanical engineering department; everyday I would walk by display cases of hundreds of mechanisms, most you'd never even think of if you spent your entire life trying... Several nobody has figured out what they are useful for at all. Anyway basically Reuleaux (French, I believe) spent much of his life designing and building (all made by hand) this collection of mechanisms... Gears, linkages, etc... And eventually the collection was obtained by Ezra Cornell sometime at the turn of the 20th century. Given a desired input and output (linear or rotary, dynamic speed or constant), you can probably find a mechanism there that does what you need, or you can combine them to get a new mechanism. They're in the process of putting the entire library online with 3d models and videos, so definately check it out:
http://kmoddl.library.cornell.edu/model.php
And if you ever visit Cornell's campus, walk through the second floor of Upson Hall to see them all for yourself. Also, there are some cool ones (but not quite as cool as the Reuleaux collection) at Boston's museum of science.
There's even a mechanism that could be used to machine an almost perfect square hole

__________________
Systems Engineer - Kiva Systems, Woburn MA
Alumni, Former Mechanical Team Leader - Cornell University Robocup - 1999, 2000, 2002, 2003 World Champions
Founder - Team 639 - Ithaca High School / Cornell University
Alumni - Team 190 - Mass Academy / WPI