The chart on this webpage should sum it pretty nicely. For some reason I can't directly link to it, and google images doesn't turn up any other good ones,
so go to this website and search for "Graph illustrating the effect of proper fillet radii on stress concentration" (minus the quotes) and there it will be.
The factor on the left (Kf) is a multiplier to the max torsional stress (and when you exceed the max torsional stress of the material, it will fail!) you find if there was no change in diameter. So, you can see it isn't hard at all to multiply your stress significantly for relatively small changes in shaft diameter.