This is a big topic...
I found a lecture online that is not too bad:
Beam Stress Lecture with example problems
The bottom line is that you need to use the formula
S = Mc/I
S=stress
M is moment on the section of the beam in our case the height of the tooth times the stall torque of the motor (with a safety factor if you want -- I calculate the stress first and put in the safety factor afterwards)
c is 1/2 the thickness of the tooth at its weakest point near the base
I is the standard I (moment of inertia) for a rectangular cross section 1/12 b h^3 where b = width and h is the height (in our case the thickness of the tooth).
Solving you get:
Stress = 6 * Ftooth * ToothHeight / (ToothFaceWidth * ToothThicknessMin ^ 2)
Look on a table to see what material you are using and what stress it can take. I would not go above 30,000psi for unheated steel if I an help it (this is where I have my safety factor).
You can either assume your max stress and calculate your required facewidth or you can assume your facewidth and calculate your stress.
This is a very crude approximation but it is good enough for most of what I do in my daily job.
That is about as simple as I have time for right now.
Good luck.
Joe J.