My Solution, not as simple as @Jon_Stratis' or @AriMB's
Strategy: Use the law of cosines to get cosine and sine of the three angles at one of the large circle’s center and use an angle sum formula. Here we’ll do A.
BC²=AC² + AB² - 2 AC AB cos(CAB)
BD²=AD² + AB² - 2 AD AB cos(DAB)
CD²=AC² + AD² - 2 AC AD cos(CAD)
As the line segment connecting the centers of externally tangent circles is the sum of the radii, we can easily see that AB=3, AC=4, BC=5. Letting the radius of the small circle be r, we also have: AD=1+r, BD=2+r, CD=3+r.
Solving for the cosines, we have:
cos(CAB) =-(5²-4²-3²) / (2·4·3) = 0/24 = 0
cos(DAB) = -((r+2)² - (r+1)² - 3²) / (2(r+1)3) = (3-r)/(3r+3)
cos(CAD) =-((r+3) ² - 4² - (r+1)²) / (2·4(r+1)) =(2-r)/(2r+2)
When cosφ = x/y , sinφ = √(1-x²/y²) = √(y²-x²)/y, so
sin(DAB) = √((3r+3)² - (r-3)²) / (3r+3)= √(8r²+24r)/(3r+3)
sin(CAD) = √((2r+2)² - (r-2)²) / (2r+2)= √(3r²+12r)/(2r+2)
But cos(α+β)=cosαcosβ – sinαsinβ, so
cos(CAB) = cos(CAB) cos(DAB) - sin(DAB) sin(CAD)
As the left hand side is zero and the denominators of the two terms on the right match and are clearly positive, we’ll multiply by that denominator and just look at the numerators:
0= (r-3)(r-2) - √(8r²+24r) √(3r²+12r)
√(24r⁴+168r³+288 r²) = r² - 5r + 6
24r⁴ + 168r³ + 288r² = r⁴ -10r³ +37r² - 60r + 36
23r⁴ + 178r³ +251r² +60r - 36 = 0
Inspired by the rational root theorem and that the central circle is about a quarter the radius of the smallest outer circle (radius 1), we try factoring this by 23r-6, and succeed! This factors as (23r-6)(r+6)(r+1)². The negative roots are not interesting as a solution to this problem, so r=6/23.
Given that r is rational, let’s see if the coordinates of the inscribed circle are, as well : 1+6/23=29/23. It is well known that 29² = 20² + 21², and using the Pythagorean theorem, it is easy to prove that (21/23,20/23) is 29/23 from A, 52/23 from B, and 75/23 from C.
But then, when I did this solution, I hadn’t assigned coordinates for the circle centers, I was just working with the radii of 1, 2, and 3 (the simplest case I could think of where none of the three radii were equal). I didn’t realize that the centers made a right triangle until I found that cos(CAB)=0.