Yikes, what happened to Oglala Lakota County (Shannon County) in SD? Were county seats used as an endpoint to calculate distance? This is the only explanation I can think of (Oglala Lakota County is the only county in the US (that I know of) to have its county seat outside of the county).
Is it possible to recalculate the distances as Manhattan distance? Then average the Euclidean and Manhattan? I have a feeling that will give a much closer representation of real-world travel distances. (Whether or not it is actually useful is another matter entirely.)
If I had the free time I would dive into this: https://stackoverflow.com/questions/17267807/python-google-maps-driving-time Select maybe 2 dozen candidate cities vs the team subset and let it go to town (no pun intended). Pick the city with the lowest residuals, boom. (It might hic-up where no road network is present (i.e. Hawaii)).
Also a quick cartographic tip to the OP (not trying to be mean), since you are dealing with difference and not a variance (i.e. positive only values) you should be using a monochromatic color scheme. The current color you are using is really only commonly applicable to +/- situations, such as standard deviation. The point I am trying to make is the eye is drawn to the ring of low saturation hues, and not to the point of your map: the geographic center of a distribution.
While the map is easy enough to read, the color scheme adds complexity where there doesn’t need to be any.