I have not done the formal math or testing on this, but I took a couple of thermodynamics courses and have thought about it a bunch over the years. I am pretty certain that if a Stirling engine could be feasibly made to work in an automobile, this engine model would see much more use in land and sea based applications. If someone with more specific experience can tune or correct my thinking, please do so!
First of all, the standard claim that the Stirling is the most efficient engine possible is based on the ideal thermodynamic model - a heat source at a constant temperature, and a heat sink at a constant (lower) temperature, driving a closed-cycle heat engine*. With such a setup, it is possible to prove that the maximum possible efficiency is equal to the ratio of the two temperatures, measured in kelvin (or an equivalent system where absolute zero has value zero), and that the Stirling engine can theoretically achieve this efficiency.
That heat sink is a problem for an automobile (or worse for an aircraft), because in the Stirling model, it has to be able to dissipate ALL of the heat generated in the heat source and maintain a relatively low temperature (perhaps 420K or 150C or 300F). When radiation and air are your media of heat transport, this means a large, probably heavy, radiator. For land and sea based systems, the earth or ocean or other body of water can be used as your heat sink, greatly reducing size and weight (and oh by the way, it isn’t as big a problem in the first place) – and yet, Stirling engines are NOT the engine of choice for power plants or ships, where this problem is mitigated or solved.
The internal combustion engine (whether Diesel or Otto or otherwise) get around this limitation by eliminating much of the heat of combustion through the exhaust of the combustion by-products. Steam locomotives and many other steam engines do similarly by exhausting reduced-temperature steam, greatly reducing the required size of the heat sink. Thermonuclear steam plants are closed-cycled to contain the radiation, but most of the heat is still usually taken away by a secondary water circulation which is either released into the environment or passed through enormous cooling towers.
The internal combustion engine further reduces the system’s mass-to-power-ratio by performing the combustion “on demand” inside the fluid working chamber, greatly reducing the waste due to radiation and conduction and exhaust of the products of combustion. That is, in a steam locomotive, a significant fraction of the heat of the coal burning is sent up the stack, without generating any work. Only the heat which transfers to the water in the boiler generates steam. In a gasoline or diesel engine, the gasses produced by combustion ARE the working fluid that does the work - by doing work, they cool, and are only ejected when they have done all the useful work they can. As such, an internal combustion engine can achieve better efficiency than any Stirling engine of comparable size utilizing the same fuel.
- I believe it also covers the case in which the working fluid is taken from the heat source and eliminated into the heat sink. I expect that this use case is utilized in some geothermal plants.