In theory, it’s possible to drive mixed sized wheels by using pulleys or sprockets with identical ratios. So, put a 24 tooth pulley on a 4" wheel and a 36 tooth pulley on a 6" wheel.
In reality, I’m worried about the 6" wheel actually being 5.95" and the 4" wheel actually being 4.03", in which case the wheels would constantly fight each other. But, that problem also exists when you have different wheels of the same nominal diameter (e.g. a 6" traction and a 6" omni), and doesn’t seem to cause problems there.
Has anybody done different wheel sizes? What issues did you run into?