Just an update:
After a few tweaks to optimize the code, we are tracking the target at 640x480, 30! frames per second. I Don't think it can get any better.
of course, unless we get a camera with 60fps capability
Will try to get a video posted if possible.