I just realized that I wrote this paper and posted it on my website, but never actually mentioned it here. Anyway, after writing RoboGUI and RoboIDE, I've become fairly proficient with the tokenizer that Parallax now provides, so I thought I'd try to pass on some of that knowledge.
As of right now, the document pertains specifically to Windows and the dll version, but could be easily adapted to Linux by using the functions dlopen, dlsym, and dlclose instead of the Windows LoadLibrary, etc.
Anyway, as always, it is available at my new website:
http://www.robbayer.com under the HOW-TO's section
--Rob