PBASIC Tokenizer

[Neal Probert]

Tokenizers available for Linux and Windows, to wrap your own Integrated Development Environment (IDE) around.

http://www.parallaxinc.com/html_file..._tokenizer.htm

Word is that the Basic Stamp Editors will soon be available for Mac OS X and Linux soon. That's the last excuse for keeping Windows on my laptop.

Now, I think it would be neat to have a simulator that will work off the tokenizer's output.

[rbayer]

WOW! I was asking Parallax for this stuff last year, but now I have it! Look for a new release of my Graphical program generator (RoboGUI ) that includes the ability to download straight to the RC.

[Greg Ross]

I guess I'll have to update my BASIC Stamp Preprocessor too!

[mtaman02]

looks cool

[rbayer]

Has anyone gotten this stuff to work yet? I keep getting ESP (the register, not the psychic ability) errors. Here's what I have:


typedef bool (*VerifyProc) (TModuleRec *);
struct TModuleRec {...all kinds of stuff...};


then in main I have:
VerifyProc TestRecAlignment;
HINSTANCE ret;
TModuleRec tModRec;

ret=LoadLibrary("tokenizer.dll");
TestRecAlignment=(VerifyProc)GetProcAddress(ret, "TestRecAlignment");
(TestRecAlignment) (&tModRec);

This last call always generates an esp verify error and tells me it is probably because my function pointer uses a different calling convention than the original function. However, I believe I am doing it correctly, but it is hard to verify as the people at Parallax hid the EXPORTs from QuickView.

[EDIT]
Nevermind... I just figured it out. Instead of using the normal __cdecl calling convention, it uses __stdcall. Thus, if you're trying to do something with these libraries, make sure you define your pointers as __stdcall:

typedef bool (__stdcall *TestRecAlignmentProc) (TModuleRec *);

[/EDIT]

[rbayer]

Originally posted by Neal Probert

Now, I think it would be neat to have a simulator that will work off the tokenizer's output.

I'm just curious...why would you want to do it that way? Aside from a purely academic excercise I can't really see a reason. Anyway, if you give me a good one, I may modify my emulator to include this feature, but it would be a LOT of work...

Speaking of which, my emulator is literaly days from its first official release. I'm just working out the bugs with the install program (thank goodness from InstallShield included with MSVC++).

[Greg Ross]

Originally posted by rbayer


I'm just curious...why would you want to do it that way? Aside from a purely academic excercise I can't really see a reason. Anyway, if you give me a good one, I may modify my emulator to include this feature, but it would be a LOT of work...

One reason might be that it's easier to process tokens than to parse possibly syntactically incorrect source code. (I know, you've already coded that, but how sure are you of your code? )

[rbayer]

True, but I don't really feel like reverse-engineering hex code. What I am considering doing, however, is using the Compile function to check syntax and then do my own processing, still based on the original ASCII code. Anyway, I just finished modifying RoboGUI and RoboEmu, and will release them as soon as I mail Parallax my signed license agreement to redistribute their library. Arghh... why couldn't they have open-sourced it (or even better, GPl'd it)?!?

[Greg Ross]

Originally posted by rbayer
True, but I don't really feel like reverse-engineering hex code.

I understand there is a third party book out there that documents the tokenized code. If anyone is interested, I will try digging up the info again.

[Greg Ross]

Originally posted by gwross

I understand there is a third party book out there that documents the tokenized code. If anyone is interested, I will try digging up the info again.

RBayer finally expressed some interest in this book, so I found it again. It is Inside the BASIC Stamp II by Brian Forbes