"compile" vs. "tokenize" (was Re: Password Problem)

[Kenneth Brody]

"Jean-Pierre A. Radley" wrote:
[...]
> | Well, to be "buzzword compliant", I would say that "filePro compiles to
> | a bytecode for the filePro Virtual Machine".
> |
> | Basically, it gets compiled to machine code.  It's just that the machine
> | that it compiles for is not the physical machine that you're running.
> | http://mailman.celestial.com/mailman/listinfo/filepro-list
> 
> So: in the context of filePro, what does "tokenize" mean?

That Dave made a bad choice in terminology?  ;-)

To me, "tokenize" has meant "convert keywords into special 'tokens'
so that the runtime interpreter doesn't have to waste time finding
keywords".

For example, a BASIC interpreter that tokenizes "PRINT" into 0x80,
"GOTO" into 0x81, and "GOSUB" into 0x82 might "tokenize":

    gosub 50 ; print "hello" ; goto 10

into:

    0x82  5  0  ;  0x80  "  h  e  l  l  o  "  ;  0x81  1  0  0x00

Nothing else has been done, other than to pre-recognize keywords.

In filePro "tokenize" means "compile into a tok file".  ;-)

-- 
+-------------------------+--------------------+-----------------------------+
| Kenneth J. Brody        | www.hvcomputer.com |                             |
| kenbrody at spamcop.net | www.fptech.com     | #include <std_disclaimer.h> |
+-------------------------+--------------------+-----------------------------+