

		The regular expressions plugin

This plugin allows for text processing using regular 
expressions. The GPL'ed package {pcre} is included
for this purpose (pcre stands for Perl compatible 
regular expressions).

To use the plugin, the plugin has to be loaded through:

	Use("pcre");

The {pcre} plugin offers two principal functions:
{PcreLexer} and {PcreNextToken}. {PcreLexer} sets up
and initializes the tokenizer to scan for a set of
regular expressions. A call to {PcreNextToken} 
returns the result of the scan in the current file,
starting from the current position.

{PcreLexer} does not interfere with normal reading of a 
file. When normal read calls are made, the default 
tokenizer is used. The tokenizer set up by {PcreLexer}
is only used when {PcreNextToken} is called.

The syntax for {PcreLexer} is:

	PcreLexer(tokens);

The argument {tokens} is a list of token definitions. 
A token definition in turn is a list, with the first
element a string containing the regular expression, and
the second containing an atom that can describe the type
of the token (for easy recognition later).

Calling {PcreNextToken()} will then return the first
matching token from the current input.

The following example sets up the tokenizer to recognize
either an integer, or a word consisting only of lowercase
letters.

	In> DllLoad("pcre")
	Out> True
	In> PcreLexer({{"[0-9]+",Integer},{"[a-z]+",Word}})
	Out> True
	In> FromString("123abc")PcreNextToken()
	Out> {"123",Integer}
	In> FromString("===abc")PcreNextToken()
	Out> {"abc",Word}



