antlr::TokenStreamRewriteEngine Class Reference
#include <TokenStreamRewriteEngine.hpp>
List of all members.
Classes |
class | DeleteOp |
struct | executeOperation |
class | InsertBeforeOp |
class | ReplaceOp |
class | RewriteOperation |
struct | tokenToStream |
Public Types |
typedef std::vector
< antlr::RefTokenWithIndex > | token_list |
typedef std::list
< RewriteOperation * > | operation_list |
| list of rewrite operations
|
typedef std::map< std::string,
operation_list > | program_map |
| map program name to <program counter,program> tuple
|
Public Member Functions |
| TokenStreamRewriteEngine (TokenStream &upstream) |
| TokenStreamRewriteEngine (TokenStream &upstream, size_t initialSize) |
RefToken | nextToken (void) |
void | rollback (size_t instructionIndex) |
void | rollback (const std::string &programName, size_t instructionIndex) |
void | deleteProgram () |
void | deleteProgram (const std::string &programName) |
void | insertAfter (RefTokenWithIndex t, const std::string &text) |
void | insertAfter (size_t index, const std::string &text) |
void | insertAfter (const std::string &programName, RefTokenWithIndex t, const std::string &text) |
void | insertAfter (const std::string &programName, size_t index, const std::string &text) |
void | insertBefore (RefTokenWithIndex t, const std::string &text) |
void | insertBefore (size_t index, const std::string &text) |
void | insertBefore (const std::string &programName, RefTokenWithIndex t, const std::string &text) |
void | insertBefore (const std::string &programName, size_t index, const std::string &text) |
void | replace (size_t index, const std::string &text) |
void | replace (size_t from, size_t to, const std::string &text) |
void | replace (RefTokenWithIndex indexT, const std::string &text) |
void | replace (RefTokenWithIndex from, RefTokenWithIndex to, const std::string &text) |
void | replace (const std::string &programName, size_t from, size_t to, const std::string &text) |
void | replace (const std::string &programName, RefTokenWithIndex from, RefTokenWithIndex to, const std::string &text) |
void | remove (size_t index) |
void | remove (size_t from, size_t to) |
void | remove (RefTokenWithIndex indexT) |
void | remove (RefTokenWithIndex from, RefTokenWithIndex to) |
void | remove (const std::string &programName, size_t from, size_t to) |
void | remove (const std::string &programName, RefTokenWithIndex from, RefTokenWithIndex to) |
void | discard (int ttype) |
RefToken | getToken (size_t i) |
size_t | getTokenStreamSize () const |
void | originalToStream (std::ostream &out) const |
void | originalToStream (std::ostream &out, size_t start, size_t end) const |
void | toStream (std::ostream &out) const |
void | toStream (std::ostream &out, const std::string &programName) const |
void | toStream (std::ostream &out, size_t start, size_t end) const |
void | toStream (std::ostream &out, const std::string &programName, size_t firstToken, size_t lastToken) const |
void | toDebugStream (std::ostream &out) const |
void | toDebugStream (std::ostream &out, size_t start, size_t end) const |
size_t | getLastRewriteTokenIndex () const |
size_t | getLastRewriteTokenIndex (const std::string &programName) const |
Static Public Attributes |
static const char * | DEFAULT_PROGRAM_NAME = "default" |
static const size_t | MIN_TOKEN_INDEX = 0 |
static const int | PROGRAM_INIT_SIZE = 100 |
Protected Member Functions |
void | addToSortedRewriteList (RewriteOperation *op) |
void | addToSortedRewriteList (const std::string &programName, RewriteOperation *op) |
Protected Attributes |
TokenStream & | stream |
size_t | index |
token_list | tokens |
program_map | programs |
BitSet | discardMask |
Detailed Description
This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser.
This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1
You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)
Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.
Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,
TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2"));
You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer.
If you don't use named rewrite streams, a "default" stream is used.
Terence Parr, parrt@cs.usfca.edu University of San Francisco February 2004
Member Typedef Documentation
list of rewrite operations
map program name to <program counter,program> tuple
Constructor & Destructor Documentation
antlr::TokenStreamRewriteEngine::TokenStreamRewriteEngine |
( |
TokenStream & |
upstream |
) |
|
antlr::TokenStreamRewriteEngine::TokenStreamRewriteEngine |
( |
TokenStream & |
upstream, |
|
|
size_t |
initialSize | |
|
) |
| | |
Member Function Documentation
void antlr::TokenStreamRewriteEngine::addToSortedRewriteList |
( |
const std::string & |
programName, |
|
|
RewriteOperation * |
op | |
|
) |
| | [protected] |
void antlr::TokenStreamRewriteEngine::addToSortedRewriteList |
( |
RewriteOperation * |
op |
) |
[inline, protected] |
If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear
void antlr::TokenStreamRewriteEngine::deleteProgram |
( |
const std::string & |
programName |
) |
[inline] |
Reset the program so that no instructions exist
void antlr::TokenStreamRewriteEngine::deleteProgram |
( |
|
) |
[inline] |
void antlr::TokenStreamRewriteEngine::discard |
( |
int |
ttype |
) |
[inline] |
size_t antlr::TokenStreamRewriteEngine::getLastRewriteTokenIndex |
( |
const std::string & |
programName |
) |
const [inline] |
Return the last index for the program named programName return 0 if the program does not exist or the program is empty. (Note this is different from the java implementation that returns -1)
size_t antlr::TokenStreamRewriteEngine::getLastRewriteTokenIndex |
( |
|
) |
const [inline] |
RefToken antlr::TokenStreamRewriteEngine::getToken |
( |
size_t |
i |
) |
[inline] |
size_t antlr::TokenStreamRewriteEngine::getTokenStreamSize |
( |
|
) |
const [inline] |
void antlr::TokenStreamRewriteEngine::insertAfter |
( |
const std::string & |
programName, |
|
|
size_t |
index, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::insertAfter |
( |
const std::string & |
programName, |
|
|
RefTokenWithIndex |
t, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::insertAfter |
( |
size_t |
index, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::insertAfter |
( |
RefTokenWithIndex |
t, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::insertBefore |
( |
const std::string & |
programName, |
|
|
size_t |
index, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::insertBefore |
( |
const std::string & |
programName, |
|
|
RefTokenWithIndex |
t, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::insertBefore |
( |
size_t |
index, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::insertBefore |
( |
RefTokenWithIndex |
t, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
RefToken antlr::TokenStreamRewriteEngine::nextToken |
( |
void |
|
) |
[virtual] |
void antlr::TokenStreamRewriteEngine::originalToStream |
( |
std::ostream & |
out, |
|
|
size_t |
start, |
|
|
size_t |
end | |
|
) |
| | const |
void antlr::TokenStreamRewriteEngine::originalToStream |
( |
std::ostream & |
out |
) |
const [inline] |
void antlr::TokenStreamRewriteEngine::remove |
( |
const std::string & |
programName, |
|
|
size_t |
from, |
|
|
size_t |
to | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::remove |
( |
size_t |
from, |
|
|
size_t |
to | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::remove |
( |
size_t |
index |
) |
[inline] |
void antlr::TokenStreamRewriteEngine::replace |
( |
const std::string & |
programName, |
|
|
size_t |
from, |
|
|
size_t |
to, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::replace |
( |
RefTokenWithIndex |
indexT, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::replace |
( |
size_t |
from, |
|
|
size_t |
to, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::replace |
( |
size_t |
index, |
|
|
const std::string & |
text | |
|
) |
| | [inline] |
void antlr::TokenStreamRewriteEngine::rollback |
( |
const std::string & |
programName, |
|
|
size_t |
instructionIndex | |
|
) |
| | |
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!
void antlr::TokenStreamRewriteEngine::rollback |
( |
size_t |
instructionIndex |
) |
[inline] |
void antlr::TokenStreamRewriteEngine::toDebugStream |
( |
std::ostream & |
out, |
|
|
size_t |
start, |
|
|
size_t |
end | |
|
) |
| | const |
void antlr::TokenStreamRewriteEngine::toDebugStream |
( |
std::ostream & |
out |
) |
const [inline] |
void antlr::TokenStreamRewriteEngine::toStream |
( |
std::ostream & |
out, |
|
|
const std::string & |
programName, |
|
|
size_t |
firstToken, |
|
|
size_t |
lastToken | |
|
) |
| | const |
void antlr::TokenStreamRewriteEngine::toStream |
( |
std::ostream & |
out, |
|
|
size_t |
start, |
|
|
size_t |
end | |
|
) |
| | const [inline] |
void antlr::TokenStreamRewriteEngine::toStream |
( |
std::ostream & |
out, |
|
|
const std::string & |
programName | |
|
) |
| | const [inline] |
void antlr::TokenStreamRewriteEngine::toStream |
( |
std::ostream & |
out |
) |
const [inline] |
Member Data Documentation
Which (whitespace) token(s) to throw out
You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)
Who do we suck tokens from?
Track the incoming list of tokens
The documentation for this class was generated from the following files: