antlr::TokenStreamRewriteEngine Class Reference

#include <TokenStreamRewriteEngine.hpp>

Inheritance diagram for antlr::TokenStreamRewriteEngine:
[legend]
Collaboration diagram for antlr::TokenStreamRewriteEngine:
[legend]

List of all members.

Classes

class  DeleteOp
struct  executeOperation
class  InsertBeforeOp
class  ReplaceOp
class  RewriteOperation
struct  tokenToStream

Public Types

typedef std::vector
< antlr::RefTokenWithIndex
token_list
typedef std::list
< RewriteOperation * > 
operation_list
 list of rewrite operations
typedef std::map< std::string,
operation_list
program_map
 map program name to <program counter,program> tuple

Public Member Functions

 TokenStreamRewriteEngine (TokenStream &upstream)
 TokenStreamRewriteEngine (TokenStream &upstream, size_t initialSize)
RefToken nextToken (void)
void rollback (size_t instructionIndex)
void rollback (const std::string &programName, size_t instructionIndex)
void deleteProgram ()
void deleteProgram (const std::string &programName)
void insertAfter (RefTokenWithIndex t, const std::string &text)
void insertAfter (size_t index, const std::string &text)
void insertAfter (const std::string &programName, RefTokenWithIndex t, const std::string &text)
void insertAfter (const std::string &programName, size_t index, const std::string &text)
void insertBefore (RefTokenWithIndex t, const std::string &text)
void insertBefore (size_t index, const std::string &text)
void insertBefore (const std::string &programName, RefTokenWithIndex t, const std::string &text)
void insertBefore (const std::string &programName, size_t index, const std::string &text)
void replace (size_t index, const std::string &text)
void replace (size_t from, size_t to, const std::string &text)
void replace (RefTokenWithIndex indexT, const std::string &text)
void replace (RefTokenWithIndex from, RefTokenWithIndex to, const std::string &text)
void replace (const std::string &programName, size_t from, size_t to, const std::string &text)
void replace (const std::string &programName, RefTokenWithIndex from, RefTokenWithIndex to, const std::string &text)
void remove (size_t index)
void remove (size_t from, size_t to)
void remove (RefTokenWithIndex indexT)
void remove (RefTokenWithIndex from, RefTokenWithIndex to)
void remove (const std::string &programName, size_t from, size_t to)
void remove (const std::string &programName, RefTokenWithIndex from, RefTokenWithIndex to)
void discard (int ttype)
RefToken getToken (size_t i)
size_t getTokenStreamSize () const
void originalToStream (std::ostream &out) const
void originalToStream (std::ostream &out, size_t start, size_t end) const
void toStream (std::ostream &out) const
void toStream (std::ostream &out, const std::string &programName) const
void toStream (std::ostream &out, size_t start, size_t end) const
void toStream (std::ostream &out, const std::string &programName, size_t firstToken, size_t lastToken) const
void toDebugStream (std::ostream &out) const
void toDebugStream (std::ostream &out, size_t start, size_t end) const
size_t getLastRewriteTokenIndex () const
size_t getLastRewriteTokenIndex (const std::string &programName) const

Static Public Attributes

static const char * DEFAULT_PROGRAM_NAME = "default"
static const size_t MIN_TOKEN_INDEX = 0
static const int PROGRAM_INIT_SIZE = 100

Protected Member Functions

void addToSortedRewriteList (RewriteOperation *op)
void addToSortedRewriteList (const std::string &programName, RewriteOperation *op)

Protected Attributes

TokenStreamstream
size_t index
token_list tokens
program_map programs
BitSet discardMask

Detailed Description

This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser.

This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2"));

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer.

If you don't use named rewrite streams, a "default" stream is used.

Terence Parr, parrt@cs.usfca.edu University of San Francisco February 2004


Member Typedef Documentation

list of rewrite operations

map program name to <program counter,program> tuple


Constructor & Destructor Documentation

antlr::TokenStreamRewriteEngine::TokenStreamRewriteEngine ( TokenStream upstream  ) 
antlr::TokenStreamRewriteEngine::TokenStreamRewriteEngine ( TokenStream upstream,
size_t  initialSize 
)

Member Function Documentation

void antlr::TokenStreamRewriteEngine::addToSortedRewriteList ( const std::string &  programName,
RewriteOperation op 
) [protected]
void antlr::TokenStreamRewriteEngine::addToSortedRewriteList ( RewriteOperation op  )  [inline, protected]

If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear

void antlr::TokenStreamRewriteEngine::deleteProgram ( const std::string &  programName  )  [inline]

Reset the program so that no instructions exist

void antlr::TokenStreamRewriteEngine::deleteProgram (  )  [inline]
void antlr::TokenStreamRewriteEngine::discard ( int  ttype  )  [inline]
size_t antlr::TokenStreamRewriteEngine::getLastRewriteTokenIndex ( const std::string &  programName  )  const [inline]

Return the last index for the program named programName return 0 if the program does not exist or the program is empty. (Note this is different from the java implementation that returns -1)

size_t antlr::TokenStreamRewriteEngine::getLastRewriteTokenIndex (  )  const [inline]
RefToken antlr::TokenStreamRewriteEngine::getToken ( size_t  i  )  [inline]
size_t antlr::TokenStreamRewriteEngine::getTokenStreamSize (  )  const [inline]
void antlr::TokenStreamRewriteEngine::insertAfter ( const std::string &  programName,
size_t  index,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::insertAfter ( const std::string &  programName,
RefTokenWithIndex  t,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::insertAfter ( size_t  index,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::insertAfter ( RefTokenWithIndex  t,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::insertBefore ( const std::string &  programName,
size_t  index,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::insertBefore ( const std::string &  programName,
RefTokenWithIndex  t,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::insertBefore ( size_t  index,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::insertBefore ( RefTokenWithIndex  t,
const std::string &  text 
) [inline]
RefToken antlr::TokenStreamRewriteEngine::nextToken ( void   )  [virtual]

Implements antlr::TokenStream.

void antlr::TokenStreamRewriteEngine::originalToStream ( std::ostream &  out,
size_t  start,
size_t  end 
) const
void antlr::TokenStreamRewriteEngine::originalToStream ( std::ostream &  out  )  const [inline]
void antlr::TokenStreamRewriteEngine::remove ( const std::string &  programName,
RefTokenWithIndex  from,
RefTokenWithIndex  to 
) [inline]
void antlr::TokenStreamRewriteEngine::remove ( const std::string &  programName,
size_t  from,
size_t  to 
) [inline]
void antlr::TokenStreamRewriteEngine::remove ( RefTokenWithIndex  from,
RefTokenWithIndex  to 
) [inline]
void antlr::TokenStreamRewriteEngine::remove ( RefTokenWithIndex  indexT  )  [inline]
void antlr::TokenStreamRewriteEngine::remove ( size_t  from,
size_t  to 
) [inline]
void antlr::TokenStreamRewriteEngine::remove ( size_t  index  )  [inline]
void antlr::TokenStreamRewriteEngine::replace ( const std::string &  programName,
RefTokenWithIndex  from,
RefTokenWithIndex  to,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::replace ( const std::string &  programName,
size_t  from,
size_t  to,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::replace ( RefTokenWithIndex  from,
RefTokenWithIndex  to,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::replace ( RefTokenWithIndex  indexT,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::replace ( size_t  from,
size_t  to,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::replace ( size_t  index,
const std::string &  text 
) [inline]
void antlr::TokenStreamRewriteEngine::rollback ( const std::string &  programName,
size_t  instructionIndex 
)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

void antlr::TokenStreamRewriteEngine::rollback ( size_t  instructionIndex  )  [inline]
void antlr::TokenStreamRewriteEngine::toDebugStream ( std::ostream &  out,
size_t  start,
size_t  end 
) const
void antlr::TokenStreamRewriteEngine::toDebugStream ( std::ostream &  out  )  const [inline]
void antlr::TokenStreamRewriteEngine::toStream ( std::ostream &  out,
const std::string &  programName,
size_t  firstToken,
size_t  lastToken 
) const
void antlr::TokenStreamRewriteEngine::toStream ( std::ostream &  out,
size_t  start,
size_t  end 
) const [inline]
void antlr::TokenStreamRewriteEngine::toStream ( std::ostream &  out,
const std::string &  programName 
) const [inline]
void antlr::TokenStreamRewriteEngine::toStream ( std::ostream &  out  )  const [inline]

Member Data Documentation

const char * antlr::TokenStreamRewriteEngine::DEFAULT_PROGRAM_NAME = "default" [static]

Which (whitespace) token(s) to throw out

track index of tokens

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)

Who do we suck tokens from?

Track the incoming list of tokens


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Friends Defines

Generated on 16 Nov 2015 for ANTLR Support Libraries 2.7.1+ by  doxygen 1.6.1