Download ANTLR Reference Manual

Token Streams
Traditionally, a lexer and parser are tightly coupled objects; that is, one does not imagine
anything sitting between the parser and the lexer, modifying the stream of tokens. However,
language recognition and translation can benefit greatly from treating the connection between
lexer and parser as a token stream. This idea is analogous to Java I/O streams, where you can
pipeline lots of stream objects to produce highly-processed data streams.
Introduction
ANTLR identifies a stream of Token objects as any object that satisfies the TokenStream
interface (prior to 2.6, this interface was called Tokenizer); i.e., any object that implements the
following method.
Token nextToken();
Graphically, a normal stream of tokens from a lexer (producer) to a parser (consumer) might
look like the following at some point during the parse.
The most common token stream is a lexer, but once you imagine a physical stream between the
lexer and parser, you start imagining interesting things that you can do. For example, you can:
•
•
•
•
filter a stream of tokens to strip out unwanted tokens
insert imaginary tokens to help the parser recognize certain nasty structures
split a single stream into multiple streams, sending certain tokens of interest down the
various streams
multiplex multiple token streams onto one stream, thus, "simulating" the lexer states of
tools like PCCTS, lex, and so on.
The beauty of the token stream concept is that parsers and lexers are not affected--they are
merely consumers and producers of streams. Stream objects are filters that produce, process,
combine, or separate token streams for use by consumers. Existing lexers and parsers may be
combined in new and interesting ways without modification.
This document formalizes the notion of a token stream and describes in detail some very useful
stream filters.
Pass-Through Token Stream
A token stream is any object satisfying the following interface.
public interface TokenStream {
public Token nextToken()
throws java.io.IOException;
}
For example, a "no-op" or pass-through filter stream looks like:
69