antlr

Class PythonCodeGenerator

public class PythonCodeGenerator extends CodeGenerator

Generate MyParser.java, MyLexer.java and MyParserTokenTypes.java
Field Summary
intastVarNumber
static intcaseSizeThreshold
StringcommonExtraArgs
StringcommonExtraParams
StringcommonLocalVars
StringcurrentASTResult
Tracks the rule or labeled subrule being generated.
RuleBlockcurrentRule
Tracks the rule being generated.
HashtabledeclaredASTVariables
Used to keep track of which AST variables have been defined in a rule (except for the #rule_name and #rule_name_in var's
StringexceptionThrown
protected booleangenAST
static StringinitHeaderAction
StringlabeledElementASTType
StringlabeledElementInit
StringlabeledElementType
StringlexerClassName
Stringlt1Value
static StringmainHeaderAction
protected static StringNONUNIQUE
Special value used to mark duplicate in treeVariableMap
StringparserClassName
protected booleansaveText
protected intsyntacticPredLevel
StringthrowNoViable
HashtabletreeVariableMap
Mapping between the ids used in the current alt, and the names of variables used to represent their AST values.
StringtreeWalkerClassName
Constructor Summary
PythonCodeGenerator()
Method Summary
protected intaddSemPred(String predicate)
Adds a semantic predicate string to the sem pred vector These strings will be used to build an array of sem pred names when building a debugging parser.
protected voidcheckCurrentOutputStream()
voidexitIfError()
protected StringextractIdOfAction(String s, int line, int column)
Get the identifier portion of an argument-action.
protected StringextractTypeOfAction(String s, int line, int column)
Get the type portion of an argument-action.
protected voidflushTokens()
voidgen()
Generate the parser, lexer, treeparser, and token types in Java
voidgen(ActionElement action)
Generate code for the given grammar element.
voidgen(AlternativeBlock blk)
Generate code for the given grammar element.
voidgen(BlockEndElement end)
Generate code for the given grammar element.
voidgen(CharLiteralElement atom)
Generate code for the given grammar element.
voidgen(CharRangeElement r)
Generate code for the given grammar element.
voidgen(LexerGrammar g)
Generate the lexer Java file
voidgen(OneOrMoreBlock blk)
Generate code for the given grammar element.
voidgen(ParserGrammar g)
Generate the parser Java file
voidgen(RuleRefElement rr)
Generate code for the given grammar element.
voidgen(StringLiteralElement atom)
Generate code for the given grammar element.
voidgen(TokenRangeElement r)
Generate code for the given grammar element.
voidgen(TokenRefElement atom)
Generate code for the given grammar element.
voidgen(TreeElement t)
voidgen(TreeWalkerGrammar g)
Generate the tree-parser Java file
voidgen(WildcardElement wc)
Generate code for the given grammar element.
voidgen(ZeroOrMoreBlock blk)
Generate code for the given grammar element.
protected voidgenAlt(Alternative alt, AlternativeBlock blk)
Generate an alternative.
protected voidgenASTDeclaration(AlternativeElement el)
protected voidgenASTDeclaration(AlternativeElement el, String node_type)
protected voidgenASTDeclaration(AlternativeElement el, String var_name, String node_type)
protected voidgenBitsets(Vector bitsetList, int maxVocabulary)
Generate all the bitsets to be used in the parser or lexer Generate the raw bitset data like "long _tokenSet1_data[] = {...}" and the BitSet object declarations like "BitSet _tokenSet1 = new BitSet(_tokenSet1_data)" Note that most languages do not support object initialization inside a class definition, so other code-generators may have to separate the bitset declarations from the initializations (e.g., put the initializations in the generated constructor instead).
protected voidgenBlockInitAction(AlternativeBlock blk)
Generate the init action for a block, which may be a RuleBlock or a plain AlternativeBLock.
protected voidgenBlockPreamble(AlternativeBlock blk)
Generate the header for a block, which may be a RuleBlock or a plain AlternativeBLock.
protected voidgenCases(BitSet p)
Generate a series of case statements that implement a BitSet test.
PythonBlockFinishingInfogenCommonBlock(AlternativeBlock blk, boolean noTestForSingle)
Generate common code for a block of alternatives; return a postscript that needs to be generated at the end of the block.
protected voidgenHeader()
Generate a header that is common to all Python files
protected voidgenHeaderInit(Grammar grammar)
protected voidgenHeaderMain(Grammar grammar)
protected voidgenJavadocComment(Grammar g)
protected voidgenJavadocComment(RuleSymbol g)
protected voidgenLexerTest()
Generate an automated test for Python CharScanner (sub)classes.
protected voidgenMatch(BitSet b)
protected voidgenMatch(GrammarAtom atom)
protected voidgenMatchUsingAtomText(GrammarAtom atom)
protected voidgenMatchUsingAtomTokenType(GrammarAtom atom)
voidgenNextToken()
Generate the nextToken() rule. nextToken() is a synthetic lexer rule that is the implicit OR of all user-defined lexer rules.
voidgenRule(RuleSymbol s, boolean startSymbol, int ruleNum)
Gen a named rule block.
protected voidgenSemPred(String pred, int line)
protected voidgenSemPredMap()
Write an array of Strings which are the semantic predicate expressions.
protected voidgenSynPred(SynPredBlock blk, String lookaheadExpr)
protected voidgenTokenASTNodeMap()
Create and set Integer token type objects that map to Java Class objects (which AST node to create).
voidgenTokenStrings()
Generate a static array containing the names of the tokens, indexed by the token type values.
protected voidgenTokenTypes(TokenManager tm)
Generate the token types Java file
StringgetASTCreateString(Vector v)
Get a string for an expression to generate creation of an AST subtree.
StringgetASTCreateString(GrammarAtom atom, String astCtorArgs)
Get a string for an expression to generate creating of an AST node
StringgetASTCreateString(String astCtorArgs)
Get a string for an expression to generate creating of an AST node.
protected StringgetLookaheadTestExpression(Lookahead[] look, int k)
protected StringgetLookaheadTestExpression(Alternative alt, int maxDepth)
Generate a lookahead test expression for an alternate.
protected StringgetLookaheadTestTerm(int k, BitSet p)
Generate a depth==1 lookahead test expression given the BitSet.
StringgetRangeExpression(int k, int[] elems)
Return an expression for testing a contiguous renage of elements
static booleanisEmpty(String s)
protected booleanisspace(char c)
protected booleanlookaheadIsEmpty(Alternative alt, int maxDepth)
Is the lookahead for this alt empty?
StringmapTreeId(String idParam, ActionTransInfo transInfo)
Map an identifier to it's corresponding tree-node variable.
protected voidod(String s, int i, int end, String msg)
protected voidprintAction(String s)
protected voidprintActionCode(String actionStr, int line)
protected voidprintGrammarAction(Grammar grammar)
protected voidprintMainFunc(String s)
protected voidprintTabs()
Create a Java code-generator using the given Grammar.
protected StringprocessActionCode(String actionStr, int line)
protected StringprocessActionForSpecialSymbols(String actionStr, int line, RuleBlock currentRule, ActionTransInfo tInfo)
Lexically process $var and tree-specifiers in the action.
voidsetupOutput(String className)
This method exists so a subclass, namely VAJCodeGenerator, can open the file in its own evil way.
StringtoString(boolean v)
protected void_printAction(String s)
protected void_printJavadoc(String s)

Field Detail

astVarNumber

int astVarNumber

caseSizeThreshold

public static final int caseSizeThreshold

commonExtraArgs

String commonExtraArgs

commonExtraParams

String commonExtraParams

commonLocalVars

String commonLocalVars

currentASTResult

String currentASTResult
Tracks the rule or labeled subrule being generated. Used for AST generation.

currentRule

RuleBlock currentRule
Tracks the rule being generated. Used for mapTreeId

declaredASTVariables

Hashtable declaredASTVariables
Used to keep track of which AST variables have been defined in a rule (except for the #rule_name and #rule_name_in var's

exceptionThrown

String exceptionThrown

genAST

protected boolean genAST

initHeaderAction

public static final String initHeaderAction

labeledElementASTType

String labeledElementASTType

labeledElementInit

String labeledElementInit

labeledElementType

String labeledElementType

lexerClassName

String lexerClassName

lt1Value

String lt1Value

mainHeaderAction

public static final String mainHeaderAction

NONUNIQUE

protected static final String NONUNIQUE
Special value used to mark duplicate in treeVariableMap

parserClassName

String parserClassName

saveText

protected boolean saveText

syntacticPredLevel

protected int syntacticPredLevel

throwNoViable

String throwNoViable

treeVariableMap

Hashtable treeVariableMap
Mapping between the ids used in the current alt, and the names of variables used to represent their AST values.

treeWalkerClassName

String treeWalkerClassName

Constructor Detail

PythonCodeGenerator

public PythonCodeGenerator()

Method Detail

addSemPred

protected int addSemPred(String predicate)
Adds a semantic predicate string to the sem pred vector These strings will be used to build an array of sem pred names when building a debugging parser. This method should only be called when the debug option is specified

checkCurrentOutputStream

protected void checkCurrentOutputStream()

exitIfError

public void exitIfError()

extractIdOfAction

protected String extractIdOfAction(String s, int line, int column)
Get the identifier portion of an argument-action. For Python the ID of an action is assumed to be everything before the assignment, as Python does not support a type.

Parameters: s The action text line Line used for error reporting. column Line used for error reporting.

Returns: A string containing the text of the identifier

extractTypeOfAction

protected String extractTypeOfAction(String s, int line, int column)
Get the type portion of an argument-action. Python does not have a type declaration before an identifier, so we just return the empty string.

Parameters: s The action text line Line used for error reporting.

Returns: A string containing the text of the type

flushTokens

protected void flushTokens()

gen

public void gen()
Generate the parser, lexer, treeparser, and token types in Java

gen

public void gen(ActionElement action)
Generate code for the given grammar element.

Parameters: blk The {...} action to generate

gen

public void gen(AlternativeBlock blk)
Generate code for the given grammar element.

Parameters: blk The "x|y|z|..." block to generate

gen

public void gen(BlockEndElement end)
Generate code for the given grammar element.

Parameters: blk The block-end element to generate. Block-end elements are synthesized by the grammar parser to represent the end of a block.

gen

public void gen(CharLiteralElement atom)
Generate code for the given grammar element.

Parameters: blk The character literal reference to generate

gen

public void gen(CharRangeElement r)
Generate code for the given grammar element.

Parameters: blk The character-range reference to generate

gen

public void gen(LexerGrammar g)
Generate the lexer Java file

gen

public void gen(OneOrMoreBlock blk)
Generate code for the given grammar element.

Parameters: blk The (...)+ block to generate

gen

public void gen(ParserGrammar g)
Generate the parser Java file

gen

public void gen(RuleRefElement rr)
Generate code for the given grammar element.

Parameters: blk The rule-reference to generate

gen

public void gen(StringLiteralElement atom)
Generate code for the given grammar element.

Parameters: blk The string-literal reference to generate

gen

public void gen(TokenRangeElement r)
Generate code for the given grammar element.

Parameters: blk The token-range reference to generate

gen

public void gen(TokenRefElement atom)
Generate code for the given grammar element.

Parameters: blk The token-reference to generate

gen

public void gen(TreeElement t)

gen

public void gen(TreeWalkerGrammar g)
Generate the tree-parser Java file

gen

public void gen(WildcardElement wc)
Generate code for the given grammar element.

Parameters: wc The wildcard element to generate

gen

public void gen(ZeroOrMoreBlock blk)
Generate code for the given grammar element.

Parameters: blk The (...)* block to generate

genAlt

protected void genAlt(Alternative alt, AlternativeBlock blk)
Generate an alternative.

Parameters: alt The alternative to generate blk The block to which the alternative belongs

genASTDeclaration

protected void genASTDeclaration(AlternativeElement el)

genASTDeclaration

protected void genASTDeclaration(AlternativeElement el, String node_type)

genASTDeclaration

protected void genASTDeclaration(AlternativeElement el, String var_name, String node_type)

genBitsets

protected void genBitsets(Vector bitsetList, int maxVocabulary)
Generate all the bitsets to be used in the parser or lexer Generate the raw bitset data like "long _tokenSet1_data[] = {...}" and the BitSet object declarations like "BitSet _tokenSet1 = new BitSet(_tokenSet1_data)" Note that most languages do not support object initialization inside a class definition, so other code-generators may have to separate the bitset declarations from the initializations (e.g., put the initializations in the generated constructor instead).

Parameters: bitsetList The list of bitsets to generate. maxVocabulary Ensure that each generated bitset can contain at least this value.

genBlockInitAction

protected void genBlockInitAction(AlternativeBlock blk)
Generate the init action for a block, which may be a RuleBlock or a plain AlternativeBLock.

UNKNOWN: The block for which the preamble is to be generated.

genBlockPreamble

protected void genBlockPreamble(AlternativeBlock blk)
Generate the header for a block, which may be a RuleBlock or a plain AlternativeBLock. This generates any variable declarations and syntactic-predicate-testing variables.

UNKNOWN: The block for which the preamble is to be generated.

genCases

protected void genCases(BitSet p)
Generate a series of case statements that implement a BitSet test.

Parameters: p The Bitset for which cases are to be generated

genCommonBlock

public PythonBlockFinishingInfo genCommonBlock(AlternativeBlock blk, boolean noTestForSingle)
Generate common code for a block of alternatives; return a postscript that needs to be generated at the end of the block. Other routines may append else-clauses and such for error checking before the postfix is generated. If the grammar is a lexer, then generate alternatives in an order where alternatives requiring deeper lookahead are generated first, and EOF in the lookahead set reduces the depth of the lookahead. @param blk The block to generate @param noTestForSingle If true, then it does not generate a test for a single alternative.

genHeader

protected void genHeader()
Generate a header that is common to all Python files

genHeaderInit

protected void genHeaderInit(Grammar grammar)

genHeaderMain

protected void genHeaderMain(Grammar grammar)

genJavadocComment

protected void genJavadocComment(Grammar g)

genJavadocComment

protected void genJavadocComment(RuleSymbol g)

genLexerTest

protected void genLexerTest()
Generate an automated test for Python CharScanner (sub)classes.

genMatch

protected void genMatch(BitSet b)

genMatch

protected void genMatch(GrammarAtom atom)

genMatchUsingAtomText

protected void genMatchUsingAtomText(GrammarAtom atom)

genMatchUsingAtomTokenType

protected void genMatchUsingAtomTokenType(GrammarAtom atom)

genNextToken

public void genNextToken()
Generate the nextToken() rule. nextToken() is a synthetic lexer rule that is the implicit OR of all user-defined lexer rules.

genRule

public void genRule(RuleSymbol s, boolean startSymbol, int ruleNum)
Gen a named rule block. ASTs are generated for each element of an alternative unless the rule or the alternative have a '!' modifier. If an alternative defeats the default tree construction, it must set _AST to the root of the returned AST. Each alternative that does automatic tree construction, builds up root and child list pointers in an ASTPair structure. A rule finishes by setting the returnAST variable from the ASTPair.

Parameters: rule The name of the rule to generate startSymbol true if the rule is a start symbol (i.e., not referenced elsewhere)

genSemPred

protected void genSemPred(String pred, int line)

genSemPredMap

protected void genSemPredMap()
Write an array of Strings which are the semantic predicate expressions. The debugger will reference them by number only

genSynPred

protected void genSynPred(SynPredBlock blk, String lookaheadExpr)

genTokenASTNodeMap

protected void genTokenASTNodeMap()
Create and set Integer token type objects that map to Java Class objects (which AST node to create).

genTokenStrings

public void genTokenStrings()
Generate a static array containing the names of the tokens, indexed by the token type values. This static array is used to format error messages so that the token identifers or literal strings are displayed instead of the token numbers. If a lexical rule has a paraphrase, use it rather than the token label.

genTokenTypes

protected void genTokenTypes(TokenManager tm)
Generate the token types Java file

getASTCreateString

public String getASTCreateString(Vector v)
Get a string for an expression to generate creation of an AST subtree.

Parameters: v A Vector of String, where each element is an expression in the target language yielding an AST node.

getASTCreateString

public String getASTCreateString(GrammarAtom atom, String astCtorArgs)
Get a string for an expression to generate creating of an AST node

Parameters: atom The grammar node for which you are creating the node str The arguments to the AST constructor

getASTCreateString

public String getASTCreateString(String astCtorArgs)
Get a string for an expression to generate creating of an AST node. Parse the first (possibly only) argument looking for the token type. If the token type is a valid token symbol, ask for it's AST node type and add to the end if only 2 arguments. The forms are #[T], #[T,"t"], and as of 2.7.2 #[T,"t",ASTclassname].

Parameters: str The arguments to the AST constructor

getLookaheadTestExpression

protected String getLookaheadTestExpression(Lookahead[] look, int k)

getLookaheadTestExpression

protected String getLookaheadTestExpression(Alternative alt, int maxDepth)
Generate a lookahead test expression for an alternate. This will be a series of tests joined by '&&' and enclosed by '()', the number of such tests being determined by the depth of the lookahead.

getLookaheadTestTerm

protected String getLookaheadTestTerm(int k, BitSet p)
Generate a depth==1 lookahead test expression given the BitSet. This may be one of: 1) a series of 'x==X||' tests 2) a range test using >= && <= where possible, 3) a bitset membership test for complex comparisons

Parameters: k The lookahead level p The lookahead set for level k

getRangeExpression

public String getRangeExpression(int k, int[] elems)
Return an expression for testing a contiguous renage of elements

Parameters: k The lookahead level elems The elements representing the set, usually from BitSet.toArray().

Returns: String containing test expression.

isEmpty

static boolean isEmpty(String s)

isspace

protected boolean isspace(char c)

lookaheadIsEmpty

protected boolean lookaheadIsEmpty(Alternative alt, int maxDepth)
Is the lookahead for this alt empty?

mapTreeId

public String mapTreeId(String idParam, ActionTransInfo transInfo)
Map an identifier to it's corresponding tree-node variable. This is context-sensitive, depending on the rule and alternative being generated

Parameters: idParam The identifier name to map

Returns: The mapped id (which may be the same as the input), or null if the mapping is invalid due to duplicates

od

protected void od(String s, int i, int end, String msg)

printAction

protected void printAction(String s)

printActionCode

protected void printActionCode(String actionStr, int line)

printGrammarAction

protected void printGrammarAction(Grammar grammar)

printMainFunc

protected void printMainFunc(String s)

printTabs

protected void printTabs()
Create a Java code-generator using the given Grammar. The caller must still call setTool, setBehavior, and setAnalyzer before generating code.

processActionCode

protected String processActionCode(String actionStr, int line)

processActionForSpecialSymbols

protected String processActionForSpecialSymbols(String actionStr, int line, RuleBlock currentRule, ActionTransInfo tInfo)
Lexically process $var and tree-specifiers in the action. This will replace #id and #(...) with the appropriate function calls and/or variables etc...

setupOutput

public void setupOutput(String className)
This method exists so a subclass, namely VAJCodeGenerator, can open the file in its own evil way. JavaCodeGenerator simply opens a text file...

toString

String toString(boolean v)

_printAction

protected void _printAction(String s)

_printJavadoc

protected void _printJavadoc(String s)