public abstract class GraphTokenFilter extends TokenFilter
incrementBaseToken()
to move the root of the graph to the next
position in the TokenStream, incrementGraphToken()
to move along
the current graph, and incrementGraph()
to reset to the next graph
based at the current root.
For example, given the stream 'a b/c:2 d e`, then with the base token at
'a', incrementGraphToken() will produce the stream 'a b d e', and then
after calling incrementGraph() will produce the stream 'a c e'.Modifier and Type | Class and Description |
---|---|
private static class |
GraphTokenFilter.Token |
AttributeSource.State
Modifier and Type | Field and Description |
---|---|
private GraphTokenFilter.Token |
baseToken |
private int |
cacheSize |
private java.util.List<GraphTokenFilter.Token> |
currentGraph |
private int |
finalOffsets |
private int |
graphDepth |
private int |
graphPos |
static int |
MAX_GRAPH_STACK_SIZE
The maximum permitted number of routes through a graph
|
static int |
MAX_TOKEN_CACHE_SIZE
The maximum permitted read-ahead in the token stream
|
private OffsetAttribute |
offsetAtt |
private PositionIncrementAttribute |
posIncAtt |
private int |
stackSize |
private java.util.Deque<GraphTokenFilter.Token> |
tokenPool |
private int |
trailingPositions |
input
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
GraphTokenFilter(TokenStream input)
Create a new GraphTokenFilter
|
Modifier and Type | Method and Description |
---|---|
(package private) int |
cachedTokenCount() |
void |
end()
This method is called by the consumer after the last token has been
consumed, after
TokenStream.incrementToken() returned false
(using the new TokenStream API). |
int |
getTrailingPositions()
Return the number of trailing positions at the end of the graph
NB this should only be called after
incrementGraphToken() has returned false |
protected boolean |
incrementBaseToken()
Move the root of the graph to the next token in the wrapped TokenStream
|
protected boolean |
incrementGraph()
Reset to the root token again, and move down the next route through the graph
|
protected boolean |
incrementGraphToken()
Move to the next token in the current route through the graph
|
private boolean |
lastInStack(GraphTokenFilter.Token token) |
private GraphTokenFilter.Token |
newToken() |
private GraphTokenFilter.Token |
nextTokenInGraph(GraphTokenFilter.Token token) |
private GraphTokenFilter.Token |
nextTokenInStream(GraphTokenFilter.Token token) |
private void |
recycleToken(GraphTokenFilter.Token token) |
void |
reset()
This method is called by a consumer before it begins consumption using
TokenStream.incrementToken() . |
close
incrementToken
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
private final java.util.Deque<GraphTokenFilter.Token> tokenPool
private final java.util.List<GraphTokenFilter.Token> currentGraph
public static final int MAX_GRAPH_STACK_SIZE
public static final int MAX_TOKEN_CACHE_SIZE
private GraphTokenFilter.Token baseToken
private int graphDepth
private int graphPos
private int trailingPositions
private int finalOffsets
private int stackSize
private int cacheSize
private final PositionIncrementAttribute posIncAtt
private final OffsetAttribute offsetAtt
public GraphTokenFilter(TokenStream input)
protected final boolean incrementBaseToken() throws java.io.IOException
false
if the underlying stream is exhaustedjava.io.IOException
protected final boolean incrementGraphToken() throws java.io.IOException
false
if there are not more tokens in the current graphjava.io.IOException
protected final boolean incrementGraph() throws java.io.IOException
java.io.IOException
public int getTrailingPositions()
incrementGraphToken()
has returned false
public void end() throws java.io.IOException
TokenFilter
TokenStream.incrementToken()
returned false
(using the new TokenStream
API). Streams implementing the old API
should upgrade to use this feature.
This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.
Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.
If you override this method, always call super.end()
.
NOTE:
The default implementation chains the call to the input TokenStream, so
be sure to call super.end()
first when overriding this method.
end
in class TokenFilter
java.io.IOException
- If an I/O error occurspublic void reset() throws java.io.IOException
TokenFilter
TokenStream.incrementToken()
.
Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call super.reset()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on further usage).
NOTE:
The default implementation chains the call to the input TokenStream, so
be sure to call super.reset()
when overriding this method.
reset
in class TokenFilter
java.io.IOException
int cachedTokenCount()
private GraphTokenFilter.Token newToken()
private void recycleToken(GraphTokenFilter.Token token)
private GraphTokenFilter.Token nextTokenInGraph(GraphTokenFilter.Token token) throws java.io.IOException
java.io.IOException
private boolean lastInStack(GraphTokenFilter.Token token) throws java.io.IOException
java.io.IOException
private GraphTokenFilter.Token nextTokenInStream(GraphTokenFilter.Token token) throws java.io.IOException
java.io.IOException