public final class TokenStreamFromTermVector extends TokenStream
reset()
, so there's
no need to wrap with a caching impl.
The implementation will create an array of tokens indexed by token position. As long as there aren't massive jumps in positions, this is fine. And it assumes there aren't large numbers of tokens at the same position, since it adds them to a linked-list per position in O(N^2) complexity. When there aren't positions in the term vector, it divides the startOffset by 8 to use as a temporary substitute. In that case, tokens with the same startOffset will occupy the same final position; otherwise tokens become adjacent.
Modifier and Type | Class and Description |
---|---|
private static class |
TokenStreamFromTermVector.TokenLL |
AttributeSource.State
Modifier and Type | Field and Description |
---|---|
private TokenStreamFromTermVector.TokenLL |
firstToken |
private TokenStreamFromTermVector.TokenLL |
incrementToken |
private boolean |
initialized |
private int |
maxStartOffset |
private OffsetAttribute |
offsetAttribute |
private PayloadAttribute |
payloadAttribute |
private BytesRefArray |
payloadsBytesRefArray |
private PositionIncrementAttribute |
positionIncrementAttribute |
private BytesRefBuilder |
spareBytesRefBuilder |
private CharTermAttribute |
termAttribute |
private CharsRefBuilder |
termCharsBuilder |
private Terms |
vector |
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
TokenStreamFromTermVector(Terms vector,
int maxStartOffset)
Constructor.
|
Modifier and Type | Method and Description |
---|---|
Terms |
getTermVectorTerms() |
boolean |
incrementToken()
Consumers (i.e.,
IndexWriter ) use this method to advance the stream to
the next token. |
private void |
init() |
private TokenStreamFromTermVector.TokenLL[] |
initTokensArray() |
void |
reset()
This method is called by a consumer before it begins consumption using
TokenStream.incrementToken() . |
close, end
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
private final Terms vector
private final CharTermAttribute termAttribute
private final PositionIncrementAttribute positionIncrementAttribute
private final int maxStartOffset
private OffsetAttribute offsetAttribute
private PayloadAttribute payloadAttribute
private CharsRefBuilder termCharsBuilder
private BytesRefArray payloadsBytesRefArray
private BytesRefBuilder spareBytesRefBuilder
private TokenStreamFromTermVector.TokenLL firstToken
private TokenStreamFromTermVector.TokenLL incrementToken
private boolean initialized
public TokenStreamFromTermVector(Terms vector, int maxStartOffset) throws java.io.IOException
incrementToken
.vector
- Terms that contains the data for
creating the TokenStream. Must have positions and/or offsets.maxStartOffset
- if a token's start offset exceeds this then the token is not added. -1 disables the limit.java.io.IOException
public Terms getTermVectorTerms()
public void reset() throws java.io.IOException
TokenStream
TokenStream.incrementToken()
.
Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call super.reset()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on further usage).
reset
in class TokenStream
java.io.IOException
private void init() throws java.io.IOException
java.io.IOException
private TokenStreamFromTermVector.TokenLL[] initTokensArray() throws java.io.IOException
java.io.IOException
public boolean incrementToken() throws java.io.IOException
TokenStream
IndexWriter
) use this method to advance the stream to
the next token. Implementing classes must implement this method and update
the appropriate AttributeImpl
s with the attributes of the next
token.
The producer must make no assumptions about the attributes after the method
has been returned: the caller may arbitrarily change it. If the producer
needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.
This method is called for every token of a document, so an efficient
implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
and AttributeSource.getAttribute(Class)
,
references to all AttributeImpl
s that this stream uses should be
retrieved during instantiation.
To ensure that filters and consumers know which attributes are available,
the attributes must be added during instantiation. Filters and consumers
are not required to check for availability of attributes in
TokenStream.incrementToken()
.
incrementToken
in class TokenStream
java.io.IOException