Class | Ferret::Analysis::TokenStream |
In: |
ext/r_analysis.c
|
Parent: | Object |
A TokenStream enumerates the sequence of tokens, either from fields of a document or from query text.
This is an abstract class. Concrete subclasses are:
Tokenizer: | a TokenStream whose input is a string |
TokenFilter: | a TokenStream whose input is another TokenStream |
Return the next token from the TokenStream or nil if there are no more tokens.
Set the text attribute of the TokenStream to the text you wish to be tokenized. For example, you may do this;
token_stream.text = File.read(file_name)