public class WordTokenizer extends CharacterDelimitedTokenizer
-delimiters <value> The delimiters to use (default ' \r\n\t.,;:'"()?!').
Constructor and Description |
---|
WordTokenizer() |
Modifier and Type | Method and Description |
---|---|
String |
getRevision()
Returns the revision string.
|
String |
globalInfo()
Returns a string describing the stemmer
|
boolean |
hasMoreElements()
Tests if this enumeration contains more elements.
|
static void |
main(String[] args)
Runs the tokenizer with the given options and strings to tokenize.
|
Object |
nextElement()
Returns the next element of this enumeration if this enumeration object
has at least one more element to provide.
|
void |
tokenize(String s)
Sets the string to tokenize.
|
delimitersTipText, getDelimiters, getOptions, listOptions, setDelimiters, setOptions
runTokenizer, tokenize
public String globalInfo()
globalInfo
in class Tokenizer
public boolean hasMoreElements()
hasMoreElements
in interface Enumeration
hasMoreElements
in class Tokenizer
public Object nextElement()
nextElement
in interface Enumeration
nextElement
in class Tokenizer
public void tokenize(String s)
public String getRevision()
public static void main(String[] args)
args
- the commandline options and strings to tokenizeCopyright © 2019 University of Waikato, Hamilton, NZ. All rights reserved.