weka.core.tokenizers
Class AlphabeticTokenizer

java.lang.Object
  extended by weka.core.tokenizers.Tokenizer
      extended by weka.core.tokenizers.AlphabeticTokenizer
All Implemented Interfaces:
java.io.Serializable, java.util.Enumeration, OptionHandler, RevisionHandler

public class AlphabeticTokenizer
extends Tokenizer

Alphabetic string tokenizer, tokens are to be formed only from contiguous alphabetic sequences.

Version:
$Revision: 1.2 $
Author:
Asrhaf M. Kibriya (amk14@cs.waikato.ac.nz), FracPete (fracpete at waikato dot ac dot nz)
See Also:
Serialized Form

Constructor Summary
AlphabeticTokenizer()
           
 
Method Summary
 java.lang.String getRevision()
          Returns the revision string.
 java.lang.String globalInfo()
          Returns a string describing the stemmer
 boolean hasMoreElements()
          returns whether there are more elements still
static void main(java.lang.String[] args)
          Runs the tokenizer with the given options and strings to tokenize.
 java.lang.Object nextElement()
          returns the next element
 void tokenize(java.lang.String s)
          Sets the string to tokenize.
 
Methods inherited from class weka.core.tokenizers.Tokenizer
getOptions, listOptions, runTokenizer, setOptions, tokenize
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

AlphabeticTokenizer

public AlphabeticTokenizer()
Method Detail

globalInfo

public java.lang.String globalInfo()
Returns a string describing the stemmer

Specified by:
globalInfo in class Tokenizer
Returns:
a description suitable for displaying in the explorer/experimenter gui

hasMoreElements

public boolean hasMoreElements()
returns whether there are more elements still

Specified by:
hasMoreElements in interface java.util.Enumeration
Specified by:
hasMoreElements in class Tokenizer
Returns:
true if there are still more elements

nextElement

public java.lang.Object nextElement()
returns the next element

Specified by:
nextElement in interface java.util.Enumeration
Specified by:
nextElement in class Tokenizer
Returns:
the next element

tokenize

public void tokenize(java.lang.String s)
Sets the string to tokenize. Tokenization happens immediately.

Specified by:
tokenize in class Tokenizer
Parameters:
s - the string to tokenize

getRevision

public java.lang.String getRevision()
Returns the revision string.

Returns:
the revision

main

public static void main(java.lang.String[] args)
Runs the tokenizer with the given options and strings to tokenize. The tokens are printed to stdout.

Parameters:
args - the commandline options and strings to tokenize