public class DefaultTokenContextGenerator extends Object implements TokenContextGenerator
Modifier and Type | Field and Description |
---|---|
protected Set<String> |
inducedAbbreviations |
Constructor and Description |
---|
DefaultTokenContextGenerator()
Creates a default context generator for tokenizer.
|
DefaultTokenContextGenerator(Set<String> inducedAbbreviations)
Creates a default context generator for tokenizer.
|
Modifier and Type | Method and Description |
---|---|
protected void |
addCharPreds(String key,
char c,
List<String> preds)
Helper function for getContext.
|
protected List<String> |
createContext(String sentence,
int index)
Returns an
ArrayList of features for the specified sentence string
at the specified index. |
String[] |
getContext(String sentence,
int index)
Returns an array of features for the specified sentence string at the specified index.
|
public DefaultTokenContextGenerator()
public String[] getContext(String sentence, int index)
TokenContextGenerator
getContext
in interface TokenContextGenerator
sentence
- The string for a sentence.index
- The index to consider splitting as a token.protected List<String> createContext(String sentence, int index)
ArrayList
of features for the specified sentence string
at the specified index. Extensions of this class can override this method
to create a customized TokenContextGenerator
sentence
- the token been analyzedindex
- the index of the character been analyzedArrayList
of features for the specified sentence string
at the specified index.Copyright © 2015 The Apache Software Foundation. All rights reserved.