A TokenizerProvider provides a Tokenizer for a given language. More...
#include <zorba/tokenizer.h>
Public Member Functions | |
virtual Tokenizer::ptr | getTokenizer (locale::iso639_1::type lang, Tokenizer::Numbers &numbers) const =0 |
Creates a new Tokenizer. | |
virtual | ~TokenizerProvider () |
A TokenizerProvider provides a Tokenizer for a given language.
Definition at line 206 of file tokenizer.h.
virtual zorba::TokenizerProvider::~TokenizerProvider | ( | ) | [virtual] |
virtual Tokenizer::ptr zorba::TokenizerProvider::getTokenizer | ( | locale::iso639_1::type | lang, |
Tokenizer::Numbers & | numbers | ||
) | const [pure virtual] |
Creates a new Tokenizer.
lang | The language of the text that the tokenizer will tokenize. |
numbers | The Numbers to use. |