#include <vector>
#include <zorba/config.h>
#include <zorba/locale.h>
#include <zorba/internal/unique_ptr.h>
#include <zorba/internal/ztd.h>
Go to the source code of this file.
Classes | |
class | zorba::Tokenizer::Callback |
A Callback is called once per token. More... | |
struct | zorba::Tokenizer::Properties |
Various properties of this Tokenizer. More... | |
struct | zorba::Tokenizer::State |
A State contains inter-Tokenizer state, currently the current token, sentence, and paragraph numbers. More... | |
class | zorba::Tokenizer |
A Tokenizer breaks a string into a stream of word tokens. More... | |
class | zorba::TokenizerProvider |
A TokenizerProvider provides a Tokenizer for a given language. More... | |
Namespaces | |
zorba | |
Copyright 2006-2011 The FLWOR Foundation. | |
Macros | |
#define | ZORBA_TOKENIZER_API_H |
#define ZORBA_TOKENIZER_API_H |
Definition at line 19 of file tokenizer.h.