class Ferret::Analysis::AsciiWhiteSpaceTokenizer
Summary¶ ↑
A WhiteSpaceTokenizer is a tokenizer that divides text at white-space. Adjacent sequences of non-WhiteSpace characters form tokens.
Example¶ ↑
"Dave's résumé, at http://www.davebalmain.com/ 1234" => ["Dave's", "résumé,", "at", "http://www.davebalmain.com", "1234"]
Public Class Methods
new() → tokenizer
click to toggle source
Create a new AsciiWhiteSpaceTokenizer
static VALUE frb_a_whitespace_tokenizer_init(VALUE self, VALUE rstr) { return get_wrapped_ts(self, rstr, whitespace_tokenizer_new()); }