Class Ferret::Analysis::AsciiStandardTokenizer
In: ext/r_analysis.c
Parent: Ferret::Analysis::TokenStream

The standard tokenizer is an advanced tokenizer which tokenizes most words correctly as well as tokenizing things like email addresses, web addresses, phone numbers, etc.

Example

  "Dave's résumé, at http://www.davebalmain.com/ 1234"
    => ["Dave's", "r", "sum", "at", "http://www.davebalmain.com", "1234"]

Methods

new  

Public Class methods

[Validate]