public class SegTokenFilter
extends java.lang.Object
Filters a SegToken
by converting full-width latin to half-width, then lowercasing latin.
Additionally, all punctuation is converted into Utility.COMMON_DELIMITER
Constructor and Description |
---|
SegTokenFilter() |
Modifier and Type | Method and Description |
---|---|
SegToken |
filter(SegToken token)
Filter an input
SegToken |