com.ibm.icu.text

Class Normalizer

public final class Normalizer extends Object implements Cloneable

Unicode Normalization

Unicode normalization API

normalize transforms Unicode text into an equivalent composed or decomposed form, allowing for easier sorting and searching of text. normalize supports the standard normalization forms described in Unicode Standard Annex #15 — Unicode Normalization Forms. Characters with accents or other adornments can be encoded in several different ways in Unicode. For example, take the character A-acute. In Unicode, this can be encoded as a single character (the "composed" form):

00C1 LATIN CAPITAL LETTER A WITH ACUTE

or as two separate characters (the "decomposed" form):

0041 LATIN CAPITAL LETTER A 0301 COMBINING ACUTE ACCENT

To a user of your program, however, both of these sequences should be treated as the same "user-level" character "A with acute accent". When you are searching or comparing text, you must ensure that these two sequences are treated equivalently. In addition, you must handle characters with more than one accent. Sometimes the order of a character's combining accents is significant, while in other cases accent sequences in different orders are really equivalent. Similarly, the string "ffi" can be encoded as three separate letters:

0066 LATIN SMALL LETTER F 0066 LATIN SMALL LETTER F 0069 LATIN SMALL LETTER I <\p> or as the single character

FB03 LATIN SMALL LIGATURE FFI <\p> The ffi ligature is not a distinct semantic character, and strictly speaking it shouldn't be in Unicode at all, but it was included for compatibility with existing character sets that already provided it. The Unicode standard identifies such characters by giving them "compatibility" decompositions into the corresponding semantic characters. When sorting and searching, you will often want to use these mappings. normalize helps solve these problems by transforming text into the canonical composed and decomposed forms as shown in the first example above. In addition, you can have it perform compatibility decompositions so that you can treat compatibility characters the same as their equivalents. Finally, normalize rearranges accents into the proper canonical order, so that you do not have to worry about accent rearrangement on your own. Form FCD, "Fast C or D", is also designed for collation. It allows to work on strings that are not necessarily normalized with an algorithm (like in collation) that works under "canonical closure", i.e., it treats precomposed characters and their decomposed equivalents the same. It is not a normalization form because it does not provide for uniqueness of representation. Multiple strings may be canonically equivalent (their NFDs are identical) and may all conform to FCD without being identical themselves. The form is defined such that the "raw decomposition", the recursive canonical decomposition of each character, results in a string that is canonically ordered. This means that precomposed characters are allowed for as long as their decompositions do not need canonical reordering. Its advantage for a process like collation is that all NFD and most NFC texts - and many unnormalized texts - already conform to FCD and do not need to be normalized (NFD) for such a process. The FCD quick check will return YES for most strings in practice. normalize(FCD) may be implemented with NFD. For more details on FCD see the collation design document: http://dev.icu-project.org/cgi-bin/viewcvs.cgi/~checkout~/icuhtml/design/collation/ICU_collation_design.htm ICU collation performs either NFD or FCD normalization automatically if normalization is turned on for the collator object. Beyond collation and string search, normalized strings may be useful for string equivalence comparisons, transliteration/transcription, unique representations, etc. The W3C generally recommends to exchange texts in NFC. Note also that most legacy character encodings use only precomposed forms and often do not encode any combining marks by themselves. For conversion to such character encodings the Unicode text needs to be normalized to NFC. For more usage examples, see the Unicode Standard Annex.

UNKNOWN: ICU 2.8

Nested Class Summary
static classNormalizer.Mode
Constants for normalization modes.
static classNormalizer.QuickCheckResult
Result values for quickCheck().
Field Summary
static intCOMPARE_CODE_POINT_ORDER
Option bit for compare: Compare strings in code point order instead of code unit order.
static intCOMPARE_IGNORE_CASE
Option bit for compare: Perform case-insensitive comparison.
static intCOMPARE_NORM_OPTIONS_SHIFT
Lowest-order bit number of compare() options bits corresponding to normalization options bits.
static Normalizer.ModeCOMPOSE
Canonical decomposition followed by canonical composition.
static Normalizer.ModeCOMPOSE_COMPAT
Compatibility decomposition followed by canonical composition.
static Normalizer.ModeDECOMP
Canonical decomposition.
static Normalizer.ModeDECOMP_COMPAT
Compatibility decomposition.
static Normalizer.ModeDEFAULT
Default normalization.
static intDONE
Constant indicating that the end of the iteration has been reached.
static Normalizer.ModeFCD
"Fast C or D" form.
static intFOLD_CASE_DEFAULT
Option bit for compare: Case sensitively compare the strings
static intFOLD_CASE_EXCLUDE_SPECIAL_I
Option value for case folding: exclude the mappings for dotted I and dotless i marked with 'I' in CaseFolding.txt.
static intIGNORE_HANGUL
Option to disable Hangul/Jamo composition and decomposition.
static intINPUT_IS_FCD
Option bit for compare: Both input strings are assumed to fulfill FCD conditions.
static Normalizer.QuickCheckResultMAYBE
Indicates it cannot be determined if string is in the normalized format without further thorough checks.
static Normalizer.ModeNFC
Canonical decomposition followed by canonical composition.
static Normalizer.ModeNFD
Canonical decomposition.
static Normalizer.ModeNFKC
Compatibility decomposition followed by canonical composition.
static Normalizer.ModeNFKD
Compatibility decomposition.
static Normalizer.QuickCheckResultNO
Indicates that string is not in the normalized format
static Normalizer.ModeNONE
No decomposition/composition.
static Normalizer.ModeNO_OP
Null operation for use with the constructors and the static normalize method.
static intUNICODE_3_2
Options bit set value to select Unicode 3.2 normalization (except NormalizationCorrections).
static Normalizer.QuickCheckResultYES
Indicates that string is in the normalized format
Constructor Summary
Normalizer(String str, Normalizer.Mode mode, int opt)
Creates a new Normalizer object for iterating over the normalized form of a given string.
Normalizer(CharacterIterator iter, Normalizer.Mode mode, int opt)
Creates a new Normalizer object for iterating over the normalized form of the given text.
Normalizer(UCharacterIterator iter, Normalizer.Mode mode, int options)
Creates a new Normalizer object for iterating over the normalized form of the given text.
Method Summary
Objectclone()
Clones this Normalizer object.
static intcompare(char[] s1, int s1Start, int s1Limit, char[] s2, int s2Start, int s2Limit, int options)
Compare two strings for canonical equivalence.
static intcompare(String s1, String s2, int options)
Compare two strings for canonical equivalence.
static intcompare(char[] s1, char[] s2, int options)
Compare two strings for canonical equivalence.
static intcompare(int char32a, int char32b, int options)
Convenience method that can have faster implementation by not allocating buffers.
static intcompare(int char32a, String str2, int options)
Convenience method that can have faster implementation by not allocating buffers.
static Stringcompose(String str, boolean compat)
Compose a string.
static Stringcompose(String str, boolean compat, int options)
Compose a string.
static intcompose(char[] source, char[] target, boolean compat, int options)
Compose a string.
static intcompose(char[] src, int srcStart, int srcLimit, char[] dest, int destStart, int destLimit, boolean compat, int options)
Compose a string.
static intconcatenate(char[] left, int leftStart, int leftLimit, char[] right, int rightStart, int rightLimit, char[] dest, int destStart, int destLimit, Normalizer.Mode mode, int options)
Concatenate normalized strings, making sure that the result is normalized as well.
static Stringconcatenate(char[] left, char[] right, Normalizer.Mode mode, int options)
Concatenate normalized strings, making sure that the result is normalized as well.
static Stringconcatenate(String left, String right, Normalizer.Mode mode, int options)
Concatenate normalized strings, making sure that the result is normalized as well.
intcurrent()
Return the current character in the normalized text->
static Stringdecompose(String str, boolean compat)
Decompose a string.
static Stringdecompose(String str, boolean compat, int options)
Decompose a string.
static intdecompose(char[] source, char[] target, boolean compat, int options)
Decompose a string.
static intdecompose(char[] src, int srcStart, int srcLimit, char[] dest, int destStart, int destLimit, boolean compat, int options)
Decompose a string.
intendIndex()
Retrieve the index of the end of the input text-> This is the end index of the CharacterIterator or the length of the String over which this Normalizer is iterating
intfirst()
Return the first character in the normalized text-> This resets the Normalizer's position to the beginning of the text->
intgetBeginIndex()
Retrieve the index of the start of the input text.
intgetEndIndex()
Retrieve the index of the end of the input text.
static intgetFC_NFKC_Closure(int c, char[] dest)
Gets the FC_NFKC closure set from the normalization data
static StringgetFC_NFKC_Closure(int c)
Gets the FC_NFKC closure set from the normalization data
intgetIndex()
Retrieve the current iteration position in the input text that is being normalized.
intgetLength()
Gets the length of underlying text storage
Normalizer.ModegetMode()
Return the basic operation performed by this Normalizer
intgetOption(int option)
Determine whether an option is turned on or off.
intgetText(char[] fillIn)
Gets the underlying text storage
StringgetText()
Returns the text under iteration as a string
static booleanisNFSkippable(int c, Normalizer.Mode mode)
Internal API
static booleanisNormalized(char[] src, int start, int limit, Normalizer.Mode mode, int options)
Test if a string is in a given normalization form.
static booleanisNormalized(String str, Normalizer.Mode mode, int options)
Convenience Method
static booleanisNormalized(int char32, Normalizer.Mode mode, int options)
Convenience Method
intlast()
Return the last character in the normalized text-> This resets the Normalizer's position to be just before the the input text corresponding to that normalized character.
intnext()
Return the next character in the normalized text and advance the iteration position by one.
static Stringnormalize(String str, Normalizer.Mode mode, int options)
Normalizes a String using the given normalization operation.
static Stringnormalize(String src, Normalizer.Mode mode)
Normalize a string.
static intnormalize(char[] source, char[] target, Normalizer.Mode mode, int options)
Normalize a string.
static intnormalize(char[] src, int srcStart, int srcLimit, char[] dest, int destStart, int destLimit, Normalizer.Mode mode, int options)
Normalize a string.
static Stringnormalize(int char32, Normalizer.Mode mode, int options)
Normalize a codepoint accoding to the given mode
static Stringnormalize(int char32, Normalizer.Mode mode)
Conveinience method to normalize a codepoint accoding to the given mode
intprevious()
Return the previous character in the normalized text and decrement the iteration position by one.
static Normalizer.QuickCheckResultquickCheck(String source, Normalizer.Mode mode)
Convenience method.
static Normalizer.QuickCheckResultquickCheck(String source, Normalizer.Mode mode, int options)
Convenience method.
static Normalizer.QuickCheckResultquickCheck(char[] source, Normalizer.Mode mode, int options)
Convenience method.
static Normalizer.QuickCheckResultquickCheck(char[] source, int start, int limit, Normalizer.Mode mode, int options)
Performing quick check on a string, to quickly determine if the string is in a particular normalization format.
voidreset()
Reset the index to the beginning of the text.
intsetIndex(int index)
Set the iteration position in the input text that is being normalized and return the first normalized character at that position.
voidsetIndexOnly(int index)
Set the iteration position in the input text that is being normalized, without any immediate normalization.
voidsetMode(Normalizer.Mode newMode)
Set the normalization mode for this object.
voidsetOption(int option, boolean value)
Set options that affect this Normalizer's operation.
voidsetText(StringBuffer newText)
Set the input text over which this Normalizer will iterate.
voidsetText(char[] newText)
Set the input text over which this Normalizer will iterate.
voidsetText(String newText)
Set the input text over which this Normalizer will iterate.
voidsetText(CharacterIterator newText)
Set the input text over which this Normalizer will iterate.
voidsetText(UCharacterIterator newText)
Set the input text over which this Normalizer will iterate.
intstartIndex()
Retrieve the index of the start of the input text.

Field Detail

COMPARE_CODE_POINT_ORDER

public static final int COMPARE_CODE_POINT_ORDER
Option bit for compare: Compare strings in code point order instead of code unit order.

UNKNOWN: ICU 2.8

COMPARE_IGNORE_CASE

public static final int COMPARE_IGNORE_CASE
Option bit for compare: Perform case-insensitive comparison.

UNKNOWN: ICU 2.8

COMPARE_NORM_OPTIONS_SHIFT

public static final int COMPARE_NORM_OPTIONS_SHIFT
Lowest-order bit number of compare() options bits corresponding to normalization options bits. The options parameter for compare() uses most bits for itself and for various comparison and folding flags. The most significant bits, however, are shifted down and passed on to the normalization implementation. (That is, from compare(..., options, ...), options>>COMPARE_NORM_OPTIONS_SHIFT will be passed on to the internal normalization functions.)

See Also: Normalizer

UNKNOWN: ICU 2.6

COMPOSE

public static final Normalizer.Mode COMPOSE

Deprecated: ICU 2.8. Use Normalier.NFC

Canonical decomposition followed by canonical composition. Used with the constructors and the static normalize method to determine the operation to be performed.

If all optional features (e.g. IGNORE_HANGUL) are turned off, this operation produces output that is in Unicode Canonical Form C.

See Also: Normalizer NFC

COMPOSE_COMPAT

public static final Normalizer.Mode COMPOSE_COMPAT

Deprecated: ICU 2.8. Use Normalizer.NFKC

Compatibility decomposition followed by canonical composition. Used with the constructors and the static normalize method to determine the operation to be performed.

If all optional features (e.g. IGNORE_HANGUL) are turned off, this operation produces output that is in Unicode Canonical Form KC.

See Also: Normalizer NFKC

DECOMP

public static final Normalizer.Mode DECOMP

Deprecated: ICU 2.8. Use Normalizer.NFD

Canonical decomposition. This value is passed to the constructors and the static normalize method to determine the operation to be performed.

If all optional features (e.g. IGNORE_HANGUL) are turned off, this operation produces output that is in Unicode Canonical Form D.

See Also: Normalizer NFD

DECOMP_COMPAT

public static final Normalizer.Mode DECOMP_COMPAT

Deprecated: ICU 2.8. Use Normalizer.NFKD

Compatibility decomposition. This value is passed to the constructors and the static normalize method to determine the operation to be performed.

If all optional features (e.g. IGNORE_HANGUL) are turned off, this operation produces output that is in Unicode Canonical Form KD.

See Also: Normalizer NFKD

DEFAULT

public static final Normalizer.Mode DEFAULT
Default normalization.

UNKNOWN: ICU 2.8

DONE

public static final int DONE
Constant indicating that the end of the iteration has been reached. This is guaranteed to have the same value as UCharacterIterator.

UNKNOWN: ICU 2.8

FCD

public static final Normalizer.Mode FCD
"Fast C or D" form.

UNKNOWN: ICU 2.8

FOLD_CASE_DEFAULT

public static final int FOLD_CASE_DEFAULT
Option bit for compare: Case sensitively compare the strings

UNKNOWN: ICU 2.8

FOLD_CASE_EXCLUDE_SPECIAL_I

public static final int FOLD_CASE_EXCLUDE_SPECIAL_I
Option value for case folding: exclude the mappings for dotted I and dotless i marked with 'I' in CaseFolding.txt.

UNKNOWN: ICU 2.8

IGNORE_HANGUL

public static final int IGNORE_HANGUL

Deprecated: ICU 2.8. This option is no longer supported. TODO: check with Ram

Option to disable Hangul/Jamo composition and decomposition. This option applies to Korean text, which can be represented either in the Jamo alphabet or in Hangul characters, which are really just two or three Jamo combined into one visual glyph. Since Jamo takes up more storage space than Hangul, applications that process only Hangul text may wish to turn this option on when decomposing text.

The Unicode standard treates Hangul to Jamo conversion as a canonical decomposition, so this option must be turned off if you wish to transform strings into one of the standard Unicode Normalization Forms.

See Also: Normalizer

INPUT_IS_FCD

public static final int INPUT_IS_FCD
Option bit for compare: Both input strings are assumed to fulfill FCD conditions.

UNKNOWN: ICU 2.8

MAYBE

public static final Normalizer.QuickCheckResult MAYBE
Indicates it cannot be determined if string is in the normalized format without further thorough checks.

UNKNOWN: ICU 2.8

NFC

public static final Normalizer.Mode NFC
Canonical decomposition followed by canonical composition.

UNKNOWN: ICU 2.8

NFD

public static final Normalizer.Mode NFD
Canonical decomposition.

UNKNOWN: ICU 2.8

NFKC

public static final Normalizer.Mode NFKC
Compatibility decomposition followed by canonical composition.

UNKNOWN: ICU 2.8

NFKD

public static final Normalizer.Mode NFKD
Compatibility decomposition.

UNKNOWN: ICU 2.8

NO

public static final Normalizer.QuickCheckResult NO
Indicates that string is not in the normalized format

UNKNOWN: ICU 2.8

NONE

public static final Normalizer.Mode NONE
No decomposition/composition.

UNKNOWN: ICU 2.8

NO_OP

public static final Normalizer.Mode NO_OP

Deprecated: ICU 2.8. Use Nomalizer.NONE

Null operation for use with the constructors and the static normalize method. This value tells the Normalizer to do nothing but return unprocessed characters from the underlying String or CharacterIterator. If you have code which requires raw text at some times and normalized text at others, you can use NO_OP for the cases where you want raw text, rather than having a separate code path that bypasses Normalizer altogether.

See Also: Normalizer NONE

UNICODE_3_2

public static final int UNICODE_3_2
Options bit set value to select Unicode 3.2 normalization (except NormalizationCorrections). At most one Unicode version can be selected at a time.

UNKNOWN: ICU 2.6

YES

public static final Normalizer.QuickCheckResult YES
Indicates that string is in the normalized format

UNKNOWN: ICU 2.8

Constructor Detail

Normalizer

public Normalizer(String str, Normalizer.Mode mode, int opt)
Creates a new Normalizer object for iterating over the normalized form of a given string.

The options parameter specifies which optional Normalizer features are to be enabled for this object.

Parameters: str The string to be normalized. The normalization will start at the beginning of the string. mode The normalization mode. opt Any optional features to be enabled. Currently the only available option is UNICODE_3_2. If you want the default behavior corresponding to one of the standard Unicode Normalization Forms, use 0 for this argument.

UNKNOWN: ICU 2.6

Normalizer

public Normalizer(CharacterIterator iter, Normalizer.Mode mode, int opt)
Creates a new Normalizer object for iterating over the normalized form of the given text.

Parameters: iter The input text to be normalized. The normalization will start at the beginning of the string. mode The normalization mode. opt Any optional features to be enabled. Currently the only available option is UNICODE_3_2. If you want the default behavior corresponding to one of the standard Unicode Normalization Forms, use 0 for this argument.

UNKNOWN: ICU 2.6

Normalizer

public Normalizer(UCharacterIterator iter, Normalizer.Mode mode, int options)
Creates a new Normalizer object for iterating over the normalized form of the given text.

Parameters: iter The input text to be normalized. The normalization will start at the beginning of the string. mode The normalization mode. options The normalization options, ORed together (0 for no options).

UNKNOWN: ICU 2.6

Method Detail

clone

public Object clone()
Clones this Normalizer object. All properties of this object are duplicated in the new object, including the cloning of any CharacterIterator that was passed in to the constructor or to setText. However, the text storage underlying the CharacterIterator is not duplicated unless the iterator's clone method does so.

UNKNOWN: ICU 2.8

compare

public static int compare(char[] s1, int s1Start, int s1Limit, char[] s2, int s2Start, int s2Limit, int options)
Compare two strings for canonical equivalence. Further options include case-insensitive comparison and code point order (as opposed to code unit order). Canonical equivalence between two strings is defined as their normalized forms (NFD or NFC) being identical. This function compares strings incrementally instead of normalizing (and optionally case-folding) both strings entirely, improving performance significantly. Bulk normalization is only necessary if the strings do not fulfill the FCD conditions. Only in this case, and only if the strings are relatively long, is memory allocated temporarily. For FCD strings and short non-FCD strings there is no memory allocation. Semantically, this is equivalent to strcmp[CodePointOrder](foldCase(NFD(s1)), foldCase(NFD(s2))) where code point order and foldCase are all optional.

Parameters: s1 First source character array. s1Start start index of source s1Limit limit of the source s2 Second source character array. s2Start start index of the source s2Limit limit of the source options A bit set of options: - FOLD_CASE_DEFAULT or 0 is used for default options: Case-sensitive comparison in code unit order, and the input strings are quick-checked for FCD. - INPUT_IS_FCD Set if the caller knows that both s1 and s2 fulfill the FCD conditions.If not set, the function will quickCheck for FCD and normalize if necessary. - COMPARE_CODE_POINT_ORDER Set to choose code point order instead of code unit order - COMPARE_IGNORE_CASE Set to compare strings case-insensitively using case folding, instead of case-sensitively. If set, then the following case folding options are used.

Returns: <0 or 0 or >0 as usual for string comparisons

See Also: Normalizer FCD

UNKNOWN: ICU 2.8

compare

public static int compare(String s1, String s2, int options)
Compare two strings for canonical equivalence. Further options include case-insensitive comparison and code point order (as opposed to code unit order). Convenience method.

Parameters: s1 First source string. s2 Second source string. options A bit set of options: - FOLD_CASE_DEFAULT or 0 is used for default options: Case-sensitive comparison in code unit order, and the input strings are quick-checked for FCD. - INPUT_IS_FCD Set if the caller knows that both s1 and s2 fulfill the FCD conditions. If not set, the function will quickCheck for FCD and normalize if necessary. - COMPARE_CODE_POINT_ORDER Set to choose code point order instead of code unit order - COMPARE_IGNORE_CASE Set to compare strings case-insensitively using case folding, instead of case-sensitively. If set, then the following case folding options are used.

Returns: <0 or 0 or >0 as usual for string comparisons

See Also: Normalizer FCD

UNKNOWN: ICU 2.8

compare

public static int compare(char[] s1, char[] s2, int options)
Compare two strings for canonical equivalence. Further options include case-insensitive comparison and code point order (as opposed to code unit order). Convenience method.

Parameters: s1 First source string. s2 Second source string. options A bit set of options: - FOLD_CASE_DEFAULT or 0 is used for default options: Case-sensitive comparison in code unit order, and the input strings are quick-checked for FCD. - INPUT_IS_FCD Set if the caller knows that both s1 and s2 fulfill the FCD conditions. If not set, the function will quickCheck for FCD and normalize if necessary. - COMPARE_CODE_POINT_ORDER Set to choose code point order instead of code unit order - COMPARE_IGNORE_CASE Set to compare strings case-insensitively using case folding, instead of case-sensitively. If set, then the following case folding options are used.

Returns: <0 or 0 or >0 as usual for string comparisons

See Also: Normalizer FCD

UNKNOWN: ICU 2.8

compare

public static int compare(int char32a, int char32b, int options)
Convenience method that can have faster implementation by not allocating buffers.

Parameters: char32a the first code point to be checked against the char32b the second code point options A bit set of options

UNKNOWN: ICU 2.8

compare

public static int compare(int char32a, String str2, int options)
Convenience method that can have faster implementation by not allocating buffers.

Parameters: char32a the first code point to be checked against str2 the second string options A bit set of options

UNKNOWN: ICU 2.8

compose

public static String compose(String str, boolean compat)
Compose a string. The string will be composed to according the the specified mode.

Parameters: str The string to compose. compat If true the string will be composed accoding to NFKC rules and if false will be composed according to NFC rules.

Returns: String The composed string

UNKNOWN: ICU 2.8

compose

public static String compose(String str, boolean compat, int options)
Compose a string. The string will be composed to according the the specified mode.

Parameters: str The string to compose. compat If true the string will be composed accoding to NFKC rules and if false will be composed according to NFC rules. options The only recognized option is UNICODE_3_2

Returns: String The composed string

UNKNOWN: ICU 2.6

compose

public static int compose(char[] source, char[] target, boolean compat, int options)
Compose a string. The string will be composed to according the the specified mode.

Parameters: source The char array to compose. target A char buffer to receive the normalized text. compat If true the char array will be composed accoding to NFKC rules and if false will be composed according to NFC rules. options The normalization options, ORed together (0 for no options).

Returns: int The total buffer size needed;if greater than length of result, the output was truncated.

Throws: IndexOutOfBoundsException if target.length is less than the required length

UNKNOWN: ICU 2.6

compose

public static int compose(char[] src, int srcStart, int srcLimit, char[] dest, int destStart, int destLimit, boolean compat, int options)
Compose a string. The string will be composed to according the the specified mode.

Parameters: src The char array to compose. srcStart Start index of the source srcLimit Limit index of the source dest The char buffer to fill in destStart Start index of the destination buffer destLimit End index of the destination buffer compat If true the char array will be composed accoding to NFKC rules and if false will be composed according to NFC rules. options The normalization options, ORed together (0 for no options).

Returns: int The total buffer size needed;if greater than length of result, the output was truncated.

Throws: IndexOutOfBoundsException if target.length is less than the required length

UNKNOWN: ICU 2.6

concatenate

public static int concatenate(char[] left, int leftStart, int leftLimit, char[] right, int rightStart, int rightLimit, char[] dest, int destStart, int destLimit, Normalizer.Mode mode, int options)
Concatenate normalized strings, making sure that the result is normalized as well. If both the left and the right strings are in the normalization form according to "mode", then the result will be dest=normalize(left+right, mode) With the input strings already being normalized, this function will use next() and previous() to find the adjacent end pieces of the input strings. Only the concatenation of these end pieces will be normalized and then concatenated with the remaining parts of the input strings. It is allowed to have dest==left to avoid copying the entire left string.

Parameters: left Left source array, may be same as dest. leftStart start in the left array. leftLimit limit in the left array (==length) right Right source array. rightStart start in the right array. rightLimit limit in the right array (==length) dest The output buffer; can be null if destStart==destLimit==0 for pure preflighting. destStart start in the destination array destLimit limit in the destination array (==length) mode The normalization mode. options The normalization options, ORed together (0 for no options).

Returns: Length of output (number of chars) when successful or IndexOutOfBoundsException

Throws: IndexOutOfBoundsException whose message has the string representation of destination capacity required. IndexOutOfBoundsException if target capacity is less than the required length

See Also: Normalizer Normalizer Normalizer

UNKNOWN: ICU 2.8

concatenate

public static String concatenate(char[] left, char[] right, Normalizer.Mode mode, int options)
Concatenate normalized strings, making sure that the result is normalized as well. If both the left and the right strings are in the normalization form according to "mode", then the result will be dest=normalize(left+right, mode) For details see concatenate

Parameters: left Left source string. right Right source string. mode The normalization mode. options The normalization options, ORed together (0 for no options).

Returns: result

See Also: Normalizer Normalizer Normalizer Normalizer Normalizer

UNKNOWN: ICU 2.8

concatenate

public static String concatenate(String left, String right, Normalizer.Mode mode, int options)
Concatenate normalized strings, making sure that the result is normalized as well. If both the left and the right strings are in the normalization form according to "mode", then the result will be dest=normalize(left+right, mode) For details see concatenate

Parameters: left Left source string. right Right source string. mode The normalization mode. options The normalization options, ORed together (0 for no options).

Returns: result

See Also: Normalizer Normalizer Normalizer Normalizer Normalizer

UNKNOWN: ICU 2.8

current

public int current()
Return the current character in the normalized text->

Returns: The codepoint as an int

UNKNOWN: ICU 2.8

decompose

public static String decompose(String str, boolean compat)
Decompose a string. The string will be decomposed to according the the specified mode.

Parameters: str The string to decompose. compat If true the string will be decomposed accoding to NFKD rules and if false will be decomposed according to NFD rules.

Returns: String The decomposed string

UNKNOWN: ICU 2.8

decompose

public static String decompose(String str, boolean compat, int options)
Decompose a string. The string will be decomposed to according the the specified mode.

Parameters: str The string to decompose. compat If true the string will be decomposed accoding to NFKD rules and if false will be decomposed according to NFD rules. options The normalization options, ORed together (0 for no options).

Returns: String The decomposed string

UNKNOWN: ICU 2.6

decompose

public static int decompose(char[] source, char[] target, boolean compat, int options)
Decompose a string. The string will be decomposed to according the the specified mode.

Parameters: source The char array to decompose. target A char buffer to receive the normalized text. compat If true the char array will be decomposed accoding to NFKD rules and if false will be decomposed according to NFD rules. options The normalization options, ORed together (0 for no options).

Returns: int The total buffer size needed;if greater than length of result,the output was truncated.

Throws: IndexOutOfBoundsException if the target capacity is less than the required length

UNKNOWN: ICU 2.6

decompose

public static int decompose(char[] src, int srcStart, int srcLimit, char[] dest, int destStart, int destLimit, boolean compat, int options)
Decompose a string. The string will be decomposed to according the the specified mode.

Parameters: src The char array to compose. srcStart Start index of the source srcLimit Limit index of the source dest The char buffer to fill in destStart Start index of the destination buffer destLimit End index of the destination buffer compat If true the char array will be decomposed accoding to NFKD rules and if false will be decomposed according to NFD rules. options The normalization options, ORed together (0 for no options).

Returns: int The total buffer size needed;if greater than length of result,the output was truncated.

Throws: IndexOutOfBoundsException if the target capacity is less than the required length

UNKNOWN: ICU 2.6

endIndex

public int endIndex()
Retrieve the index of the end of the input text-> This is the end index of the CharacterIterator or the length of the String over which this Normalizer is iterating

Returns: The current iteration position

UNKNOWN: ICU 2.8

first

public int first()
Return the first character in the normalized text-> This resets the Normalizer's position to the beginning of the text->

Returns: The codepoint as an int

UNKNOWN: ICU 2.8

getBeginIndex

public int getBeginIndex()

Deprecated: ICU 2.2. Use startIndex() instead.

Retrieve the index of the start of the input text. This is the begin index of the CharacterIterator or the start (i.e. 0) of the String over which this Normalizer is iterating

Returns: The codepoint as an int

See Also: Normalizer

getEndIndex

public int getEndIndex()

Deprecated: ICU 2.2. Use endIndex() instead.

Retrieve the index of the end of the input text. This is the end index of the CharacterIterator or the length of the String over which this Normalizer is iterating

Returns: The codepoint as an int

See Also: Normalizer

getFC_NFKC_Closure

public static int getFC_NFKC_Closure(int c, char[] dest)

Deprecated: This API is ICU internal only.

Gets the FC_NFKC closure set from the normalization data

Parameters: c The code point whose closure set is to be retrieved dest The char array to recive the closure set

UNKNOWN:

getFC_NFKC_Closure

public static String getFC_NFKC_Closure(int c)

Deprecated: This API is ICU internal only.

Gets the FC_NFKC closure set from the normalization data

Parameters: c The the code point whose closure set is to be retrieved

Returns: String representation of the closure set

UNKNOWN:

getIndex

public int getIndex()
Retrieve the current iteration position in the input text that is being normalized. This method is useful in applications such as searching, where you need to be able to determine the position in the input text that corresponds to a given normalized output character.

Note: This method sets the position in the input, while Normalizer and Normalizer iterate through characters in the output. This means that there is not necessarily a one-to-one correspondence between characters returned by next and previous and the indices passed to and returned from setIndex and Normalizer.

Returns: The current iteration position

UNKNOWN: ICU 2.8

getLength

public int getLength()
Gets the length of underlying text storage

Returns: the length

UNKNOWN: ICU 2.8

getMode

public Normalizer.Mode getMode()
Return the basic operation performed by this Normalizer

See Also: Normalizer

UNKNOWN: ICU 2.8

getOption

public int getOption(int option)
Determine whether an option is turned on or off.

See Also: Normalizer

UNKNOWN: ICU 2.6

getText

public int getText(char[] fillIn)
Gets the underlying text storage

Parameters: fillIn the char buffer to fill the UTF-16 units. The length of the buffer should be equal to the length of the underlying text storage

Throws: IndexOutOfBoundsException

See Also: Normalizer

UNKNOWN: ICU 2.8

getText

public String getText()
Returns the text under iteration as a string

Returns: a copy of the text under iteration.

UNKNOWN: ICU 2.8

isNFSkippable

public static boolean isNFSkippable(int c, Normalizer.Mode mode)

Deprecated: This API is ICU internal only.

Internal API

UNKNOWN:

isNormalized

public static boolean isNormalized(char[] src, int start, int limit, Normalizer.Mode mode, int options)
Test if a string is in a given normalization form. This is semantically equivalent to source.equals(normalize(source, mode)). Unlike quickCheck(), this function returns a definitive result, never a "maybe". For NFD, NFKD, and FCD, both functions work exactly the same. For NFC and NFKC where quickCheck may return "maybe", this function will perform further tests to arrive at a true/false result.

Parameters: src The input array of characters to be checked to see if it is normalized start The strart index in the source limit The limit index in the source mode the normalization mode options Options for use with exclusion set an tailored Normalization The only option that is currently recognized is UNICODE_3_2

Returns: Boolean value indicating whether the source string is in the "mode" normalization form

UNKNOWN: ICU 2.6

isNormalized

public static boolean isNormalized(String str, Normalizer.Mode mode, int options)
Convenience Method

Parameters: str the input string to be checked to see if it is normalized mode the normalization mode options Options for use with exclusion set an tailored Normalization The only option that is currently recognized is UNICODE_3_2

See Also: Normalizer

UNKNOWN: ICU 2.6

isNormalized

public static boolean isNormalized(int char32, Normalizer.Mode mode, int options)
Convenience Method

Parameters: char32 the input code point to be checked to see if it is normalized mode the normalization mode options Options for use with exclusion set an tailored Normalization The only option that is currently recognized is UNICODE_3_2

See Also: Normalizer

UNKNOWN: ICU 2.6

last

public int last()
Return the last character in the normalized text-> This resets the Normalizer's position to be just before the the input text corresponding to that normalized character.

Returns: The codepoint as an int

UNKNOWN: ICU 2.8

next

public int next()
Return the next character in the normalized text and advance the iteration position by one. If the end of the text has already been reached, DONE is returned.

Returns: The codepoint as an int

UNKNOWN: ICU 2.8

normalize

public static String normalize(String str, Normalizer.Mode mode, int options)
Normalizes a String using the given normalization operation.

The options parameter specifies which optional Normalizer features are to be enabled for this operation. Currently the only available option is UNICODE_3_2. If you want the default behavior corresponding to one of the standard Unicode Normalization Forms, use 0 for this argument.

Parameters: str the input string to be normalized. mode the normalization mode options the optional features to be enabled.

Returns: String the normalized string

UNKNOWN: ICU 2.6

normalize

public static String normalize(String src, Normalizer.Mode mode)
Normalize a string. The string will be normalized according the the specified normalization mode and options.

Parameters: src The string to normalize. mode The normalization mode; one of Normalizer.NONE, Normalizer.NFD, Normalizer.NFC, Normalizer.NFKC, Normalizer.NFKD, Normalizer.DEFAULT

Returns: the normalized string

UNKNOWN: ICU 2.8

normalize

public static int normalize(char[] source, char[] target, Normalizer.Mode mode, int options)
Normalize a string. The string will be normalized according the the specified normalization mode and options.

Parameters: source The char array to normalize. target A char buffer to receive the normalized text. mode The normalization mode; one of Normalizer.NONE, Normalizer.NFD, Normalizer.NFC, Normalizer.NFKC, Normalizer.NFKD, Normalizer.DEFAULT options The normalization options, ORed together (0 for no options).

Returns: int The total buffer size needed;if greater than length of result, the output was truncated.

Throws: IndexOutOfBoundsException if the target capacity is less than the required length

UNKNOWN: ICU 2.6

normalize

public static int normalize(char[] src, int srcStart, int srcLimit, char[] dest, int destStart, int destLimit, Normalizer.Mode mode, int options)
Normalize a string. The string will be normalized according the the specified normalization mode and options.

Parameters: src The char array to compose. srcStart Start index of the source srcLimit Limit index of the source dest The char buffer to fill in destStart Start index of the destination buffer destLimit End index of the destination buffer mode The normalization mode; one of Normalizer.NONE, Normalizer.NFD, Normalizer.NFC, Normalizer.NFKC, Normalizer.NFKD, Normalizer.DEFAULT options The normalization options, ORed together (0 for no options).

Returns: int The total buffer size needed;if greater than length of result, the output was truncated.

Throws: IndexOutOfBoundsException if the target capacity is less than the required length

UNKNOWN: ICU 2.6

normalize

public static String normalize(int char32, Normalizer.Mode mode, int options)
Normalize a codepoint accoding to the given mode

Parameters: char32 The input string to be normalized. mode The normalization mode options Options for use with exclusion set an tailored Normalization The only option that is currently recognized is UNICODE_3_2

Returns: String The normalized string

See Also: UNICODE_3_2

UNKNOWN: ICU 2.6

normalize

public static String normalize(int char32, Normalizer.Mode mode)
Conveinience method to normalize a codepoint accoding to the given mode

Parameters: char32 The input string to be normalized. mode The normalization mode

Returns: String The normalized string

See Also: UNICODE_3_2

UNKNOWN: ICU 2.6

previous

public int previous()
Return the previous character in the normalized text and decrement the iteration position by one. If the beginning of the text has already been reached, DONE is returned.

Returns: The codepoint as an int

UNKNOWN: ICU 2.8

quickCheck

public static Normalizer.QuickCheckResult quickCheck(String source, Normalizer.Mode mode)
Convenience method.

Parameters: source string for determining if it is in a normalized format mode normalization format (Normalizer.NFC,Normalizer.NFD, Normalizer.NFKC,Normalizer.NFKD)

Returns: Return code to specify if the text is normalized or not (Normalizer.YES, Normalizer.NO or Normalizer.MAYBE)

UNKNOWN: ICU 2.8

quickCheck

public static Normalizer.QuickCheckResult quickCheck(String source, Normalizer.Mode mode, int options)
Convenience method.

Parameters: source string for determining if it is in a normalized format mode normalization format (Normalizer.NFC,Normalizer.NFD, Normalizer.NFKC,Normalizer.NFKD) options Options for use with exclusion set an tailored Normalization The only option that is currently recognized is UNICODE_3_2

Returns: Return code to specify if the text is normalized or not (Normalizer.YES, Normalizer.NO or Normalizer.MAYBE)

UNKNOWN: ICU 2.6

quickCheck

public static Normalizer.QuickCheckResult quickCheck(char[] source, Normalizer.Mode mode, int options)
Convenience method.

Parameters: source Array of characters for determining if it is in a normalized format mode normalization format (Normalizer.NFC,Normalizer.NFD, Normalizer.NFKC,Normalizer.NFKD) options Options for use with exclusion set an tailored Normalization The only option that is currently recognized is UNICODE_3_2

Returns: Return code to specify if the text is normalized or not (Normalizer.YES, Normalizer.NO or Normalizer.MAYBE)

UNKNOWN: ICU 2.6

quickCheck

public static Normalizer.QuickCheckResult quickCheck(char[] source, int start, int limit, Normalizer.Mode mode, int options)
Performing quick check on a string, to quickly determine if the string is in a particular normalization format. Three types of result can be returned Normalizer.YES, Normalizer.NO or Normalizer.MAYBE. Result Normalizer.YES indicates that the argument string is in the desired normalized format, Normalizer.NO determines that argument string is not in the desired normalized format. A Normalizer.MAYBE result indicates that a more thorough check is required, the user may have to put the string in its normalized form and compare the results.

Parameters: source string for determining if it is in a normalized format start the start index of the source limit the limit index of the source it is equal to the length mode normalization format (Normalizer.NFC,Normalizer.NFD, Normalizer.NFKC,Normalizer.NFKD) options Options for use with exclusion set an tailored Normalization The only option that is currently recognized is UNICODE_3_2

Returns: Return code to specify if the text is normalized or not (Normalizer.YES, Normalizer.NO or Normalizer.MAYBE)

UNKNOWN: ICU 2.6

reset

public void reset()
Reset the index to the beginning of the text. This is equivalent to setIndexOnly(startIndex)).

UNKNOWN: ICU 2.8

setIndex

public int setIndex(int index)

Deprecated: ICU 3.2

Set the iteration position in the input text that is being normalized and return the first normalized character at that position.

Note: This method sets the position in the input text, while Normalizer and Normalizer iterate through characters in the normalized output. This means that there is not necessarily a one-to-one correspondence between characters returned by next and previous and the indices passed to and returned from setIndex and Normalizer.

Parameters: index the desired index in the input text->

Returns: the first normalized character that is the result of iterating forward starting at the given index. The codepoint as an int

Throws: IllegalArgumentException if the given index is less than Normalizer or greater than Normalizer.

UNKNOWN: ICU 3.2

setIndexOnly

public void setIndexOnly(int index)
Set the iteration position in the input text that is being normalized, without any immediate normalization. After setIndexOnly(), getIndex() will return the same index that is specified here.

Parameters: index the desired index in the input text.

UNKNOWN: ICU 2.8

setMode

public void setMode(Normalizer.Mode newMode)
Set the normalization mode for this object.

Note:If the normalization mode is changed while iterating over a string, calls to Normalizer and Normalizer may return previously buffers characters in the old normalization mode until the iteration is able to re-sync at the next base character. It is safest to call setText(), Normalizer, Normalizer, etc. after calling setMode.

Parameters: newMode the new mode for this Normalizer. The supported modes are:

See Also: Normalizer

UNKNOWN: ICU 2.8

setOption

public void setOption(int option, boolean value)
Set options that affect this Normalizer's operation. Options do not change the basic composition or decomposition operation that is being performed , but they control whether certain optional portions of the operation are done. Currently the only available option is:

Parameters: option the option whose value is to be set. value the new setting for the option. Use true to turn the option on and false to turn it off.

See Also: Normalizer

UNKNOWN: ICU 2.6

setText

public void setText(StringBuffer newText)
Set the input text over which this Normalizer will iterate. The iteration position is set to the beginning of the input text->

Parameters: newText The new string to be normalized.

UNKNOWN: ICU 2.8

setText

public void setText(char[] newText)
Set the input text over which this Normalizer will iterate. The iteration position is set to the beginning of the input text->

Parameters: newText The new string to be normalized.

UNKNOWN: ICU 2.8

setText

public void setText(String newText)
Set the input text over which this Normalizer will iterate. The iteration position is set to the beginning of the input text->

Parameters: newText The new string to be normalized.

UNKNOWN: ICU 2.8

setText

public void setText(CharacterIterator newText)
Set the input text over which this Normalizer will iterate. The iteration position is set to the beginning of the input text->

Parameters: newText The new string to be normalized.

UNKNOWN: ICU 2.8

setText

public void setText(UCharacterIterator newText)
Set the input text over which this Normalizer will iterate. The iteration position is set to the beginning of the string.

Parameters: newText The new string to be normalized.

UNKNOWN: ICU 2.8

startIndex

public int startIndex()
Retrieve the index of the start of the input text. This is the begin index of the CharacterIterator or the start (i.e. 0) of the String over which this Normalizer is iterating

Returns: The current iteration position

UNKNOWN: ICU 2.8

Copyright (c) 2007 IBM Corporation and others.