--- title: Unicode description: The default text tokenizer in ParadeDB canonical: https://www.paradedb.com/docs/reference/tokenizers/available-tokenizers/unicode --- The `unicode_words` tokenizer splits text according to word boundaries defined by the [Unicode Standard Annex #29](https://www.unicode.org/reports/tr29/) rules. All characters are [lowercased](/reference/token-filters/lowercase) by default. This tokenizer is the default text tokenizer. If no tokenizer is specified for a text field, the `unicode_words` tokenizer will be used. ```sql -- The following two configurations are equivalent CREATE INDEX search_idx ON mock_items USING paradedb (id, description); ``` Or specify the tokenizer explicitly: ```sql CREATE INDEX search_idx ON mock_items USING paradedb (id, (description::pdb.unicode_words)); ``` To get a feel for this tokenizer, run the following command and replace the text with your own: ```sql SELECT 'Tokenize me!'::pdb.unicode_words::text[]; ``` ```ini Expected Response text --------------- {tokenize,me} (1 row) ``` ## Remove Emojis By default, emojis in the source text are preserved. To remove emojis, set `remove_emojis` to `true`. ```sql SELECT 'Tokenize me! 😊'::pdb.unicode_words('remove_emojis=true')::text[]; ``` ```ini Expected Response text --------------- {tokenize,me} (1 row) ```