--- title: Chinese Compatible description: A simple tokenizer for Chinese, Japanese, and Korean characters canonical: https://www.paradedb.com/docs/reference/tokenizers/available-tokenizers/chinese-compatible --- The Chinese compatible tokenizer is like the [simple](/reference/tokenizers/available-tokenizers/simple) tokenizer -- it lowercases non-CJK characters and splits on any non-alphanumeric character. Additionally, it treats each CJK character as its own token. ```sql CREATE INDEX search_idx ON mock_items USING paradedb (id, (description::pdb.chinese_compatible)); ``` To get a feel for this tokenizer, run the following command and replace the text with your own: ```sql SELECT 'Hello world! 你好!'::pdb.chinese_compatible::text[]; ``` ```ini Expected Response text --------------------- {hello,world,你,好} (1 row) ``` ## Convert Between Traditional and Simplified Chinese Use `chinese_convert` to convert Traditional and Simplified Chinese to the same form before tokenization, so queries can match documents written in either form. ```sql SQL CREATE INDEX search_idx ON mock_items USING paradedb (id, (description::pdb.chinese_compatible('chinese_convert=t2s'))); ``` ```ts Drizzle import { indexing, tokenizer } from "@paradedb/drizzle-paradedb"; indexing .paradedbIndex("search_idx") .on( mockItems.id, indexing.paradedbField( mockItems.description, tokenizer.chineseCompatible({ chinese_convert: "t2s" }), ), ); ``` ```python Django from django.db import connection from paradedb.indexes import ParadeDBIndex from paradedb.search import Tokenizer with connection.schema_editor() as schema_editor: schema_editor.add_index( MockItem, ParadeDBIndex( fields={'id': {}, 'description': {'tokenizer': Tokenizer.chinese_compatible(options={"chinese_convert": "t2s"})}}, name="search_idx", ), ) ``` ```python SQLAlchemy from sqlalchemy import Index from paradedb.sqlalchemy import indexing from paradedb.sqlalchemy import tokenizer idx = Index( "search_idx", indexing.ParadeDBField(MockItem.id), indexing.ParadeDBField(MockItem.description, tokenizer=tokenizer.chinese_compatible(options={"chinese_convert": "t2s"})), postgresql_using="paradedb", ) with engine.begin() as conn: idx.create(conn) ``` ```ruby Rails ActiveRecord::Base.connection.add_paradedb_index( :mock_items, fields: { id: {}, description: { tokenizer: ParadeDB::Tokenizer.chinese_compatible(options: { chinese_convert: "t2s" }) }, }, name: :search_idx ) ``` ```cs EF Core modelBuilder.Entity() .HasParadeDbIndex("search_idx", e => e.Id) .HasField(e => e.Description, Tokenizer.ChineseCompatible(new() { ["chinese_convert"] = "t2s" })); ``` The following conversion modes are supported: | Mode | Description | | ------- | --------------------------------------------- | | `t2s` | Traditional to Simplified | | `s2t` | Simplified to Traditional | | `tw2s` | Traditional Taiwan to Simplified | | `tw2sp` | Traditional Taiwan to Simplified, with idioms | | `s2tw` | Simplified to Traditional Taiwan | | `s2twp` | Simplified to Traditional Taiwan, with idioms |