--- title: Jieba description: The most advanced Chinese tokenizer that leverages both a dictionary and statistical models canonical: https://www.paradedb.com/docs/reference/tokenizers/available-tokenizers/jieba --- The Jieba tokenizer is a tokenizer for Chinese text that leverages both a dictionary and statistical models. It is generally considered to be better at identifying ambiguous Chinese word boundaries compared to the [Chinese Lindera](/reference/tokenizers/available-tokenizers/lindera) and [Chinese compatible](/reference/tokenizers/available-tokenizers/chinese-compatible) tokenizers, but the tradeoff is that it is slower. ```sql CREATE INDEX search_idx ON mock_items USING paradedb (id, (description::pdb.jieba)); ``` To get a feel for this tokenizer, run the following command and replace the text with your own: ```sql SELECT 'Hello world! 你好!'::pdb.jieba::text[]; ``` ```ini Expected Response text -------------------------------- {hello," ",world,!," ",你好,!} (1 row) ``` ## Search Mode The Jieba tokenizer enables `search_mode` by default, which splits compound words into smaller tokens in addition to keeping the original words: ```sql SELECT '南京市长江大桥'::pdb.jieba::text[]; ``` ```ini Expected Response text ------------------------------------------------ {南京,京市,南京市,长江,大桥,长江大桥} (1 row) ``` To keep only the original words, set `search_mode=false`. This can be useful when tokenizing a search query to avoid matching documents that contain only part of a compound word: ```sql SELECT '南京市长江大桥'::pdb.jieba('search_mode=false')::text[]; ``` ```ini Expected Response text --------------------- {南京市,长江大桥} (1 row) ``` ## Convert Between Traditional and Simplified Chinese Use `chinese_convert` to convert Traditional and Simplified Chinese to the same form before tokenization, so queries can match documents written in either form. ```sql SQL CREATE INDEX search_idx ON mock_items USING paradedb (id, (description::pdb.jieba('chinese_convert=t2s'))); ``` ```ts Drizzle import { indexing, tokenizer } from "@paradedb/drizzle-paradedb"; indexing .paradedbIndex("search_idx") .on( mockItems.id, indexing.paradedbField( mockItems.description, tokenizer.jieba({ chinese_convert: "t2s" }), ), ); ``` ```python Django from django.db import connection from paradedb.indexes import ParadeDBIndex from paradedb.search import Tokenizer with connection.schema_editor() as schema_editor: schema_editor.add_index( MockItem, ParadeDBIndex( fields={'id': {}, 'description': {'tokenizer': Tokenizer.jieba(options={"chinese_convert": "t2s"})}}, name="search_idx", ), ) ``` ```python SQLAlchemy from sqlalchemy import Index from paradedb.sqlalchemy import indexing from paradedb.sqlalchemy import tokenizer idx = Index( "search_idx", indexing.ParadeDBField(MockItem.id), indexing.ParadeDBField(MockItem.description, tokenizer=tokenizer.jieba(options={"chinese_convert": "t2s"})), postgresql_using="paradedb", ) with engine.begin() as conn: idx.create(conn) ``` ```ruby Rails ActiveRecord::Base.connection.add_paradedb_index( :mock_items, fields: { id: {}, description: { tokenizer: ParadeDB::Tokenizer.jieba(options: { chinese_convert: "t2s" }) }, }, name: :search_idx ) ``` ```cs EF Core modelBuilder.Entity() .HasParadeDbIndex("search_idx", e => e.Id) .HasField(e => e.Description, Tokenizer.Jieba(new() { ["chinese_convert"] = "t2s" })); ``` The following conversion modes are supported: | Mode | Description | | ------- | --------------------------------------------- | | `t2s` | Traditional to Simplified | | `s2t` | Simplified to Traditional | | `tw2s` | Traditional Taiwan to Simplified | | `tw2sp` | Traditional Taiwan to Simplified, with idioms | | `s2tw` | Simplified to Traditional Taiwan | | `s2twp` | Simplified to Traditional Taiwan, with idioms |