--- title: Query-Time Tokenizer description: Use a different tokenizer at search time than at index time canonical: https://www.paradedb.com/docs/reference/tokenizers/search-tokenizer --- By default, ParadeDB uses the same tokenizer at both index time and search time. This makes sense for most cases — you want queries tokenized the same way the data was indexed. But sometimes you need different tokenizers. The classic example is **autocomplete**: - **Index time** — `edge_ngram`: `"shoes"` → `s`, `sh`, `sho`, `shoe`, `shoes` - **Search time** — `unicode_words`: `"sho"` → `sho` If you used `edge_ngram` at search time too, typing `"sho"` would produce `s`, `sh`, `sho` — matching far too many documents. ## Usage Set `search_tokenizer` as a `WITH` option on the index to define a default search-time tokenizer for all text and JSON fields: ```sql SQL CREATE INDEX search_idx ON mock_items USING paradedb (id, (description::pdb.edge_ngram(1, 10))) WITH (search_tokenizer = 'unicode_words'); ``` ```ts Drizzle import { indexing, tokenizer } from "@paradedb/drizzle-paradedb"; indexing .paradedbIndex("search_idx", { searchTokenizer: tokenizer.unicodeWords() }) .on( mockItems.id, indexing.paradedbField(mockItems.description, tokenizer.edgeNgram(1, 10)), ); ``` ```python Django from django.db import connection from paradedb.indexes import ParadeDBIndex from paradedb.search import Tokenizer with connection.schema_editor() as schema_editor: schema_editor.add_index( MockItem, ParadeDBIndex( fields={'id': {}, 'description': {'tokenizer': Tokenizer.edge_ngram(min_gram=1, max_gram=10)}}, name="search_idx", search_tokenizer=Tokenizer.unicode_words(), ), ) ``` ```python SQLAlchemy from sqlalchemy import Index from paradedb.sqlalchemy import indexing from paradedb.sqlalchemy import tokenizer idx = Index( "search_idx", indexing.ParadeDBField(MockItem.id), indexing.ParadeDBField(MockItem.description, tokenizer=tokenizer.edge_ngram(min_gram=1, max_gram=10)), postgresql_using="paradedb", postgresql_with={'search_tokenizer': 'unicode_words'}, ) with engine.begin() as conn: idx.create(conn) ``` ```ruby Rails ActiveRecord::Base.connection.add_paradedb_index( :mock_items, fields: { id: {}, description: { tokenizer: ParadeDB::Tokenizer.edge_ngram(1, 10) }, }, name: :search_idx, index_options: { search_tokenizer: "unicode_words" } ) ``` ```cs EF Core modelBuilder.Entity() .HasParadeDbIndex("search_idx", e => e.Id) .HasField(e => e.Description, Tokenizer.EdgeNgram(1, 10)) .HasSearchTokenizer(Tokenizer.Unicode()); ``` With this configuration: - **Index time**: `description` is tokenized into edge ngrams - **Search time**: queries against `description` automatically use the `unicode_words` tokenizer The `search_tokenizer` value can include parameters, e.g. `search_tokenizer='simple(lowercase=false)'`. Because `search_tokenizer` only affects query-time behavior, you can change it without reindexing: ```sql ALTER INDEX search_idx SET (search_tokenizer = 'simple(lowercase=false)'); ``` ## Example ```sql CREATE TABLE products ( id serial8 NOT NULL PRIMARY KEY, title text ); INSERT INTO products (title) VALUES ('shoes'), ('shirt'), ('shorts'), ('shoelaces'), ('socks'); CREATE INDEX idx_products ON products USING paradedb (id, (title::pdb.edge_ngram(1, 10))) WITH (search_tokenizer = 'unicode_words'); -- "sho" stays as one token → matches shoes, shorts, shoelaces SELECT id, title FROM products WHERE title ||| 'sho' ORDER BY id; -- "s" stays as one token → matches all five titles SELECT id, title FROM products WHERE title ||| 's' ORDER BY id; ``` Without `search_tokenizer`, the query `'sho'` would be tokenized into the edge ngrams `s`, `sh`, `sho` and match every title starting with `s` — not just those starting with `sho`. ## Overriding at Query Time You can still override the search tokenizer for a specific query by casting the query string: ```sql -- Force edge ngram tokenization at query time SELECT id, title FROM products WHERE title ||| 'sho'::pdb.edge_ngram(1, 10) ORDER BY id; ``` ## Priority When resolving which tokenizer to use at search time, ParadeDB checks in this order: 1. **Query-level cast** — e.g. `'sho'::pdb.edge_ngram(...)` (highest priority) 2. **Index-level WITH option** — e.g. `WITH (search_tokenizer='unicode_words')` 3. **Index-time tokenizer** — the tokenizer used to build the index (fallback) ## Supported Tokenizers Any [available tokenizer](/reference/tokenizers/overview) can be used as a `search_tokenizer`: `unicode_words`, `simple`, `whitespace`, `ngram`, `edge_ngram`, `regex_pattern`, `literal`, `literal_normalized`, `chinese_compatible`, `lindera`, `icu`, `jieba`, `source_code`.