# Benchmark results

| | |
|---|---|
| PostgreSQL | PostgreSQL 18.4 (Homebrew) on aarch64-apple-darwin24.6.0, compiled by Apple clang version 17.0.0 (clang-1700.6.4.2), 64-bit |
| ICU | 78.3 |
| CPU | Apple M4 Pro |
| Corpus | Wikipedia article titles, dump 20260901, 500000 titles per language |
| Runs | median of 5 after one warm-up run, one backend, no parallel workers, JIT off |
| LC_CTYPE | C.UTF-8 |

## Tokenizing

Time to tokenize every title once, minus the time to scan the table. "vs default" divides by the default parser's time.

| Language | Titles | Avg bytes | Method | µs per title | vs default |
|---|--:|--:|---|--:|--:|
| en | 500,000 | 21.0 | default parser | 0.748 | 1.00× |
| en | 500,000 | 21.0 | intl_simple | 0.700 | 0.94× |
| en | 500,000 | 21.0 | intl_search | 1.000 | 1.34× |
| en | 500,000 | 21.0 | intl_search_ngram | 1.081 | 1.45× |
| en | 500,000 | 21.0 | pg_trgm | 1.218 | 1.63× |
| en | 500,000 | 21.0 | pg_bigm | 0.934 | 1.25× |
| zh | 500,000 | 16.9 | default parser | 0.455 | 1.00× |
| zh | 500,000 | 16.9 | intl_simple | 1.587 | 3.49× |
| zh | 500,000 | 16.9 | intl_search | 2.109 | 4.64× |
| zh | 500,000 | 16.9 | intl_search_ngram | 2.325 | 5.11× |
| zh | 500,000 | 16.9 | pg_trgm | 0.773 | 1.70× |
| zh | 500,000 | 16.9 | pg_bigm | 0.485 | 1.07× |
| ja | 500,000 | 22.2 | default parser | 0.602 | 1.00× |
| ja | 500,000 | 22.2 | intl_simple | 1.888 | 3.14× |
| ja | 500,000 | 22.2 | intl_search | 2.367 | 3.93× |
| ja | 500,000 | 22.2 | intl_search_ngram | 2.661 | 4.42× |
| ja | 500,000 | 22.2 | pg_trgm | 1.020 | 1.70× |
| ja | 500,000 | 22.2 | pg_bigm | 0.660 | 1.10× |
| ko | 500,000 | 19.1 | default parser | 0.555 | 1.00× |
| ko | 500,000 | 19.1 | intl_simple | 0.745 | 1.34× |
| ko | 500,000 | 19.1 | intl_search | 1.056 | 1.91× |
| ko | 500,000 | 19.1 | intl_search_ngram | 1.412 | 2.55× |
| ko | 500,000 | 19.1 | pg_trgm | 0.898 | 1.62× |
| ko | 500,000 | 19.1 | pg_bigm | 0.632 | 1.14× |
| th | 366,332 | 45.9 | default parser | 1.094 | 1.00× |
| th | 366,332 | 45.9 | intl_simple | 1.831 | 1.67× |
| th | 366,332 | 45.9 | intl_search | 2.498 | 2.28× |
| th | 366,332 | 45.9 | intl_search_ngram | 3.434 | 3.14× |
| th | 366,332 | 45.9 | pg_trgm | 2.235 | 2.04× |
| th | 366,332 | 45.9 | pg_bigm | 1.576 | 1.44× |

## GIN index build

Single-process `CREATE INDEX` with `maintenance_work_mem = 1GB`.

| Language | Method | Build seconds | Index size MB |
|---|---|--:|--:|
| en | default parser | 1.4 | 25.5 |
| en | intl_simple | 1.3 | 23.7 |
| en | intl_search | 1.5 | 23.3 |
| en | intl_search_ngram | 1.5 | 23.4 |
| en | pg_trgm | 1.7 | 29.5 |
| en | pg_bigm | 1.7 | 20.4 |
| zh | default parser | 1.3 | 38.2 |
| zh | intl_simple | 1.6 | 14.0 |
| zh | intl_search | 1.8 | 13.3 |
| zh | intl_search_ngram | 2.2 | 17.1 |
| zh | pg_trgm | 2.7 | 72.0 |
| zh | pg_bigm | 2.5 | 41.8 |
| ja | default parser | 1.4 | 39.6 |
| ja | intl_simple | 1.7 | 18.4 |
| ja | intl_search | 2.0 | 18.2 |
| ja | intl_search_ngram | 3.0 | 20.7 |
| ja | pg_trgm | 2.9 | 63.7 |
| ja | pg_bigm | 2.5 | 33.9 |
| ko | default parser | 1.4 | 30.1 |
| ko | intl_simple | 1.6 | 28.6 |
| ko | intl_search | 1.7 | 28.7 |
| ko | intl_search_ngram | 2.6 | 24.5 |
| ko | pg_trgm | 2.3 | 48.9 |
| ko | pg_bigm | 2.0 | 24.3 |
| th | default parser | 1.5 | 37.6 |
| th | intl_simple | 1.5 | 13.7 |
| th | intl_search | 2.2 | 13.6 |
| th | intl_search_ngram | 5.8 | 45.5 |
| th | pg_trgm | 1.6 | 23.4 |
| th | pg_bigm | 1.6 | 18.8 |

## Word queries

Each language runs its query terms one at a time as `count(*)`, through each method's index. The tsvector methods use `plainto_tsquery`, and the n-gram methods use `LIKE '%term%'`. Recall and overmatch come from the labeled sample below: recall is the share of real matches the method returned, and overmatch is the share of the method's returned titles that aren't real matches.

| Language | Terms | Method | Queries using the index | Avg ms per query | Hits | Recall | Overmatch |
|---|--:|---|--:|--:|--:|--:|--:|
| en | 200 | default parser | 200 | 0.234 | 123,406 | — | — |
| en | 200 | intl_simple | 200 | 0.275 | 119,996 | — | — |
| en | 200 | intl_search | 200 | 0.266 | 121,265 | — | — |
| en | 200 | intl_search_ngram | 200 | 0.272 | 121,265 | — | — |
| en | 200 | pg_trgm | 182 | 3.060 | 708,250 | — | — |
| en | 200 | pg_bigm | 200 | 2.063 | 708,250 | — | — |
| zh | 200 | default parser | 200 | 0.037 | 5,396 | 10.3% | 0.0% |
| zh | 200 | intl_simple | 200 | 0.164 | 68,070 | 92.6% | 15.6% |
| zh | 200 | intl_search | 200 | 0.185 | 97,880 | 92.6% | 15.6% |
| zh | 200 | intl_search_ngram | 200 | 0.336 | 111,835 | 99.0% | 17.6% |
| zh | 200 | pg_trgm | 22 | 23.104 | 79,536 | 100.0% | 18.4% |
| zh | 200 | pg_bigm | 200 | 0.111 | 79,536 | 100.0% | 18.4% |
| ja | 200 | default parser | 200 | 0.036 | 7,373 | 7.0% | 0.0% |
| ja | 200 | intl_simple | 200 | 0.132 | 86,443 | 86.1% | 19.5% |
| ja | 200 | intl_search | 200 | 0.140 | 88,779 | 86.1% | 19.5% |
| ja | 200 | intl_search_ngram | 200 | 0.530 | 184,736 | 100.0% | 53.6% |
| ja | 200 | pg_trgm | 57 | 19.852 | 175,927 | 100.0% | 54.0% |
| ja | 200 | pg_bigm | 200 | 0.285 | 175,927 | 100.0% | 54.0% |
| ko | 200 | default parser | 200 | 0.070 | 35,053 | 46.7% | 0.0% |
| ko | 200 | intl_simple | 200 | 0.099 | 36,678 | 48.2% | 1.1% |
| ko | 200 | intl_search | 200 | 0.082 | 36,678 | 48.2% | 1.1% |
| ko | 200 | intl_search_ngram | 200 | 0.398 | 93,757 | 100.0% | 22.0% |
| ko | 200 | pg_trgm | 91 | 17.704 | 93,655 | 100.0% | 22.0% |
| ko | 200 | pg_bigm | 200 | 0.139 | 93,655 | 100.0% | 22.0% |
| th | 200 | default parser | 200 | 0.037 | 6,997 | 5.3% | 0.0% |
| th | 200 | intl_simple | 200 | 0.294 | 215,162 | 78.9% | 43.8% |
| th | 200 | intl_search | 200 | 0.292 | 215,162 | 78.9% | 43.8% |
| th | 200 | intl_search_ngram | 200 | 0.586 | 447,620 | 98.2% | 69.4% |
| th | 200 | pg_trgm | 133 | 17.208 | 620,044 | 100.0% | 77.2% |
| th | 200 | pg_bigm | 200 | 1.089 | 620,044 | 100.0% | 77.2% |

## Labeled sample

For each language, `bench/pairs.sql` draws term/title pairs at random from every title in the sample that contains one of the query terms as a substring, and `bench/labels.csv` marks each pair `match` or `over`. A pair is a `match` when the title uses the term as a word: on its own, with grammatical endings (Korean particles, Japanese inflection), or as part of a compound whose meaning includes it (`機場` in `國際機場`). It is `over` when the characters are there but the word isn't: inside a transliterated name (`阿拉` in `阿拉巴马州`), across a word boundary (`京都` in `東京都`), or inside an unrelated word. Every method is scored on the same pairs, so a method that returns every substring scores 100% recall and pays in overmatch.

| Language | Labeled pairs | Real matches |
|---|--:|--:|
| zh | 250 | 81.6% |
| ja | 250 | 46.0% |
| ko | 250 | 78.0% |
| th | 250 | 22.8% |