-- pg_turbovec v1.25.0 -- Gap-B fix: turbovec.hi_dim_rerank, a -- dimension-aware exact-L2 rerank-window widening for high-dim recall. -- -- This file is a *reference mirror*. The authoritative install -- script is generated by `cargo pgrx schema`. -- -- NO wire-format change (stays VERSION 6, byte-identical to v1.24.0). -- Existing v4/v5/v6 indexes all decode unchanged. NO REINDEX. -- -- New SQL surface (additive, minor bump): one new GUC, -- `turbovec.hi_dim_rerank` (enum: off | auto | on; default auto). -- No new operators, types, functions, or opclasses. -- -- What it does (scan-side only): the offline Gap-B investigation -- established that the -- high-dimensional recall gap (e.g. GIST-1M/960d capping ~0.86) is -- NOT retrieval-bound -- the true nearest neighbours DO land in the -- probed cells (measured cell recall 0.98-0.996 at probes 64-128) -- -- but an IN-CELL quantized-RANKING loss: the lossy 4-bit score is -- noisy enough at high dim that a true neighbour often sits at rank -- ~200-800 within the probed cells, below a small search_k. The cure -- is to fetch a WIDER candidate set so the always-on exact-L2 reorder -- recheck (xs_recheckorderby) re-ranks enough survivors to recover -- the true top-k. Measured: an SQ4 analog lifts R@10 0.666 -> 0.978 -- at 960-dim by reranking ~800 candidates instead of ~64. -- -- It must NOT blanket-widen (low dim already plateaus by search_k~=25, -- so widening there is pure latency tax), so `auto` applies a -- DIM-SCALED candidate floor of clamp(dim, 256..=1024) ONLY for -- indexes with dim >= 256, and only ever RAISES the count -- an -- explicit search_k/oversample override past the floor always wins. -- SIFT-128 is untouched; GIST-960 / OpenAI-1536 / the 512-768d -- families get the wider window automatically. `on` applies the floor -- regardless of dim; `off` is the pre-fix behaviour (honour -- search_k/oversample exactly). The result set is identical to -- setting search_k/oversample by hand to the same candidate count. -- -- Migration: `ALTER EXTENSION pg_turbovec UPDATE TO '1.25.0';`. -- No REINDEX, no wire change. The new default (auto) improves high-dim -- recall out of the box at a small high-dim-only latency cost; set -- `turbovec.hi_dim_rerank = off` to restore exact pre-1.25.0 -- scan-candidate behaviour.