# Scale plan — from here to best-at-TB–PB-under-pooled-concurrency

> **Status:** DRAFT roadmap, 2026-10-15 (revised 2026-10-16 after
> `.agent/notes/SCALE_PLAN_REVIEW.md` — 2 blockers + 8 should-fixes resolved;
> blocker [B2] since RESOLVED by the S2 feasibility spike,
> `.agent/notes/S2_SPIKE_FINDINGS.md`; blocker [B1] since RESOLVED by the
> `log_newpage` investigation, `.agent/notes/B1_log_newpage_investigation.md`).
> Grounded in the measured
> pg_vectorsearch comparison (`benches/results/vs_pgvectorsearch_20261014/FINDINGS.md`),
> the code-level bottleneck analysis (`.agent/notes/SCALE_RESEARCH_concurrency.md`,
> every claim cited to `file:line`), and the benchmark methodology
> (`benches/METHODOLOGY.md`). Not a release schedule — a direction with
> phases, each gated by the methodology's pooled-concurrency headline and the
> HARD MANDATE in `AGENTS.md` (no corruption, old formats stay readable,
> every change proven by a test that fails-before/passes-after).

## 1. The goal, measured

Be the best vector-search extension for **TB–PB indexes hit constantly by
thousands of pooled clients on a tuned host** (huge pages, large
`shared_buffers`, ZFS + ZIL/SLOG + L2ARC on NVMe), and competitive at every
smaller scale. "Best" is defined by the methodology's headline metrics:
sustained **QPS and QPS/core**, **tail latency (p95/p99/p99.9) under a
pooler**, **memory-per-backend × backend-count**, over a **working set larger
than RAM**. Measured on the three-tier distributed rig (DB on metal, pooler on
its own instance, load fleet of small spot instances).

## 2. Where we are (the honest baseline)

From the measured comparison and verified in code:

| axis | prism (pg_vectorsearch) | pg_turbovec v2.13.0 | gap |
|---|---|---|---|
| CPU/query (1M flat, 0.95) | ~1.0 CPU-ms, 1 thread | ~5.5 CPU-ms, 5 threads | ~5× CPU |
| QPS @ 96 clients (1M) | 49,641 | 2,066–2,108 | **15–24×** |
| QPS @ 96 clients (10M) | 22,294 | 194–229 | **~100×** |
| per-backend resident RAM | ~3 MB | 0.5 GB (1M) … 5 GB (10M) | **100–1000×** |
| cold first-query (fresh backend) | ~40–100 ms | 250–500 ms | ~5× |
| 10M warm p50 @ 0.95 | 2.2 ms | 63 ms | **17–29×** |
| build WAL vs index size | ~1× | ~2× | 2× |

We win: 1M at high recall with the flat index (near-exact ~2 ms), exact
filtered results, deterministic builds, Postgres-native ACID/joins, storage
efficiency vs HNSW.

**The one sentence that explains every loss:** prism serves straight from
`shared_buffers` at ~1 core and a few hundred µs/query, holding ~3 MB of
bounded per-backend state; pg_turbovec holds a **multi-GB private
per-backend decoded copy** of the index and spends **5–16 cores and
40–200 ms/query**. At pooled scale that private cache is disqualifying before
QPS even enters: 1,000 backends × 0.5–5 GB = **0.5–5 TB of duplicated RAM for
one on-disk artefact.**

## 3. Target architecture (the "there")

Three structural shifts, each turning a per-backend/per-query cost into a
per-relation/shared cost:

1. **One decoded index per relation in shared memory, mapped read-only by
   every backend** — not a private copy per backend. This is the keystone.
2. **One core per query by default** under concurrency — intra-query threads
   become an explicit low-concurrency latency knob, not the default.
3. **No per-query index construction** — the IVF path precomputes and shares
   per-cell structures; nothing that is a pure function of the index is
   rebuilt per query (rotation matrix, codebook, blocked layout, compact
   sub-indexes).

Supporting shifts: a rerank window bounded by actual `LIMIT`/recall rather
than dim; bulk-build WAL via `log_newpage` instead of per-page full-page
images.

## 4. Phased roadmap (ordered by impact for the target market)

Each phase is independently shippable, keeps old on-disk formats readable, and
is validated on the pooled rig against the phase before it. Severity ranks map
to the research doc's ranked bottlenecks (#1–#7). **Order:**
S0 → S1 → S4 → S2 → S3 → S5 → S6 — S4 runs *before* the keystone S2 because it
is the cheapest change in the plan and helps at **every** scale, while S2 is
the highest-risk phase and may need a spike before it is even known to be
feasible (see S2 [B2]).

### Phase S0 — honesty + measurement foundation (now, no code risk)

The prerequisite for all of it: we cannot tune what we cannot see.

- Publish the measured comparison honestly in README/docs (done).
- Land `benches/METHODOLOGY.md` (done) and build the **three-tier pooled
  harness** under `benches/scripts/pooled/` (DB metal + pooler instance + spot
  load fleet, open+closed loop, >RAM corpus, tuned storage). This harness is
  the gate for every later phase — a change is only "better" if it moves the
  pooled headline.
- Build a **multi-backend test harness** (two or more real backends against
  one cluster, driven out-of-process), because the HARD-MANDATE rule "every
  logic change ships with a fails-before/passes-after test" is
  **unsatisfiable by a single-backend `#[pg_test]`** for anything
  cross-backend — and S2 is entirely cross-backend. A `#[pg_test]`'s outer
  transaction rolls back before `PreCommit`, so shared-cache behaviour cannot
  be exercised in-process (AGENTS.md). This harness is a hard prerequisite for
  S2, not a detail.
- Establish the **baseline pooled numbers** for v2.13.0 on the rig so every
  phase has a before/after. (The current numbers are from a collapsed
  single-host topology; re-measure on the real rig first.)

### Phase S1 — one core per query under concurrency (bottleneck #2)

Removes the **oversubscription collapse**; a prerequisite, not a gap-closer on
its own.

- Make `scan_parallelism = 1` the effective default under concurrency —
  either document it as the pooled default and ship a tuned `postgresql.conf`
  fragment, or auto-collapse to 1 thread when the backend/active-query count
  indicates a loaded server. Keep multi-thread as an explicit low-concurrency
  latency knob (it already measures correctly at `sp=1`: `FINDINGS.md:254`).
- Remove the per-query pool `install`/futex handoff cost at `sp=1`
  (`cache::bounded`/`single_thread_pool`, `src/cache.rs:1303-1330`).
- *Honest scope:* S1 removes the collapse where ~20 concurrent queries × 5
  threads saturate 96 cores, which is real and matters at concurrency — but it
  does **not** close the single-core CPU/query gap. The equal-CPU row shows
  `sp=1` still loses **23× at 1M** (2,108 vs 49,641 TPS) and **~115× at 10M**
  (194 vs 22,294) because the per-query *fixed cost* (brute-force flat scan,
  always-full exact recheck, per-query IVF build) dominates even at one
  thread (`FINDINGS.md` P5; research §1). S3 and S4 close that gap, not S1.
- *Gate:* pooled QPS/core at high concurrency stops collapsing under
  oversubscription; the CPU/query gap is left to S3/S4.
- *Risk:* minor (default thread count changed) — same class as v2.13.0.

### Phase S4 — bound the exact rerank window by LIMIT, not dim (bottleneck #4)

Promoted ahead of the keystone: the cheapest change in the plan, and the
single biggest per-query CPU lever at `sp=1`, at **every** scale.

- Make the exact-rerank candidate count a function of actual `LIMIT`/recall
  need, not `clamp(dim, 256..=1024)`
  (`hi_dim_rerank_candidate_count`, `src/guc.rs:224-261`,
  `HI_DIM_RERANK_MIN_DIM=256` at `src/guc.rs:214`). A `LIMIT 10` query at
  1024-d should not exact-recompute ~1024 heap tuples by default:
  `hi_dim_rerank=off` + `search_k=16` already measures ~5 ms vs ~42 ms flat
  CPU (~8×) at R@10 0.998 (`p0_findings.md:332`).
- **The in-extension lever is candidate count ONLY.** A "tighter order-by
  lower bound so the reorder queue stops early" is a documented dead end:
  under `xs_recheckorderby`, core's `IndexNextWithReorder` unconditionally
  heap-fetches and exact-recomputes every tuple `amgettuple` returns *before*
  reading our advertised value — investigated and REJECTED as a no-op
  (`src/index/scan.rs:834-857`). Cutting candidate count is the only lever
  short of an upstream core patch to `IndexNextWithReorder`; that core change
  is out of scope for this phase (file it separately if pursued).
- *Release contract:* the exact kernels and scoring stay bit-identical
  (AGENTS.md; `normalise_into` untouched). Changing the *candidate count* is a
  recall/latency trade — a tunable + default change is a **minor**, not a
  scoring-bit change, so it does not break the patch bit-identity rule. It
  *will* shift which rows/order come back for some queries, so prove recall on
  the matched-recall Pareto.
- *Gate:* flat and IVF CPU/query drop ~5–8× at matched recall, at all scales.

### Phase S2 — one decoded index per relation in shared memory (bottleneck #1) — THE KEYSTONE

The structural fix that removes the 0.5–5 TB duplication and the 250–500 ms
cold build. **Feasibility [B2] is RESOLVED (spike, 2026-10-16): buildable, but
only via the turbovec API additions in S2a below — there is no fallback.**
Highest *engineering* risk in the plan; prototype behind a GUC and prove on
the rig before defaulting.

- **What is actually duplicated vs already shared** (the win is precise): the
  *raw relfile pages* already live once in `shared_buffers`; sharing them
  again buys little. What is duplicated per backend — and what S2 shares — is
  the **decoded / SIMD-blocked layout** produced by `prepare()`, plus
  `slot_to_id`, the rotation matrix, the coarse centroids, and the centroid
  graph. That re-decode + re-block *is* the 250–500 ms cold cost
  (`FINDINGS.md:214`) and the 0.5–5 GB private footprint. S2 materialises that
  decoded form **once per index relation** in DSA / a dynamic shared-memory
  segment, mapped read-only by all backends.
- **Cross-backend lifecycle (the OOM/leak class, not corruption).**
  Invalidation keyed on `(relfilenode, freshness)` as today, but freeing a
  superseded DSA allocation is now **cross-process refcounting**: a
  `REINDEX`/`VACUUM`/writer bumps the generation and backends re-map, and the
  old segment is freed only when the last reader unmaps. A single leaked ref
  pins the segment for *every* backend — worse than the historical private
  leaked-`Arc` OOM (AGENTS.md, ~80 min to OOM). S2's gate therefore includes a
  sustained concurrent-write **soak watching backend RSS *and* segment
  occupancy**, not just `turbovec_check`.
- **Corruption posture (HARD MANDATE).** The shared image is **read-only** to
  scanners; the raw relfile stays the source of truth; a map failure falls
  back to the private path, never to a wrong answer. The mutable (`aminsert`)
  arm stays per-backend until S5 — so during the S2 window a **write-heavy
  index gets no sharing benefit** and the memory wall is lifted only for
  read-mostly indexes; state that limit in the gate.
- **NUMA + triple-caching (metal-host first-order concerns).** On a
  multi-socket metal box a single segment pinned to one NUMA node makes every
  cross-node backend pay remote-memory latency per scan — scope first-touch /
  interleave / per-node replication of the shared image. And the raw bytes can
  now live in **three** places — ZFS ARC, `shared_buffers`, and the shared
  decoded segment — so size the shared image against the memory budget the
  whole exercise is trying to shrink; do not let triple-caching eat the win.
- *Gate:* per-backend resident RAM drops from GB to ~MB; cold first-query on a
  fresh pooled backend drops from 250–500 ms toward prism's ~tens of ms; a
  thousand-backend pool fits on the host; the write-soak shows bounded segment
  occupancy. Biggest lever on the "memory-per-backend × backend-count"
  headline — for read-mostly indexes.
- *Risk:* major engineering, gated by the spike; prototype behind a GUC
  (`turbovec.shared_cache = off` default) and prove it on the rig before
  defaulting. Likely a minor when first shipped off-by-default.

### Phase S3 — kill per-query index construction on the IVF path (bottleneck #3)

The fix for the 17–29× 10M loss. **Depends on S2 (or its fallback):** the
per-cell sub-indexes must live in the shared image, or they are just another
private cache.

- Cache the per-cell compact sub-indexes (or at least their blocked layout +
  the shared codebook/rotation) across queries, keyed by cell id, so a probed
  cell is built once per relation (in the S2 shared image), not once per chunk
  per query (`search_compact_chunk` throwaway `from_parts`+`prepare()`,
  `src/cache.rs:1135/1167`). The codebook/rotation are a pure function of
  `(bit_width, dim)` and identical across queries — they must not be rebuilt.
- Longer horizon: evaluate a **posting-list + precomputed-code** IVF layout
  (prism-style). This reorganises the *physical page layout of the codes*
  (contiguous posting lists vs today's cell-range gather) — a genuine wire
  change, **not** a free `kind`-byte add like `KIND_BQ` (which selected a
  decode path over the *same* bytes). Per the HARD MANDATE it needs
  transparent old-format decode **or** an offline in-place page converter
  (a corpus REINDEX is banned); plan it as "**new `kind` + offline converter**,
  not additive," so nobody scopes it as free and discovers otherwise.
- *Gate:* 10M IVF CPU/query and p50 fall toward prism's; the "our flat beats
  our own IVF" inversion (`FINDINGS.md:155-157`) disappears.

### Phase S5 — write path for write-heavy TB-scale (bottleneck #6, then #1-writer)

- **[B1] RESOLVED by investigation** (2026-10-16,
  `.agent/notes/B1_log_newpage_investigation.md`): `log_newpage` is a **dead
  end for this goal and the §11a ban stays.** Measured: the ~2× build WAL is
  **99.56% full-page-image payload**, and `log_newpage` emits the *identical*
  FPI through the *identical* core mechanism (`XLOG_FPI`/`RM_XLOG_ID` vs
  GenericXLog's `GENERIC_XLOG_FULL_IMAGE`/`RM_GENERIC_ID` — same bytes,
  rmgr-agnostic), so switching WAL primitive cannot cut the 2× (≤0.44%
  framing is all that's reachable). `log_newpage` is in fact portability-safe
  (not a custom rmgr), but allowing it buys nothing, so keep §11a's clarity
  intact. **No §11a change; S5 stays 100% GenericXLog.**
- **The real cause of the 2× (what S5 must attack):** build logs **7093 FPIs
  for 3623 index pages (~1.96× re-logging)** — `extend_to`, per-chain
  `write_chain_at`, and the trailing `write_meta` each WAL-log overlapping
  pages as *full* images. The lever is the **number of full images emitted**,
  not the WAL API. Two sub-targets, both inside GenericXLog, wire-neutral:
  1. **Write each build page once.** Audit `extend_to` + `write_chain_at` +
     `write_meta` (`src/index/relfile.rs:413,588,632`) so a build touches each
     block a single time under one pass — the extend-then-rewrite and
     trailing-meta FPIs are pure waste. Estimated upside: toward ~1× (the real
     S5 goal), reachable with no §11a change.
  2. **Drop `GENERIC_XLOG_FULL_IMAGE` only on genuinely-partial-page
     rewrites** — `copy_slot_in_chain` (`:2443,2467`, VACUUM swap-remove) and
     the production meta-field pokes change a few bytes of an *existing* page,
     so `computeDelta` can log just the fragment and `generic_redo` replays it
     safely (`REGBUF_STANDARD`). Keep FULL_IMAGE on the whole-page/`PageInit`'d
     writes (`:413,588,632`) — dropping it there risks the hole-zeroing
     corruption the `relfile.rs:414-418` comments warn about. (Sites
     `:3287,3325,3357,3408,3451` are `#[cfg(test)]` helpers — do not count
     them.) Small for the bulk build, real for VACUUM/incremental flushes.
  **Must-keep guard:** any FULL_IMAGE drop needs a fails-before/passes-after
  corruption test AND a sustained-load soak (HARD MANDATE) — a torn/zeroed
  page here is exactly the corruption class fixed six times.
- Revisit the writer-side cache once S2 lands (shared read image + per-backend
  write buffer, flushed to the shared image on commit).
- *Must respect:* durability stays 100% crash-safe and 100% GenericXLog
  (§11a unchanged).

### Phase S6 — residual contention + polish (bottleneck #7)

- Once S3 removes the per-chunk `codebook` call, the process-wide `MEMO` mutex
  (`~/src/turbovec/.../codebook.rs:34`) leaves the hot path. Shard or
  lock-free-read the `CACHE` map (`src/cache.rs:1585`) only if it profiles hot
  on the pooled rig (it is µs-bounded today).
- Reduce the CBOR heap-datum decode cost (5127 B/row vs pgvector 4104 B,
  `FINDINGS.md:365`) — ties into the long-planned raw-vector-varlena work
  (`design/raw-vector-varlena`, already renumbered; reconcile with this plan).

### Phase S7 — sparse bitmaps for tombstones + the IVF probe mask (independent)

pg_turbovec rolls its own **dense** bitmaps in two places that are sparse in
practice and grow with the corpus — so at TB–PB scale they waste on-disk/WAL
bytes and per-query RAM. `~/ws/sparsemap` (our C fork, v5.8.3) is a
compressed sparse bitmap built for exactly this (long runs of set/unset
bits). This phase is **independent of the S2 keystone chain** and can run in
parallel.

- **On-disk tombstone bitmap.** `relfile::read/write_tombstones` +
  `src/index/vacuum.rs:358-371,405-466` persist a dense `Vec<u8>` of
  `n_vectors/8` bytes — one bit per slot, written to the relfile even when few
  slots are dead. At billions of slots this is large and almost all zeros.
  Replacing it with a sparsemap-encoded chunk shrinks the persisted tombstone
  region and its WAL. **Wire-format impact:** the tombstone chain layout
  changes, so this is additive-decode (read old dense, write new sparse) or a
  new `kind`/version per the HARD MANDATE — scope it like any format change,
  with a fails-before/passes-after corruption test and a sustained-load soak.
- **Per-query IVF probe mask.** `ivf.rs:195` `probe_mask() -> Vec<bool>`
  allocates one byte per vector over the whole corpus per query (the in-kernel
  allowlist/cell mask). A sparse representation cuts per-query allocation and
  cache footprint — overlaps the per-query-cost work in S3/S4. In-memory only,
  no wire impact.
- **Prerequisite / scope:** confirm whether we want a Rust binding to the C
  library (FFI/`cc` build, like the turbovec kernel) or a pure-Rust
  equivalent; measure the compressed size + encode/decode cost against the
  dense baseline on a realistically-sparse tombstone set BEFORE adopting —
  sparsemap wins only when the bitmap is actually sparse, so a densely-churned
  index must not regress. **sparsemap currency:** pg_turbovec does **not**
  depend on sparsemap today (no cargo dep, no vendored copy); `~/ws/sparsemap`
  is at v5.8.3. If/when adopted, pin the version and track it.

## 5. Sequencing and why

- **S0 first, always** — no measurement rig and no multi-backend test harness,
  no credible claim of "better" and no way to test S2 at all.
- **S1 then S4** — both ship early as minors with no format change. S1 removes
  the oversubscription collapse (necessary, not sufficient); S4 is the
  cheapest change and the biggest single-core CPU/query lever, at every scale.
  Neither alone closes the gap to prism, and the plan says so plainly (S1
  honest-scope note).
- **S4 before S2** — S4 is cheap, broad, and low-risk; S2 is the highest
  *engineering* effort (its feasibility is now resolved — see [B2] — but it
  needs the S2a turbovec API work first).
- **S2 is the keystone; [B2] feasibility now RESOLVED** (spike 2026-10-16) —
  it is the only fix for the memory-per-backend wall. It is **feasible only
  via turbovec API additions** (S2a: `serialize_prepared` + a borrowed-view
  `search_from_prepared`), which is additive fork work we own; the
  small-structures fallback is a **dead end** (saves ~0) and has been dropped,
  so S2 has a single route. S3/S5 still depend on S2b (the shared image)
  landing. The remaining S2 risk is engineering effort + the cross-backend
  lifecycle/NUMA items, not whether it can be built.
- **S3 then S5** turn the 10M and write-heavy stories around; both depend on
  S2 (or its fallback). **S5's [B1] blocker is RESOLVED** (stays in
  GenericXLog; the lever is emitting each page's image once + delta-logging
  partial rewrites, not `log_newpage`), so S5 is no longer design-only on that
  axis.
- **S6 is cleanup** gated on profiles, not speculation.
- **S7 (sparse bitmaps) is independent** of the S2 keystone chain and can run
  in parallel with any phase; sequence it by appetite, not dependency. Its
  tombstone half is a wire-format change (additive-decode), so it carries the
  same corruption-test + soak bar as S2b/S3.

## 6. Non-negotiables threaded through every phase

- **No corruption, ever; old formats stay readable; wire bumps are additive
  via a new `kind` byte where possible** (AGENTS.md HARD MANDATE + the v4→v8
  precedent). An offline in-place converter if a break is truly unavoidable;
  never a corpus rebuild.
- **Every logic change ships with a test that fails before / passes after**,
  and persist/format changes get sustained-load validation on the pooled rig,
  not just `turbovec_check`.
- **Bit-identical scoring across patches** stays true; candidate-count and
  thread-count changes are minors, scoring-bit changes are not.
- **Every claim measured on the three-tier rig** per `benches/METHODOLOGY.md`;
  single-host numbers are diagnostics.

## 7. Open questions to resolve before committing engineering

- **[B2 — RESOLVED]** Spike (2026-10-16, `.agent/notes/S2_SPIKE_FINDINGS.md`)
  proved turbovec serialized bytes are position-independent and
  foreign-read-only-mappable, but the decoded index has no borrowed-view
  constructor — so S2 needs turbovec API additions (`serialize_prepared` +
  `search_from_prepared`, phase S2a), which is additive fork work we own. pgrx
  exposes the full DSA/DSM/shmem surface. The small-structures fallback saves
  ~0 and is dropped. No open question remains on *whether* S2 is buildable;
  the open items are the lifecycle/NUMA ones below.
- **[B1] — RESOLVED** (investigation, 2026-10-16): `log_newpage` is a dead end
  for the WAL goal (it emits the identical FPI bytes), so S5 stays 100%
  GenericXLog and §11a is unchanged; the real lever is fewer full-page images
  (write each build page once + delta-log partial rewrites). See
  `.agent/notes/B1_log_newpage_investigation.md`.
- DSA vs a fixed shared segment for the decoded image; cross-backend eviction
  policy for a shared cache (S2) when multiple large indexes exceed the
  segment — load-bearing for the many-large-indexes target market.
- NUMA placement of the shared segment on a multi-socket metal host
  (first-touch / interleave / per-node replication), and the ARC +
  `shared_buffers` + shared-segment triple-caching memory budget.
- Whether a posting-list IVF (S3 long horizon) is a new `kind` + offline
  converter (most likely) vs an additive `kind`-byte add (unlikely, since it
  reorganises physical page layout).
- The `assign_dups > 1` soft-assignment read-only case (AGENTS.md): a shared
  image must not worsen or mask the non-bijective `slot_to_id` load failure.
- Whether S1's "collapse to 1 thread under load" should be automatic (needs a
  cheap concurrency signal) or operator-set.
- Reconciling S6/CBOR with the `design/raw-vector-varlena` renumbering.