# pg_fts roadmap

The single plan file. Open work first, in priority order, each with its status and the
evidence behind it. Closed items are one line each; the full record of how they were
closed (and the wrong turns on the way) is in `bench/ROADMAP_HISTORY_2026-09-17.md` and the
files `bench/INDEX.md` lists.

Rules for this file: an item is either **open**, **blocked**, or **closed**. When it closes,
collapse it to one line here and leave the detail in the CHANGELOG. Do not create a second
plan file.

---

## Open -- repository and presentation (from `REVIEW_2026-09-17.md`)

These do not touch the index and are the cheapest, highest-leverage work in the project.

| # | item | status |
|---|---|---|
| R1 | **README comparison paragraph contradicted the project's own table** (claimed a rare-term lead; called the doclen sidecar future work). Rewritten as a wins / loses / caveat structure; every figure grep-verified against `BENCHMARK_SUMMARY.md` and `RESULTS_C1X`. The paragraph now states it is derived from the summary and that the summary wins on disagreement. Rule 27 in `AGENTS.md` and `RELEASING.md` step 1 keep it in sync. | **done 2026-09-17** |
| R2 | **Delete the 31 dead base SQL scripts** in the root. Only `pg_fts--<current>.sql` is installed; the rest were snapshots left by each rename. Verified none referenced by Makefile/meson/flake before `git rm`. `RELEASING.md` step 1 now says `git mv`, not copy. | **done 2026-09-17** |
| R3 | **`bench/INDEX.md`** naming current-truth vs dated-record files. | **done 2026-09-17** |
| R4 | **One plan file.** `HANDOFF`/`DEFERRED`/`CAPABILITIES` folded: DEFERRED (all resolved) and the old ROADMAP moved to `bench/` as history; CAPABILITIES moved to `doc/` (it is user-facing Q&A, not a plan). | **done 2026-09-17** |
| R5 | **C comments no longer cite `bench/` files** (9 sites -> 0). They cite the CHANGELOG release that closed the issue; the journal can be reorganised, the CHANGELOG cannot. | **done 2026-09-17** |
| R6 | **`t/010` in all CI matrices.** Added to GitHub and Forgejo; Forgejo's TAP set was also widened from 2 tests to the same 7 the nix gate runs. `RELEASING.md` step 3 now requires all three places. **Unverified until the next push runs CI on the Forgejo container** -- the wider set may hit the harness limits its comment describes. | **done, CI run pending** |
| R7 | **Agent tooling out of the working tree's face.** `.agent/`, `.claude/`, `.kiro/`, `.mcp.json`, `.agent-steering-domains.md` are gitignored but visible. `AGENTS.md` is now tracked and is the one entry point; the rest stay local. Also found and fixed while doing this: `result-1`, a nix build-output symlink, was **tracked** (now `git rm --cached`, `result*` ignored). | **done 2026-09-17** |

## Code quality (from `REVIEW_2026-09-17.md`) -- all three resolved 2026-09-17

| # | item | outcome |
|---|---|---|
| C1 | **Allocator state passed explicitly.** The four file-scope globals (`bm25_lowfree`, `_n`, `_i`, `bm25_alloc_extend_only`) became one `BM25AllocCtx` struct reachable only through `bm25_alloc_scope_enter(index, mode)` / `bm25_alloc_scope_exit(prev)`, which nest by returning the previous context -- the hand-rolled save/restore at two sites became the mechanism. `bm25_new_buffer()` now **`elog(ERROR)`s** if it finds compaction state with no scope active: exactly the dangling-pointer failure from the 1.7.1 work that only `t/007` caught. Deliberately an `elog`, not an `Assert`: the release gate is not a cassert build, and a check that only fires in a build nobody ships is documentation. Threading a struct through all 16 `bm25_new_buffer` callers was considered and rejected -- compaction is single-writer, so the context is backend-scoped in effect either way, and the scoped-lifetime design makes the failure mode a hard error at the same cost as a comment. | **done** |
| C2 | **`bm25_collect_matches` split: 412 -> 226 lines.** The 176-line per-segment loop body became `bm25_collect_segment()` (153 lines, returns `SEG_RESTART` for the positional-phrase fallback the loop used to express as `s = -1; continue`), and the 71-line pending-list walk became `bm25_collect_pending()`. State shared with the extracted evaluator travels in a `BM25CollectCtx`. Behaviour-preserving; full gate green. While doing it, the 14 `page + pd_lower` reads in `_scan.c` and the one in `pg_fts_trgm_index.c`'s blob reader (a **9th instance** of the 1.7.0 defect class, feeding a `memcpy` length) were routed through `bm25_page_data_end()`. | **done** |
| C3 | **The 12,000-line translation unit: KEEP, and document why.** Measured the cost of splitting: 15 `am.c` statics would go extern (13 for `_scan`, 5 for `_trgm`), 1 the other way, ~10 shared struct types would move into `pg_fts_am.h` (the on-disk-format header), and the hot-path `static inline` helpers (`bm25_tid_to_docid`, `bm25_docid_to_tid`, `bm25_page_data_end`, `bm25_doclen_cursor_lookup` -- inside the 45%/37% profile) would stop inlining without LTO, which PGXS does not use. In return: three `.o` files and no behaviour change. Decision recorded at the `#include` site and in both included files' headers, with the two prerequisites (an internal header; a before/after latency measurement) if separate compilation is ever needed. | **decided: keep** |

## Open -- index behaviour

| # | item | status | evidence |
|---|---|---|---|
| I1 | **Bulk-ingest write amplification.** At 1,660 terms/doc every document mints a one-doc segment; the insert-time merge rewrites a run per document. Freed pages cannot be reused in the inserting transaction because the recyclability XID gate (correctly) rejects them. 1.7.2 gated the merge on segment pressure: **-31%** (31,537 -> 21,874 MB over six 5k batches). Still **~3.7 GB per 5,000 docs** until `fts_vacuum` (which recovers ~210x in seconds). **Real fix is a design change: merge outside the inserting transaction** so freed pages pass the gate. Not a point release. | **open, mitigated** | `bench/RESULTS_KNOWN_ISSUES_2026-09-14.md` |
| I2 | **Common-term ranked latency -- the competitive gap.** `year` (df 734,896) top-10: **36.16 ms** vs pg_search 2.12, vchord 3.49, pg_textsearch 20.71; 20.7x under load. Profile: 45% doclen path, 37% candidate iteration -- per-posting scalar work. Only **item D** below can close it. | **open, architectural** | `bench/NOTE_PROFILE_COMMON_TERM_2026-09-06.md`, `RESULTS_C1X_CROSSENGINE_2026-09-11.md` |
| D | **Two-level page bitmaps + SIMD** (TIN-style). Format side is tractable via the 1.5.0 optional-per-segment-pointer + dual-read precedent (**no REINDEX**). Real cost: **no SIMD infrastructure exists** (no intrinsics, no runtime dispatch, no `-mavx2` plumbing) and a scalar fallback must be kept for non-AVX and ARM -- two implementations forever. Largest change the project has attempted, against a competitor that cannot be benchmarked. **Needs explicit sign-off.** Do **not** vectorize the vendored sparsemap. | **blocked on sign-off** | `bench/NOTE_TIN_FEASIBILITY_2026-09-14.md`, `NOTE_SIMD_VENUE_2026-09-14.md` |
| I3 | **Managed-service validation** on a compute/storage-separated backend (Aurora-style). GenericXLog-only WAL should be safe; unverified externally. | **open, external** | `doc/CAPABILITIES.md` |
| I4 | **Independent human review of WAL/crash/recovery paths.** Checklist exists in `RELEASING.md`; the review itself is a release-integrator step. | **open, external** | |

## Open -- measurement debt

| # | item |
|---|---|
| M1 | **C2 cross-engine ingest.** Rivals' ingest paths differ fundamentally; needs per-engine forms chosen as carefully as C1X's. |
| M2 | **C3 NDCG vs rivals.** Matters because pg_search (Tantivy) does not stem -- its speed is partly a smaller unit of work. `bench/ndcg.py` exists. |
| M3 | **Longer ingest run** to find where the pg_fts ingest decay (41% over 200k rows) levels off. |
| M4 | **The published competitor set on TIN's exact rig** (i7i.8xlarge, 8 vCPU / 32 GB container, Stack Exchange corpus) plus pg_fts -- places us on their axis without asserting anything about TIN. Several EC2 hours. |

## Declined (with the measurement that declined them)

- **Impact-ordered postings** -- breaks the docid ordering that `count(*)`/AND/phrase/prefix need.
- **Early termination** -- breaks exact top-k.
- **Lazy phrase gate** -- ~1.5x ceiling; adjacency is only 4.3% of the query.
- **Heap-side `positions=off`** -- saves ~16% of a `STORAGE=extended` column; no-go.
- **Parallel ranked scan** -- built, measured, reverted (`bench/NOTE_PARALLEL_RANKED.md`).
- **Parallel merge** -- 1.45x slower and 19% larger at scale; `mpmw=8` silently serial.
- **df-threshold bulk load** -- cost ~linear in df, no fixed floor.
- **Verbatim posting copy on merge** (TIN item C) -- the merge re-encodes through the build hash table; no splice point.
- **Vectorizing sparsemap** -- never a query hotspot; the one time it was the bottleneck (P0, 99.75%) the fix was algorithmic; its compressed layout is SIMD-hostile; it is vendored byte-identical to upstream on purpose.

## Closed (one line each; detail in CHANGELOG)

- **1.8.1** count-path: df fast-count gate tests (10, non-vacuous); block-run VM checking measured ~1%, kept as cleanup.
- **1.8.0** intra-word `-` `.` `/` are terms, not operators (`pkg-config` no longer parses as `pkg & !config`).
- **1.7.2** insert-time merge gated on segment pressure: bulk-ingest growth -31%.
- **1.7.1** `pd_lower` guard generalised to all 8 page-read sites; "one WAL record per page" known issue **retracted** (0.005 ms/page measured).
- **1.7.0** P0: unvalidated `pd_lower` in the merge dict walk made an index permanently unvacuumable at field shape; fixed. Huge-alloc gaps in doclen/tombstone arrays fixed. Cleanup no longer grows the index (18/18/18 MB vs 35/52/69).
- **1.6.1** P0: VACUUM never completed on a delete-heavy index (4h39m -> 393 s; dense tombstone bitmap sized by `sm_maximum`). sparsemap 5.5.1.
- **1.6.0** phrase over positionless docs returns `false`, not a silent conjunction (matches `OP_PHRASE`).
- **1.5.9 / 1.5.10** non-UTF-8 case folding; common-term 1.56x via ascending-resume + word-load `bm25_for_get`.
- **1.5.0** doclen sidecar (format v3 -> v4) with dual-read, **no REINDEX** -- the precedent for all future format changes.
- **COUNT pushdown** (CustomScan), **`fts_search` under-fetch**, **reserved keywords as literals**, **sparsemap error-path leaks**, **recovery guard on `fts_merge`/`fts_vacuum`**, **privilege lockdown**, **recently-dead exclusion from corpus stats**, **parallel-build memory ceiling** -- all shipped; see history file.