# SVS Vamana Index Integration Architecture for pgvector ## Table of Contents 1. [Executive Summary](#1-executive-summary) 2. [Background](#2-background) 3. [Architecture Overview](#3-architecture-overview) 4. [Detailed Design](#4-detailed-design) 5. [ACID Compliance](#5-acid-compliance) 6. [GUC Parameters](#6-guc-parameters) 7. [Integration Points](#7-integration-points) 8. [Performance Considerations](#8-performance-considerations) 9. [Security Objectives](#9-security-objectives) --- ## 1. Executive Summary This document describes the `svs` PostgreSQL extension, which adds a Vamana index access method backed by Intel's Scalable Vector Search (SVS) library. The extension is a standalone PostgreSQL extension that depends on pgvector (`vector` extension) for its vector types and operator classes; it does not bundle pgvector source. --- ## 2. Background ### 2.1 Intel SVS Library - **Repository:** https://github.com/intel/ScalableVectorSearch - **License:** Mixed — see below - **Key Features:** - Vamana graph-based index (similar to HNSW) - Locally-adaptive Vector Quantization (LVQ) compression - LeanVec two-level quantization compression - Optimized for Intel Xeon processors (2nd gen and newer) - Support for float32, float16 - Distance metrics: Euclidean, Inner Product, Cosine similarity - Billion-scale performance **Licensing details:** | Component | License | |---|---| | Core Vamana index (open-source SVS) | Apache 2.0 | | LVQ and LeanVec compression | Intel Simplified Software License (binary-only, proprietary) | LVQ and LeanVec are **not open-source**. They are distributed in binary form only and run only on Intel CPUs. The Intel Simplified Software License prohibits reverse engineering, decompilation, modification, or disassembly. See `ScalableVectorSearch/LICENSE` and `docs/non-open-source-notice.rst` in the SVS repository for the full terms. ### 2.2 Vamana Algorithm - Graph-based approximate nearest neighbor search - Similar to HNSW but with a single-layer graph and different construction strategy - Uses greedy search with pruning strategies - Research paper: [DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node](https://www.microsoft.com/en-us/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/) (Microsoft Research); [GitHub](https://github.com/microsoft/DiskANN) ### 2.3 Existing pgvector Index Implementations - **HNSW:** Graph-based, supports parallel build, in-memory + on-disk phases - **IVFFlat:** Clustering-based, k-means partitioning, probe-based search --- ## 3. Architecture Overview ### 3.1 High-Level Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ PostgreSQL Core │ ├─────────────────────────────────────────────────────────────┤ │ pgvector Extension (vector/halfvec/sparsevec types, │ │ HNSW index, IVFFlat index) [separate extension] │ ├─────────────────────────────────────────────────────────────┤ │ svs Extension (requires = 'vector') │ ├─────────────────────────────────────────────────────────────┤ │ Vamana Index │ │ vamana.c / vamanabuild.c / vamanainsert.c │ │ vamanascan.c / vamanavacuum.c / vamanautils.c │ │ vamanaworker.c / vamana_undo.c / vamanalauncher.c │ │ vamana_databases.c / vamana_teardown.c / vamana_warmup.c │ │ vamana_subxact_guard.c / vamana_subxid_pending_array.c │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌──────────────────────────────┐ │ Intel SVS C API Wrapper │ │ svs_wrapper.c/h │ ├──────────────────────────────┤ │ - Index Build │ │ - Index Search (batch) │ │ - Dynamic insert/delete │ │ - Compression (LVQ/LeanVec) │ └──────────────────────────────┘ │ ▼ ┌──────────────────────────────┐ │ Intel SVS Shared Library │ │ libsvs_c_api.so │ └──────────────────────────────┘ ``` ### 3.2 Component Structure **Core Implementation Areas:** The extension entry point initializes the access method and registers GUCs, the object-access hook, and a background worker *launcher* at server start. The launcher supervises one background worker per enabled database; see [BGW_LAUNCHER.md](BGW_LAUNCHER.md) for the full launcher and per-database worker model. Throughout this document, "the BGW" or "the worker" refers to the per-database worker serving the database an operation runs in. The index access method handler implements the PostgreSQL AM interface and dispatches to the appropriate subsystem for each operation. The build subsystem performs a full heap scan, buffers all vectors in memory, calls the SVS dynamic build API to construct the mutable graph, serializes it to disk, and submits a synchronous LOAD request to the BGW before returning; the BGW creates the replication slot while handling that request. The insert subsystem routes each incoming vector through the BGW via IPC, where it is added incrementally to the live graph and recorded in the per-transaction undo log. The scan subsystem executes ANN queries by submitting search requests to the BGW and mapping returned SVS indices back to heap TIDs via the TID mapping. The vacuum subsystem collects dead TIDs from the heap callback and submits batched delete, consolidate, and compact requests to the BGW. The BGW subsystem owns the in-memory SVS index state for all indexes in its database, processes IPC slots from backends (search, insert, delete, maintenance), manages the replication slot, and performs periodic checkpoints. Indexes are loaded on demand (on first access), not eagerly at worker startup. The undo log subsystem records external IDs inserted within each transaction and submits compensating BGW deletes on transaction abort. The SVS wrapper layer provides a thin, PostgreSQL-friendly interface over the Intel SVS C API for all graph operations. **Test Coverage:** Regression tests cover basic CRUD operations, all distance metrics, all compression types, GUC boundary values, incremental INSERT and DELETE + VACUUM cycles, and REINDEX. TAP tests cover index persistence across restarts, crash recovery, background worker lifecycle, batch search correctness, concurrent query isolation, and LWLock slot leak detection. --- ## 4. Detailed Design ### 4.1 Design Choice: SVS C API Batch Build SVS provides a high-level batch build API with internal parallelism management. **Implementation characteristics:** - **SVS manages parallelism internally** via threads - **No manual shared memory management** required for build - **Build and search thread counts are both launcher-governed, through the same grant calculator:** a build backend requests a thread count (`maintenance_num_threads` from `vamana_databases`, falling back to `max_parallel_maintenance_workers` when unset) and blocks until the launcher grants it from the database's CPU budget; search reads its last-published grant (derived from `svs.search_num_threads` and its per-database/cluster ceilings) non-blockingly instead of waiting on one - **Dynamic index:** Uses `svs_index_build_dynamic` to produce a mutable index; INSERT calls `SVSAddPoints` for incremental updates, DELETE + VACUUM calls `SVSDeletePoints` with consolidation, and with compaction when the deleted fraction exceeds `svs.compact_threshold_pct` - **Must buffer vectors:** All vectors loaded into memory before build ### 4.2 On-Disk Serialization After the SVS batch build completes, the index is serialized to a directory on the filesystem using `VamanaGetIndexSavePath()`. The path is keyed by the database OID and the index's relation OID. The metapage stores `hasSavedIndex`, `indexDataBlkno`, and `indexDataSize` to track serialization state. When the BGW handles the LOAD request that follows serialization, it creates a logical replication slot (`vamana__`) with `confirmed_flush_lsn` set to the current WAL flush position. The BGW creates the slot because slot creation cannot run inside the backend's write transaction. This anchors the slot LSN to the exact point where the on-disk state is complete, ensuring crash recovery replays only changes committed after that point. Indexes are loaded on demand: on the first access to an index, `LoadIndexFromPages()` reads the saved index from disk. If no saved copy exists, `VamanaRebuildFromTable()` performs an in-memory rebuild from the heap. (The worker does not eagerly load every index at startup; see [BGW_LAUNCHER.md](BGW_LAUNCHER.md).) Loading also restores the TID map sidecar file (`tidmap.bin`, one `ItemPointerData` per allocated slot, written atomically via temp-file + rename) and the metapage's dynamic-index fields (`tidMappingCapacity`, `nextExternalId`, `numDeleted`) into the in-memory cache. `hasSavedIndex` and the dynamic fields are flipped to their final values together, in a single WAL-logged transaction, only after the on-disk files are durable. An object-access hook (`VamanaInstallObjectAccessHook`) cleans up the on-disk save directory when an index or its parent table is dropped. ### 4.3 Insert and Vacuum Behavior **Inserts:** `vamanainsert` calls `VamanaWorkerWaitUntilAvailable` before submitting, then routes the insert through the background worker (slot kind `VAMANA_SLOTKIND_INSERT`). The worker calls `SVSAddPoints` to add the new vector incrementally to the live in-memory index. External IDs are allocated from a counter held in the BGW's in-memory cache (`cache->nextExternalId`); the latest value is written back to the metapage by `VamanaWriteMetaPageDynamic` so the counter survives restart. The inserting backend records the new external ID in the per-transaction undo log (`VamanaUndoAppend`); on transaction ABORT, the registered XactCallback submits a BGW DELETE for each logged ID to roll back the in-memory graph state. The worker loads indexes on demand, not eagerly at startup, so the first insert against an index it hasn't cached yet (e.g. immediately after a server restart) triggers a synchronous load via `VamanaWorkerGetOrLoadIndex`: try `LoadIndexFromPages`, falling back to `VamanaRebuildFromTable` if no saved copy exists. Subsequent inserts find the index already cached and take the fast path. **Vacuum:** `vamanabulkdelete` iterates the TID mapping, calls PostgreSQL's dead-tuple callback for each live entry, and batches dead IDs to the BGW via `VamanaWorkerSubmitDelete`. `vamanavacuumcleanup` submits a `CONSOLIDATE` request to the BGW via `VamanaWorkerSubmitMaintenance` when there are deleted entries, and a `COMPACT` request only when the deleted fraction exceeds `svs.compact_threshold_pct`. The BGW calls `SVSDeletePoints`, `SVSConsolidate`, and `SVSCompact` respectively, and updates the metapage counters atomically after each operation. **Known Limitation:** `PREPARE TRANSACTION` (two-phase commit / 2PC) is not supported on any transaction that has inserted into a Vamana index. The undo log implementation raises `ERRCODE_FEATURE_NOT_SUPPORTED` at prepare time. This is a fundamental constraint of the in-memory undo log design: prepared state cannot be persisted to disk and resumed by a later `COMMIT PREPARED`. Users with `max_prepared_transactions > 0` or distributed 2PC coordinators (e.g., Citus, XA drivers) must avoid issuing `PREPARE TRANSACTION` on transactions that touch Vamana indexes. ### 4.4 Background Worker Vamana index state lives in background worker processes, one per enabled database, supervised by a launcher registered at server start. The launcher, per-database worker lifecycle, enablement catalog, crash-restart/backoff policy, per-database shared-memory slot layout, demand-driven loading, and the drain-and-stop shutdown path are documented separately in **[BGW_LAUNCHER.md](BGW_LAUNCHER.md)**. This section covers only the worker's data-plane behavior — how it serves backend requests once running. **Request serving:** - The per-database shared-memory region contains a per-backend slot array plus variable-length areas for query vectors, result TIDs, and distances - Each slot has a kind (`VAMANA_SLOTKIND_SEARCH`, `_INSERT`, `_DELETE`, `_MAINTENANCE`, `_LOAD`, `_WARMUP`); backends set the kind and data, set status `PENDING`, and wait on a shared latch - The worker drains all pending slots each cycle: SEARCH slots are batched and dispatched to `SVSSearch`; write slots (`INSERT`, `DELETE`, `MAINTENANCE`) are dispatched one at a time via `VamanaWorkerProcessWriteSlot` - If the worker is unavailable or startup times out (controlled by `svs.worker_startup_timeout_ms`), the backend throws an error rather than silently falling back - Per-index LWLocks (`VamanaIndexLockSlot`, up to `VAMANA_MAX_INDEXES = 64` live indexes) guard each live index's state. Writers acquire the lock `LW_EXCLUSIVE`; searches acquire it `LW_SHARED`. The worker main loop handles requests one at a time, so a write in progress delays every search in the database (see [8.4](#84-known-limitations)) - On restart, the worker loads each index's last checkpoint from disk on first access, opens the persisted replication slot, and replays all committed changes made after the last checkpoint before serving the first request against that index - Reload signaling: for edge cases such as TRUNCATE or REINDEX, backends write to `reloadRequests[]` to signal index invalidation; the worker reloads the affected index on its next cycle. Normal write-path state synchronization uses replication slot replay, not reload signals **Checkpoint and durability:** The BGW periodically checkpoints the in-memory SVS graph to disk using an atomic 5-phase sequence: write to temp files, fsync, rename, fsync directory, advance the replication slot. The slot advance is the final step — it marks the on-disk state as durable and bounds the WAL that PostgreSQL must retain. Between checkpoints, the BGW continuously applies committed changes from the slot to the in-memory graph. Checkpoint frequency is governed by the debounce GUCs (`svs.checkpoint_min_ops`, `svs.checkpoint_debounce_window`, `svs.checkpoint_max_interval`) or, when set, the simple triggers `svs.checkpoint_operations` and `svs.checkpoint_interval`. A deliberate stop (database pause or explicit restart) drains and checkpoints every cached index before exiting — see [BGW_LAUNCHER.md](BGW_LAUNCHER.md). See Section 5 for the full recovery and idempotency design. **Transaction safety:** Inserts are logged to a per-transaction undo log. On ABORT, an XactCallback/SubXactCallback submits BGW DELETE operations for each logged insert, rolling back the in-memory graph state. On COMMIT, the undo log is discarded. Workers require the extension to be loaded via `shared_preload_libraries = 'svs'` (which registers the launcher; requires a server restart) and the database to be enrolled in the `vamana_databases` catalog. See [BGW_LAUNCHER.md](BGW_LAUNCHER.md). ### 4.5 Supported Vector Types and Operator Classes Vamana indexes support `vector` (float32) and `halfvec` (float16) types. `sparsevec` and `bit` are not supported. The indexed column's type is resolved from the relation (`VamanaGetTypeInfo`, `src/vamanautils.c`), which yields both the element width to read a heap datum at and the SVS storage element type to build and load with — float32 for `vector`, float16 for `halfvec`. The SVS C API only accepts `float` arrays, so a `halfvec` datum is widened on the way in and stored back down to float16 by SVS; the storage type is therefore a separate choice from the datum width. Build and load must name the same storage type or the saved index will not reload, so every storage site takes it from this one source. Under `compression_type` 1 or 2 the compressed spec owns the stored format and the storage element type is not consulted. | Operator Class | Type | Distance Operator | Metric | |---|---|---|---| | `vector_l2_ops` | `vector` | `<->` | L2 (Euclidean) | | `vector_ip_ops` | `vector` | `<#>` | Inner product (negated) | | `vector_cosine_ops` | `vector` | `<=>` | Cosine | | `halfvec_l2_ops` | `halfvec` | `<->` | L2 (Euclidean) | | `halfvec_ip_ops` | `halfvec` | `<#>` | Inner product (negated) | | `halfvec_cosine_ops` | `halfvec` | `<=>` | Cosine | ### 4.6 Index Build Phases Two progress phases are reported via `pg_stat_progress_create_index`: 1. `PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE` — initializing build state 2. `PROGRESS_VAMANA_PHASE_LOAD` — scanning heap and accumulating vectors ### 4.7 Dynamic Index Internals Dynamic behavior is split across the same files listed in §3.1: `vamanabuild.c` builds via `SVSBuildDynamicIndex`; `vamanainsert.c`'s `vamanainsert` routes through `VamanaWorkerSubmitInsert`; `vamanavacuum.c`'s `vamanabulkdelete` and `vamanavacuumcleanup` route through `VamanaWorkerSubmitDelete` and `VamanaWorkerSubmitMaintenance`; `vamanautils.c` holds `VamanaRebuildFromTable` and `VamanaWriteMetaPageDynamic`; and `vamanascan.c`'s `LoadIndexFromPages` calls `SVSLoadDynamicIndex`, with `SVSSearch` skipping any slot marked `InvalidItemPointer` (soft-delete). **Metapage fields.** In addition to the base fields, the metapage (`VamanaMetaPageData`) carries three fields that track the mutable index: `nextExternalId`, `numDeleted`, and `tidMappingCapacity`. - `nextExternalId` — SVS external IDs must be unique for the life of the index. While an index is loaded, the BGW's cached counter is authoritative: the BGW passes it to `SVSAddPoints` under the exclusive per-index LWLock and increments it. The BGW writes it to the metapage after writes, and reloads it from the metapage when the index is loaded. - `numDeleted` — drives the `SVSCompact` decision threshold (`svs.compact_threshold_pct`) and reports deletion pressure. - `tidMappingCapacity` — after deletes, the TID mapping array has holes. `numVectors` tracks live entries; `tidMappingCapacity` tracks the allocated length, which is what the sidecar `tidmap.bin` file serializes. Search bounds-checks against the capacity, then filters out `InvalidItemPointer` slots. **In-memory cache.** The BGW's per-process cache (`VamanaIndexCache`) mirrors the metapage and adds runtime-only fields (`svsIndex`, `isValid`, `memCtx`, `tidMapping`, `needsSave`). While the index is loaded, the cached `nextExternalId` is the counter the BGW uses to assign IDs, under the exclusive per-index LWLock; the BGW persists it to the metapage after each write and reloads it from the metapage when the index is loaded. ### 4.8 Read Path and Soft-Delete Semantics `SVSSearch` is the query-time hot path; two mechanisms apply to reads: - **Bounds check widened to `tidMappingCapacity`.** `tidMappingCapacity` can exceed `numVectors` because deleted slots stay allocated (as holes) until compaction, so SVS may return indices up to the full capacity. The search loop in `svs_wrapper.c` bounds-checks against `cachedIndex->tidMappingCapacity` rather than `numVectors`. - **Soft-delete filtering.** The search loop skips any slot whose TID is `InvalidItemPointer`. This is the read-side half of the bulk-delete protocol (§5.2): between VACUUM's write-lock release and the next `SVSConsolidate`, SVS may still surface deleted vectors as candidates, and this filter ensures they never reach the client. Because the filter can drop results, the loop tracks a separate "surviving results" counter rather than reusing the raw SVS result index — the returned result count may be less than the requested `k`, matching the existing pgvector contract for sparse result sets. --- ## 5. ACID Compliance The Vamana index stores its graph structure in a background worker (BGW) process using the Intel SVS library, outside PostgreSQL's normal buffer manager. This out-of-process design enables high-performance ANN search but requires explicit machinery to maintain the ACID guarantees that PostgreSQL's heap and WAL provide automatically for ordinary indexes. | Property | Mechanism | |---|---| | **Atomicity** | Per-transaction backend undo log for in-flight transactions; WAL slot filtering as safety net | | **Consistency** | VACUUM as authoritative dead-entry scanner; idempotent replay prevents divergence | | **Isolation** | Delegated to PostgreSQL's MVCC layer — dead or aborted heap TIDs are filtered by the heap fetch's MVCC visibility check before results are returned, so stale graph entries never surface to queries | | **Durability** | Logical replication slot + atomic checkpoint sequence; crash recovery replays committed changes from the slot | ### 5.1 Hybrid Design — Synchronous Writes + WAL Recovery **Design Principle: Synchronous operations for visibility. WAL for durability and recovery.** | Concern | Mechanism | |---|---| | INSERT visibility | Backend submits to BGW via IPC; BGW calls `SVSAddPoints`; backend blocks until ACK | | DELETE visibility | VACUUM submits dead TIDs to BGW via IPC; BGW calls `SVSDeletePoints` | | Atomicity (abort) | Backend undo log: on abort, the backend asks the BGW to run `SVSDeletePoints` for all externalIds added in that transaction | | Durability (crash) | Replication slot: on restart, replay committed ops from last checkpoint LSN | | Consistency (standby) | Replication slot: standby BGW replays from slot incrementally (no IPC available) | ``` PRIMARY STANDBY ─────── ─────── Backend BGW BGW ─────── ─── ─── INSERT ──IPC──► SVSAddPoints (immediate) (no IPC: read-only) COMMIT ──WAL──► [WAL record durable] WAL replay ──► SVSAddPoints │ ▲ ▼ │ Checkpoint: Replication slot: save graph + advance slot read committed changes │ from slot position ▼ Crash restart: load last checkpoint replay from slot LSN ``` ### 5.2 Write Path (Primary) #### INSERT ``` Backend BGW ─────── ─── BEGIN INSERT INTO t (val) ... → heap_insert (WAL record written) → vamanainsert(): submit (vector, heapTID) to BGW via IPC block until ACK ──► SVSAddPoints(externalId, vector) tidMapping[externalId] = heapTID ◄── ACK ────────────────────────── record (indexRelid, externalId) in the backend undo log → return to executor COMMIT → VamanaXactCallback(COMMIT): discard the backend undo log ``` #### INSERT + ABORT ``` Backend BGW ─────── ─── BEGIN INSERT INTO t (val) ... → vamanainsert(): submit to BGW via IPC block until ACK ──► SVSAddPoints(externalId, vector) ◄── ACK ────────────────────────── record (indexRelid, externalId) in the backend undo log ROLLBACK → VamanaXactCallback(ABORT): submit undo entries to BGW ──► SVSDeletePoints(externalId) invalidate tidMapping[externalId] submit CONSOLIDATE (and COMPACT if needed) for each affected index ``` On `ROLLBACK TO SAVEPOINT`, only the entries recorded by the aborting subtransaction are undone. `PREPARE TRANSACTION` is rejected for a transaction that has inserted into a Vamana index. The undo log is bounded by `svs.undo_log_max_entries`. #### DELETE (via VACUUM) ``` VACUUM BGW ────── ─── vamanabulkdelete(): load the TID map from tidmap.bin; call the dead-tuple callback for each valid entry submit dead TIDs to BGW via IPC block until ACK ──► SVSDeletePoints(externalIds...) invalidate tidMapping entries ◄── ACK ────────────────────────── vamanavacuumcleanup(): submit CONSOLIDATE; COMPACT only past svs.compact_threshold_pct ──► SVSConsolidate() / SVSCompact() ``` If the BGW is unavailable, or a delete batch fails, VACUUM logs a WARNING and skips removal. The dead vectors stay in the graph, hidden by MVCC, until a later VACUUM removes them. ### 5.3 Recovery Path (Crash Restart) ``` BGW starts after crash: 1. Load last checkpoint from disk (SVS graph + tidMapping) 2. Open replication slot (persisted by PostgreSQL across crashes) 3. Slot's confirmed_flush_lsn = LSN of last checkpoint (restart_lsn is at or before it) 4. Replay all committed changes from restart_lsn to current WAL position: - INSERT records → SVSAddPoints (idempotency check: skip if TID exists) - DELETE records → SVSDeletePoints (idempotency check: skip if TID absent) 5. Resume normal IPC operation ``` This runs per index, when the index is loaded from disk. If the slot is invalidated, over `svs.max_slot_wal_size`, or cannot be decoded, the BGW rebuilds the index from the heap instead. The reorder buffer guarantees only committed transactions are delivered. Aborted transactions are automatically filtered — even if the BGW was down when the abort happened, the aborted insert never appears in the slot replay. ### 5.4 Standby Path On a standby, there is no DML and no IPC. The BGW runs in replay-only mode: ``` Standby BGW: 1. Create/open replication slot at replay LSN 2. Every worker loop iteration (latch wake or 1 s heartbeat): replay committed changes from slot SVSAddPoints / SVSDeletePoints 3. Staleness bounded by the loop interval (about 1 s) ``` This is identical to the primary's crash-recovery catch-up path, selected by the replay role (`VamanaGetReplayRole()`). The standby needs `hot_standby_feedback = on`, which pins the primary's `catalog_xmin` so the catalog rows the standby's logical slot reads are not vacuumed. WAL retention comes from `primary_slot_name`. ### 5.5 Undo Mechanism #### Primary defense: IPC undo On abort, `VamanaXactCallback(ABORT)` submits undo entries to BGW. BGW calls `SVSDeletePoints` for each externalId inserted by the aborting transaction. This provides immediate cleanup when BGW is available. #### Safety net: WAL slot filtering If BGW is down at abort time, the undo is skipped. On BGW restart, the slot replay only delivers committed transactions (reorder buffer filtering). The aborted INSERT never appears in the replay stream. The stale externalId in the SVS graph points to a heap TID whose xmin is aborted — MVCC visibility filtering prevents it from appearing in query results. The next VACUUM cycle detects the dead TID and submits the delete via IPC. This two-layer approach provides immediate undo when possible and guaranteed eventual correctness always. #### Crash before undo: worked examples **Case 1: 3 inserts, crash before rollback (no checkpoint between)** ``` 1. BEGIN 2. INSERT v1, v2, v3 → BGW applies SVSAddPoints for each (undo log has 3 entries) 3. CRASH (before ROLLBACK or COMMIT) Recovery: - BGW loads last checkpoint (does NOT contain v1, v2, v3) - Replays slot from checkpoint LSN forward - Reorder buffer never delivers the uncommitted transaction - v1, v2, v3 are simply not in the recovered graph - Result: correct, no cleanup needed ``` **Case 2: 3 inserts, checkpoint occurs, then crash before rollback** ``` 1. BEGIN 2. INSERT v1, v2, v3 → BGW applies SVSAddPoints (undo log has 3 entries) 3. Checkpoint fires → graph saved to disk WITH v1, v2, v3; slot advanced 4. CRASH (before ROLLBACK or COMMIT) Recovery: - BGW loads checkpoint (CONTAINS v1, v2, v3) - Replays slot from checkpoint LSN forward - Reorder buffer never delivers the uncommitted transaction - No replay command arrives to add or remove v1, v2, v3 - v1, v2, v3 remain in the graph BUT their heap TIDs have aborted xmin - MVCC visibility checks on the heap fetch hide them from all query results - Next VACUUM discovers the dead TIDs and submits SVSDeletePoints via IPC - Result: invisible to users immediately, cleaned from graph at next VACUUM ``` ### 5.6 VACUUM's Role VACUUM is the authoritative delete-discovery mechanism, scanning the TID map for dead TIDs and submitting them to the BGW: ```c vamanabulkdelete(): for each valid TID in the TID map: if callback(TID) says "dead": collect its externalId submit the collected externalIds to the BGW in batches (SVSDeletePoints) return stats vamanavacuumcleanup(): if numDeleted > 0: submit CONSOLIDATE to BGW if numDeleted exceeds svs.compact_threshold_pct of numVectors: submit COMPACT to BGW ``` The replication slot replays committed deletes on crash recovery. Any delete it does not cover is rediscovered by the next VACUUM. ### 5.7 CREATE INDEX Workflow #### Problem: Race Between CREATE INDEX and First INSERT The BGW keeps the SVS graph in its private process memory. After CREATE INDEX serializes the graph to disk, the BGW must load it before serving queries. The naive approach (signal BGW to reload asynchronously) introduces a race: 1. CREATE INDEX holds AccessExclusiveLock on the index relation 2. BGW receives reload signal, tries `ConditionalLockRelationOid`, fails (lock held) 3. CREATE INDEX commits, lock released 4. First INSERT arrives, BGW has not loaded the index yet 5. INSERT fails with "index not loaded for write" #### Solution: Deterministic Load via IPC The backend absorbs the load latency at build time by submitting a LOAD request to the BGW before returning from `vamanabuild`. The BGW loads from the save directory (no `index_open`, no lock conflict) and confirms via the IPC slot. The backend blocks until DONE. The BGW creates the replication slot while handling the LOAD request, because slot creation cannot run inside the backend's write transaction. This eliminates the lazy-creation gap and anchors the slot LSN to the point where the on-disk state is complete. #### Full Sequence ``` Backend (vamanabuild) BGW main loop ======================== ======================== 1. CreateMetaPage(index) - writes build params to block 0 2. table_index_build_scan - collect vectors + TIDs into memory 3. SVSBuildDynamicIndex - build graph in backend process memory 4. SerializeIndexToPages - SVSSaveIndex → $PGDATA/vamana_indexes/// - VamanaSaveTidMapAtomically - VamanaMarkIndexSaved (metapage flag) 5. SvsMemoryHandoffBuild (hand the build's memory accounting over to residency) 6. VamanaWorkerSubmitLoad(relid, heapRelid, ─── slot PENDING, latch set ───► build config, metapage counters, vectorAttNum) - packs metadata into IPC slot - slotKind = VAMANA_SLOTKIND_LOAD - blocks on slot latch (heartbeat-aware, waits up to svs.worker_startup_timeout_ms) 7. Wakes from WaitLatch 8. VamanaWorkerProcessRequests: finds LOAD slot, transitions to PROCESSING 9. VamanaWorkerProcessLoadSlot: - Reads SVS index from $PGDATA/vamana_indexes/// - VamanaCacheIndex(relid, heapRelid, dims, ..., tidMapping) - VamanaReplicationCreate(dboid, relid) (confirmed_flush = GetFlushRecPtr()) - VamanaReplicationOpen(dboid, relid) - cache->lastReplayLsn = GetFlushRecPtr() 10. slot status = DONE ◄── latch set on backend ──────────── 11. Backend wakes, sees DONE - SvsIndexResidencyRecordLoad - SVSFreeIndex (free backend-local graph copy) - log_newpage_range (WAL-log index pages) - returns IndexBuildResult ═══ PG core commits CREATE INDEX transaction ═══ ═══ AccessExclusiveLock released ════════════════ BGW cache is warm. First INSERT/SEARCH served immediately. ``` The BGW loads from the filesystem save directory (`$PGDATA/vamana_indexes///`), not from relation pages. The relid and build parameters are passed through the IPC slot. No catalog access or relation lock required. The backend's AccessExclusiveLock protects the BGW's load: no other process can interfere with the save directory while the backend holds the lock. #### Crash Scenarios | Crash point | State on recovery | |---|---| | During steps 1–5 (before the LOAD request) | Transaction uncommitted. PG rolls back catalog entry. An orphaned save directory is removed on DROP by the object-access hook, or overwritten by the next save. | | During steps 6–10 (IPC in flight) | Save directory complete. The slot may or may not exist yet, depending on whether the worker reached step 9. Backend's transaction is uncommitted. PG rolls back. If the build aborts without a crash, the slot is retired at transaction abort (`VamanaReplicationQueueRetireOnAbort`). | | After step 11, before PG COMMIT | PG rolls back catalog entry. Orphaned save directory + replication slot persist. A later `CREATE INDEX` that is assigned the same relid drops both (`VamanaReplicationCreate` try-drops a slot of the same name before creating one). Failing that, the primary has no reconciliation pass: the slot keeps pinning WAL and `catalog_xmin` cluster-wide until dropped with `pg_drop_replication_slot()`. | #### Worker-Unavailable Handling `CREATE INDEX` gates on the state of the calling database's worker slot: a missing slot (database not enrolled) is a hard error, a reserved slot with no live worker is a wait bounded by `svs.worker_startup_timeout_ms`, and a live worker proceeds. Once the index is built and its save directory is durable, the worker loads it on first access. See the reservation handshake and three-state gate in [BGW_LAUNCHER.md](BGW_LAUNCHER.md). The first INSERT blocks on `VamanaWorkerWaitUntilAvailable` until the worker has loaded the index and marked it available, guaranteeing the cache is warm before any write is applied. #### Build Thread Grant Before the build itself runs, the backend negotiates its build thread count with the launcher, a separate protocol from the load IPC above. The backend resolves a requested count (`vamana_databases.maintenance_num_threads` for the current database, falling back to `max_parallel_maintenance_workers` when unset), claims one of a small fixed number of pending-build-request slots, and writes its request into shared memory. It then wakes the launcher and blocks on its own latch. The launcher's CPU grant pass (see [BGW_LAUNCHER.md §6](BGW_LAUNCHER.md#6-launcher-behavior)) folds this request into the same calculation it uses for search-thread grants, clamped against that database's CPU budget and `max_parallel_workers`, and writes a grant back into the backend's request slot. The backend reads the grant, caps its `ParallelContext`'s `nworkers` to it, and proceeds. On completion (success, error, or abort) the backend clears its own request slot and wakes the launcher so the capacity is reclaimed immediately rather than on the launcher's next unrelated wake. If no grant arrives within `svs.worker_timeout_ms`, the backend fails the `CREATE INDEX` with an error naming the launcher, rather than falling back to an uncoordinated thread count. If the backend itself dies before clearing its request slot, the launcher's reconcile pass detects the stale request by PID liveness and reclaims the slot. ### 5.8 Checkpoint and Atomicity #### What Is a Checkpoint A checkpoint makes the in-memory SVS graph durable to disk and marks that point as "fully processed" in the replication slot. Between checkpoints, the BGW continuously applies committed changes from the slot to the in-memory graph. **Three states to keep consistent:** 1. In-memory SVS graph (volatile, lost on crash) 2. On-disk SVS graph (durable, updated at checkpoint) 3. Slot LSN position (durable, marks "what's on disk") #### Checkpoint Triggers (Debounce Design) The default (debounce) policy is: ``` Rule 1 (burst completion): ops >= min_ops AND quiet_for >= debounce_window Rule 2 (safety timeout): ops >= min_ops AND time >= max_interval should_checkpoint = Rule 1 OR Rule 2 ``` Below `min_ops`, an index that has had any writes is still checkpointed once its slot's WAL lag exceeds half of `svs.max_slot_wal_size`. Setting `svs.checkpoint_operations` or `svs.checkpoint_interval` selects a simple mode instead: a checkpoint runs when either threshold is reached. `min_ops` acts as a **filter** that prevents expensive checkpoints for trivial changes. `max_interval` is a safety net for constant-write workloads (never-quiet) that would otherwise cause unbounded WAL accumulation. **Clean shutdown checkpoint:** When BGW receives SIGTERM, it performs a final checkpoint of each cached index before exiting, within `svs.shutdown_drain_budget_ms`, advancing the slot to the checkpoint LSN. For an index checkpointed within the budget, the next startup has zero replay cost. #### Atomic Checkpoint Sequence (5 Phases) ```c bool PerformCheckpoint(VamanaIndexCache *cache) { // Snapshot the flush LSN first, so WAL that arrives during the save is not skipped checkpoint_lsn = GetFlushRecPtr(); // Phases 1-4, inside VamanaSaveIndexToDisk: // write the SVS graph and TID map to temporary files, fsync them, // atomically rename them into place, fsync the directory VamanaSaveIndexToDisk(indexRel, cache->svsIndex, MAIN_FORKNUM, cache, &compactedOnDisk); // Phase 5: advance the slot (ONLY after the checkpoint is fully durable) VamanaSlotAdvance(cache->replicationSlot, checkpoint_lsn); } ``` **Critical invariant:** Slot LSN only advances **after** `VamanaSaveIndexToDisk` succeeds, and only to an LSN taken before the save began. | Phase | Purpose | Crash result | |---|---|---| | 1. Write temp files | Isolate writes from live files | Crash → temp incomplete, old files intact | | 2. Fsync temp files | Ensure temp files durable | Crash → temp complete but not renamed | | 3. Atomic rename | Make new files visible | Crash → either old OR new files exist | | 4. Fsync directory | Make renames durable | Crash → new files guaranteed visible | | 5. Advance slot | Mark checkpoint complete | Crash before → replay from old LSN; after → skip replay | #### Crash Scenarios and Recovery **Scenario 1: Crash after applying to memory, before checkpoint** ``` t1: BGW reads slot (LSN 100-200) → 101 vectors in memory, 100 on disk, slot LSN 100 t2: CRASH t3: Restart: Load on-disk graph → 100 vectors t4: Replay from slot LSN 100 → 101 vectors Result: ✅ Correct — changes reapplied from slot ``` **Scenario 2: Crash after SVSSaveIndex, before slot advance** ``` t1: BGW applies changes (LSN 100-200) → 101 vectors in memory, 100 on disk, slot LSN 100 t2: Checkpoint: SVSSaveIndex() → 101 vectors on disk, slot LSN still 100 t3: CRASH (before slot advance) t4: Restart: Load on-disk graph → 101 vectors t5: Replay from slot LSN 100 → externalId=42 replayed again → idempotent no-op Result: ✅ Correct — idempotency handles the duplicate ``` **Scenario 3: Successful checkpoint, then crash** ``` t1: SVSSaveIndex() → 101 vectors on disk, slot LSN 100 t2: VamanaSlotAdvance(200) → slot LSN 200 t3: CRASH t4: Restart: Load on-disk graph → 101 vectors t5: Replay from slot LSN 200 forward → no changes to replay Result: ✅ Correct — zero replay cost ``` #### Key Properties | Property | Guarantee | |---|---| | **Durability** | If slot LSN = X, on-disk graph contains all changes up to X | | **No Data Loss** | Crash recovery replays from slot LSN forward (bounded cost) | | **Idempotency** | Safe to replay the same change multiple times | | **Atomicity** | Either old checkpoint OR new checkpoint exists (never partial) | | **Ordering** | Save first, advance slot second (never reverse) | | **Search Latency** | Paused during checkpoint | | **Insert/Delete Latency** | Paused during checkpoint (same window) | | **Clean Shutdown** | Zero replay cost for indexes checkpointed within `svs.shutdown_drain_budget_ms` | ### 5.9 WAL and Replication Slot Internals #### What Is WAL WAL is PostgreSQL's transaction log — a continuous, append-only stream of records that describes every change made to the database. Changes are written to WAL **before** they're written to data pages, ensuring durability and enabling crash recovery. #### What Is LSN **LSN** (Log Sequence Number) is a 64-bit unsigned integer representing a byte offset (position) in the WAL stream. It tells you *where* to find something in WAL, not what it contains. LSNs are displayed as `XX/YYYYYYYY` (segment number / byte offset within segment). #### How the BGW Uses WAL The BGW does not write WAL — it reads from a logical replication slot. Backends' DML writes normal PostgreSQL WAL. The BGW polls the slot, which delivers only committed changes (reorder buffer filters aborted transactions). ``` Backend commits transaction: ↓ PostgreSQL writes INSERT + COMMIT records to WAL ↓ fsync() — now durable on disk ↓ BGW polls slot — receives the committed INSERT ↓ SVSAddPoints applied to in-memory graph ``` #### restart_lsn vs confirmed_flush_lsn Logical decoding encounters transactions that span multiple WAL records. The reorder buffer buffers changes until COMMIT is seen, then delivers all changes atomically. On restart, the slot must start reading from the BEGIN of the oldest buffered transaction — not from the last delivered position — to reconstruct transaction state. This is why `restart_lsn` is earlier than `confirmed_flush_lsn`. **What we control:** - When to advance `confirmed_flush_lsn` (`VamanaSlotAdvance`, at checkpoint) - What LSN to advance it to (the LSN through which the on-disk save is durable) - Ensuring idempotency for duplicate replay **What PostgreSQL controls:** - Calculating `restart_lsn` based on reorder buffer state - Delivering only committed, complete transactions - Filtering aborted transactions We never set `restart_lsn` directly: it is advanced only by PostgreSQL's logical decoding. The worker reads it to measure WAL lag, and `VamanaReplicationCatchUpSlot` decodes the slot forward so that `restart_lsn` can move. #### WAL Retention Implications PostgreSQL retains WAL from `restart_lsn` forward (not `confirmed_flush_lsn`). Causes of large lag: 1. **Long-running transactions** — `restart_lsn` anchored at BEGIN for duration 2. **BGW down** — `confirmed_flush_lsn` not advancing, WAL accumulates 3. **Infrequent checkpoints** — `confirmed_flush_lsn` advances only at checkpoint The `svs.max_slot_wal_size` GUC is the safety valve: if lag exceeds the threshold, drop slot and rebuild from heap. A checkpoint is forced earlier, at half the threshold, for any index with writes. | LSN Type | Managed by | When it advances | Affects WAL retention | |---|---|---|---| | `restart_lsn` | PostgreSQL | All buffered transactions delivered | Yes | | `confirmed_flush_lsn` | SVS extension (at checkpoint) | `VamanaSlotAdvance` call | No | #### Slot Removal on DROP INDEX Retention is cluster-wide: WAL is kept from the oldest `restart_lsn` in `pg_replication_slots`, whatever database owns it, and `catalog_xmin` holds back catalog VACUUM the same way. A slot whose index no longer exists therefore pins WAL forever with nothing left to replay into, and defeats `svs.max_slot_wal_size` for every database. Dropping the index must drop the slot. The `OAT_DROP` object-access hook fires before commit, so it only *queues* the drop (`VamanaReplicationQueueDropAtCommit`); the drop itself runs from the `XACT_EVENT_COMMIT` callback. A rolled-back `DROP INDEX` thus keeps its slot, as it must — the index is still catalogued and a slot cannot be un-dropped. The worker holds one of these slots only for the length of a single pass over it — a replay or snapshot-build WAL scan, a bounded standby activation, or, on a primary, the checkpoint `confirmed_flush` advance in `VamanaSlotAdvance`. Every one of those acquires with `nowait = true` and releases before returning, so between passes, at the top of the worker's main loop, `MyReplicationSlot == NULL`. At commit the backend makes exactly one non-blocking attempt (`ReplicationSlotDrop(nowait = true)`) rather than waiting for the holder. Waiting risks a cycle: the holder can be the worker inside `VamanaReplicationBuildSnapshot`, whose `SnapBuildWaitSnapshot` → `XactLockTableWait` waits for the dropping transaction's own XID to finish before the snapshot can reach CONSISTENT, while the backend waits for that worker to release the slot. `VamanaReplicationBuildSnapshot`'s header comment predicts it. The deadlock detector cannot break such a cycle, because only the worker's half is a lock wait — the backend's half is a condition variable. Nor can an operator: the callback runs inside `CommitTransaction`'s `HOLD_INTERRUPTS`, so a backend sleeping there ignores `pg_cancel_backend` and `pg_terminate_backend` alike, and every `DROP DATABASE` in the cluster then blocks behind the `ProcSignalBarrier` it will not accept. Independent of any cycle, waiting would sleep while holding `AccessExclusiveLock` on the index and its table — blocking every other reader of that table — and a third-party holder such as `pg_recvlogical` may never release at all. If the attempt reports the slot busy, the backend hands the relid to that database's worker (`pendingSlotDrops[]`, drained by `VamanaWorkerProcessSlotDrops` next to the reload queue, within one ~1 s heartbeat). The worker holds no slot of its own at that point in its loop, so it cannot lose the race. The handoff is written under the shmem header lock (`VamanaWorkerRequestSlotDrop`, beside the take below). Resolving the entry and then writing to it after the lock is gone would allow the entry to be released and re-reserved in between: the request would land in another database's queue, where the same relid names a different slot, and the slot it was meant for would lose its last reference. A queue entry is cleared only after its attempt returns, so a worker that dies partway leaves the request for its replacement, and the queue itself survives a worker restart. It cannot survive the shmem entry being released, though: a recycled entry belongs to another database, where the same relid would name a different slot. So the departing worker drains the queue once more in its shutdown sequence, and if it died before it could, the launcher finishes the drops itself as it retires the entry (`VamanaWorkerTakePendingSlotDrops`) — dropping a logical slot needs no connection to its database, only decoding from it does. Nothing on this path raises an error — the transaction has already committed and the index is gone. A drop that cannot be completed (no worker entry, or a full handoff queue) is reported as a `WARNING` naming the slot, with the hint to remove it via `pg_drop_replication_slot()`. --- ## 6. GUC Parameters The search-thread GUCs in §6.1 and the per-database `maintenance_num_threads` override are not applied directly; they are inputs to a launcher-side grant calculator that apportions threads across every enabled database once per reconcile pass, enforcing per-database and cluster-wide ceilings and a database's reserved floor before anything else. See [BGW_LAUNCHER.md §6, "The CPU grant pass"](BGW_LAUNCHER.md#6-launcher-behavior) for the algorithm. ### 6.1 Runtime Query Parameters | GUC | Type | Default | Range | Scope | Description | |---|---|---|---|---|---| | `svs.search_window_size` | int | 100 | 10–10000 | `PGC_USERSET` | Search window (L) for index scans. Higher values improve recall at the cost of latency. | | `svs.search_num_threads` | int | 0 | 0–1024 | `PGC_SIGHUP` | Cluster-wide default search thread count, subject to the launcher's grant path (see note below the table). `0` = auto, resolves to `1`. Superuser-only; no session `SET`. | | `svs.max_search_threads_per_db` | int | 0 | 0–1024 | `PGC_SIGHUP` | Per-database ceiling on search threads. `0` = follow `max_parallel_workers`. Superuser-only; no session `SET`. | | `svs.max_total_search_threads` | int | 0 | 0–1024 | `PGC_SIGHUP` | Cluster-wide ceiling on search threads across all databases. `0` = follow `max_parallel_workers`. Superuser-only; no session `SET`. | The three thread GUCs (`svs.search_num_threads`, `svs.max_search_threads_per_db`, `svs.max_total_search_threads`) are inputs to the launcher's grant calculator, not values a worker uses directly: search reads its last-published grant non-blockingly at query time, while a build blocks on `SvsWaitForBuildGrant` until the launcher grants it. Both paths resolve through the same calculator and the same floor/ceiling logic; search does not bypass it. ### 6.2 Background Worker Parameters The launcher runs one background worker per enabled database. Which databases are enabled is controlled by the `vamana_databases` catalog, not a GUC; see [BGW_LAUNCHER.md](BGW_LAUNCHER.md) for the launcher, enrollment, and restart/backoff model. `svs.launcher_database`, `svs.max_databases` and `svs.worker_restart_time` require a server restart: they are `PGC_POSTMASTER` when the library is loaded through `shared_preload_libraries`, which the extension requires. The timeout GUCs can be updated at runtime via `SIGHUP`. | GUC | Type | Default | Range | Scope | Description | |---|---|---|---|---|---| | `svs.launcher_database` | string | `"postgres"` | — | `PGC_POSTMASTER` | Database that holds the `vamana_databases` enrollment catalog and where the launcher connects to read it. | | `svs.max_databases` | int | 8 | 1–128 | `PGC_POSTMASTER` | Maximum number of databases that can run a worker concurrently. Sizes the shared-memory slot array. | | `svs.worker_restart_time` | int | 5 | -1–300 | `PGC_POSTMASTER` | Seconds before the crashed launcher is restarted by the postmaster. `-1` = `BGW_NEVER_RESTART`. Per-database worker restart is governed by `svs.worker_restart_backoff`. | | `svs.worker_restart_backoff` | int | 1000 | 100–300000 | `PGC_SIGHUP` | Base milliseconds for a crashed per-database worker's exponential restart backoff (capped at 60s, reset after a healthy dwell). See [BGW_LAUNCHER.md](BGW_LAUNCHER.md). | | `svs.worker_startup_timeout_ms` | int | 60000 | 1000–300000 | `PGC_SIGHUP` | Milliseconds a backend waits for the worker to finish startup before throwing an error. Startup can be slow when many large indexes are deserialized from disk. | | `svs.worker_timeout_ms` | int | 5000 | 100–60000 | `PGC_SIGHUP` | Milliseconds a backend waits for the worker to respond to an IPC request (search or write) before throwing an error. | | `svs.max_batch_size` | int | 0 | 0–1000 | `PGC_SIGHUP` | Maximum queries per SVS batch search call. `0` = `MaxBackends`. | | `svs.compact_threshold_pct` | int | 10 | 0–100 | `PGC_SUSET` | Percent-deleted threshold that triggers SVS compact during VACUUM cleanup. `0` = compact on every VACUUM with pending deletes; `100` = disable compact (consolidate still runs). Also bounds the database-wide residency credit on aborted inserts, so it is superuser-only, not per-session. | | `svs.max_slot_wal_size` | int (MB) | 10240 | 64–2147483647 | `PGC_SIGHUP` | Maximum WAL retained by the replication slot. If exceeded, the slot is dropped and the index is rebuilt from the heap. | | `svs.checkpoint_debounce_window` | int (s) | 300 | 0–86400 | `PGC_SIGHUP` | Seconds of write inactivity required after a burst before a checkpoint is triggered (AND-logic with `checkpoint_min_ops`). | | `svs.checkpoint_max_interval` | int (s) | 3600 | 1–86400 | `PGC_SIGHUP` | Maximum seconds between checkpoints; safety net for constant-write workloads that never go quiet. | | `svs.checkpoint_min_ops` | int | 10000 | 0–INT_MAX | `PGC_SIGHUP` | Minimum write operations required before any checkpoint rule fires. Prevents expensive checkpoints for trivial changes. | | `svs.checkpoint_operations` | int | -1 | -1–INT_MAX | `PGC_SIGHUP` | Simple op-count checkpoint trigger. `-1` = off. Setting a positive value activates simple mode (overrides debounce logic). | | `svs.checkpoint_interval` | int (s) | -1 | -1–86400 | `PGC_SIGHUP` | Simple time-based checkpoint trigger in seconds. `-1` = off. Setting a positive value activates simple mode. | | `svs.shutdown_drain_budget_ms` | int | 30000 | 0–600000 | `PGC_SIGHUP` | Time budget for the worker's shutdown drain, checked between indexes. A single in-progress checkpoint is not preemptible, so the real bound is this budget plus one checkpoint's worst case. | | `svs.undo_log_max_entries` | int | 1048576 | 16–16777216 | `PGC_SUSET` | Per-transaction cap on undo-log entries (one per Vamana index insert). An insert that would exceed it fails. | | `svs.worker_stop_timeout_ms` | int | 30000 | 0–600000 | `PGC_SIGHUP` | Milliseconds the launcher waits for a restarting worker to report stopped before giving up. Does not force-kill; the restart stays pending until the worker exits naturally. | ### 6.2a Memory Management Parameters Residency and build memory are two independent axes, each with its own cluster-wide ceiling and per-database default; a per-database override lives in `vamana_databases.residency_memory`/`search_work_mem` (see [Section 6.1](../USER_GUIDE.md#61-background-workers-and-per-database-enablement-advanced) of the User Guide). The `max_*` and `default_*` GUCs in each pair are deliberately independent, not one derived from the other: collapsing them would let a single unconfigured database's default consume the whole cluster ceiling. | GUC | Type | Default | Range | Scope | Description | |---|---|---|---|---|---| | `svs.max_build_memory` | int (MB) | 4096 | 1–INT_MAX | `PGC_SIGHUP` | Cluster-wide ceiling on the sum of every in-progress `CREATE INDEX`/`REINDEX` build's peak memory. Always finite; there is no "0 means unlimited" path. | | `svs.max_residency_memory` | int (MB) | 100 | 1–INT_MAX | `PGC_SIGHUP` | Cluster-wide ceiling on the sum of every database's resolved residency budget. Checked at config-time admission (`vamana_databases` enrollment), not at query time. | | `svs.default_residency_memory` | int (MB) | 100 | 1–INT_MAX | `PGC_SIGHUP` | Residency budget for a database whose `vamana_databases.residency_memory` is `NULL`. | | `svs.max_search_work_mem` | int (MB) | 100 | 1–INT_MAX | `PGC_SIGHUP` | Cluster-wide ceiling on the sum of every database's resolved search-scratch (per-query candidate buffer) budget. Checked as a plain catalog aggregate at `vamana_databases` insert/update time, not against live worker state. | | `svs.default_search_work_mem` | int (MB) | 100 | 1–INT_MAX | `PGC_SIGHUP` | Search-scratch budget for a database whose `vamana_databases.search_work_mem` is `NULL`. | ### 6.3 Index Creation Parameters Specified in the `WITH (...)` clause of `CREATE INDEX`. | Parameter | Type | Default | Range | Description | |---|---|---|---|---| | `graph_degree` | int | 64 | 16–256 | Maximum graph connections per node (R parameter). | | `alpha` | int | -1 | -1–200 | Pruning parameter scaled by 100 (`120` = α 1.20). `-1` uses SVS library default. | | `build_window_size` | int | -1 | -1–1000 | Build window size (L parameter). `-1` = `2 × graph_degree`. | | `search_window_size` | int | 100 | 10–10000 | Stored with the index. The `svs.search_window_size` GUC governs queries. | | `use_search_history` | bool | `true` | — | Maintain the visited-node set during search. | | `compression_type` | int | 0 | 0–2 | `0` = none, `1` = LeanVec, `2` = LVQ. | | `compression_primary` | int | 4 | -8–8 | Primary quantization under either scheme: `4`=UINT4, `-4`=INT4, `8`=UINT8, `-8`=INT8. | | `compression_secondary` | int | 8 | -8–8 | LeanVec secondary quantization, or the LVQ residual. Same values as primary, plus `0` = no residual (LVQ only). LVQ accepts the bit-width pairs (4,0), (8,0), (4,4), (4,8) and discards the sign; LeanVec requires `abs(primary) <= abs(secondary)`. | | `leanvec_dims` | int | -1 | -1–2000 | Reduced dimensions for LeanVec. `-1` = `dimensions / 2`. Ignored under LVQ, which quantizes in the original space. | --- ## 7. Integration Points ### 7.1 Build System (Makefile) The extension is built with `pgxs` and requires the Intel SVS C API library: ```makefile SVS_INSTALL ?= svs_install_public # Requires: $(SVS_INSTALL)/lib/libsvs_c_api.so ``` Compiler flags for SIMD optimization: ``` -march=native -ftree-vectorize -fassociative-math -fno-signed-zeros -fno-trapping-math ``` > **Note:** `-march=native` is conditionally disabled on ARM Mac, PowerPC, and RISC-V64 (see the `OPTFLAGS` block in the Makefile). On those platforms `OPTFLAGS` is empty and only the remaining flags are applied. ### 7.2 Extension Control File (`svs.control`) ``` comment = 'SVS Vamana index access method for pgvector' default_version = '0.1.0' requires = 'vector' relocatable = false module_pathname = '$libdir/svs' ``` ### 7.3 SQL Registration ```sql CREATE FUNCTION vamanahandler(internal) RETURNS index_am_handler ...; CREATE ACCESS METHOD vamana TYPE INDEX HANDLER vamanahandler; ``` Operator classes for `vector` and `halfvec` are registered as shown in Section 4.6. Users install the extension with: ```sql CREATE EXTENSION vector; -- required first: svs depends on it CREATE EXTENSION svs; ``` ### 7.4 Loading via shared_preload_libraries Loading `svs` via `shared_preload_libraries` registers the launcher at server start. The launcher then spawns one worker per database enrolled in the `vamana_databases` catalog; loading the library alone does not start any per-database worker. See [BGW_LAUNCHER.md](BGW_LAUNCHER.md) for enrollment. ``` # postgresql.conf shared_preload_libraries = 'svs' svs.launcher_database = 'postgres' # default: 'postgres'; holds vamana_databases svs.max_databases = 8 # default: 8; concurrent workers ``` `svs` depends on the `vector` extension's types at CREATE EXTENSION time, but `vector` does not need to be in `shared_preload_libraries`. --- ## 8. Performance Considerations ### 8.1 Optimization Strategies 1. **AVX-512 SIMD:** SVS automatically uses Intel hardware optimizations when compiled with `-march=native` 2. **Compression:** Enable LVQ (`compression_type=2`) for memory-constrained systems; LeanVec (`compression_type=1`) for two-level quantization 3. **Thread count:** Tune `svs.search_num_threads` to match workload; `0` resolves to `1` 4. **Background worker:** The per-database worker holds the index in memory and amortizes load cost across all backends. Enroll the database and, for latency-sensitive deployments, warm indexes ahead of query traffic with `svs_warmup_index`/`svs_warmup_database` so the first query does not pay the demand-load cost. See [BGW_LAUNCHER.md](BGW_LAUNCHER.md) 5. **Incremental writes:** Inserts update the graph incrementally; periodic `REINDEX` restores optimal graph quality after many mutations ### 8.2 Write Considerations Inserts are applied incrementally via `SVSAddPoints` and are immediately searchable. Deletes are applied during VACUUM via `SVSDeletePoints` with graph consolidation. For write-heavy workloads: - The BGW main loop handles requests one at a time, and writers hold a per-index LWLock (`VamanaIndexLockSlot`); high-throughput inserts may experience contention - One worker serves all Vamana indexes in a database, so a write to one index also delays searches and writes on the other indexes in that database; workloads in separate databases do not affect each other - Checkpoints run on the BGW main thread and block both searches and writes for their duration. This is required because `MutableVamanaIndex::save()` is not `const`: it calls `consolidate()` then `compact()` before writing, mutating shared graph state. Concurrent save + search is an unconditional data race. INSERT, DELETE, and SEARCH requests queue on their IPC slot latches and are processed once the checkpoint completes - Periodic `REINDEX CONCURRENTLY` produces a fresh, optimally-constructed graph after many incremental mutations - All writes (INSERT, DELETE, VACUUM) go through the background worker; the worker is the single owner of the live in-memory index, so all backends see a consistent view after the worker processes each slot ### 8.3 Memory Overhead The TID mapping array is sized by `tidMappingCapacity`, not by the live vector count `numVectors`. After deletions, the capacity grows monotonically until `SVSCompact` reclaims the deleted slots. In workloads with high delete churn and infrequent compaction, the TID mapping and its on-disk sidecar can be significantly larger than the live vector count alone would suggest. Lowering `svs.compact_threshold_pct` reduces this overhead at the cost of more frequent compaction during VACUUM. Each index lives in the private memory of its database's background worker, so every index needs memory at build time and at search time. Build memory is bounded cluster-wide by `svs.max_build_memory`; the resident size of all loaded indexes in a database is bounded by that database's residency budget (see below). In addition to the graph edges and stored vectors, the SVS dynamic index maintains a small amount of per-vector bookkeeping (delete flags, insertion metadata). This overhead is small relative to the graph itself but grows linearly with `tidMappingCapacity`. **Cache eviction.** `VamanaIndexCache` is a byte-bounded, relid-keyed table governed by each database's residency budget (`vamana_databases.residency_memory`, or `svs.default_residency_memory`/`svs.max_residency_memory`; see [Section 6.2a](#62a-memory-management-parameters)), not a fixed slot count. A load that would push a database's committed resident bytes over its budget is refused before the cache is touched, with an error naming `svs.max_residency_memory`. Each index's exact resident bytes, as last measured by the worker, are also durably recorded in `svs_index_residency` so a `residency_memory` decrease can be validated even while that database's worker is not running. ### 8.4 Known Limitations #### Searches wait during writes Each enrolled database has a single SVS worker that handles all of its Vamana requests in turn. While the worker is applying a write, searches on every Vamana index in that database wait until it finishes. Writes include INSERT, UPDATE, DELETE, and VACUUM. What to expect: - Under a mixed workload, search latency rises and search throughput drops as the proportion of write traffic increases. - Workloads in separate databases don't affect each other, because each database has its own worker. - VACUUM on a Vamana index can pause searches in that database for as long as each clean-up or compaction step takes. This grows with index size. The same applies to autovacuum. To limit the effect: - Schedule bulk loads and manual VACUUM outside peak search periods. - For a large initial load, insert the data first and run CREATE INDEX afterwards, rather than inserting row by row into an indexed table. - Put workloads that need fast searches during heavy writes in separate databases. #### Other limitations - Checkpoints block both searches and writes for their duration (see [8.2](#82-write-considerations)). - `PREPARE TRANSACTION` (two-phase commit) is not supported on a transaction that has inserted into a Vamana index (see [4.3](#43-insert-and-vacuum-behavior)). - Only `vector` (float32) and `halfvec` (float16) columns are supported; `sparsevec` and `bit` are not. The maximum dimension is 2000. - The index serves only `ORDER BY ... LIMIT` nearest-neighbor scans. - At most 64 Vamana indexes can be live in one database (`VAMANA_MAX_INDEXES`). - After a worker crash, reload time grows with the total serialized index size. - `ALTER INDEX ... SET TABLESPACE`, `CREATE INDEX ... TABLESPACE` and `default_tablespace` move only the index relation's own pages, including the metapage. The serialized graph and vectors are always stored under `$PGDATA/vamana_indexes/`. - `maintenance_work_mem` does not affect index builds; build memory is bounded by `svs.max_build_memory`. - Rows with a `NULL` indexed value are not added to the index. Cosine distance on a zero vector returns `NaN`. --- ## 9. Security Objectives This section summarizes the security objectives for each extension-managed asset, derived from a formal threat model using DREAD scoring. ### 9.1 Asset Security Properties | Asset | Confidentiality | Integrity | Availability | |---|---|---|---| | Vamana metapage and TID mapping pages | — | Required (page checksums) | Required | | Serialized SVS index directory (`$PGDATA/vamana_indexes//`) | Required (contains embeddings) | Required | Required | | Extension binaries (`svs.so`, `svs.control`, SQL scripts) | — | Required (RPATH integrity) | — | | Intel SVS shared library (`libsvs_c_api.so`) | Required (proprietary) | Required | — | | Per-transaction undo log (`vamana_undo.c`) | — | Required | Required (bounded growth) | | Launcher and per-database BGW main loops (one worker per enabled database) | — | Required | Required (restart recovery, crash backoff) | | BGW index cache (`VamanaIndexCache`) | Required (embeddings in memory) | Required | — | | `svs_index_residency` catalog table | — | Required (a stale entry could wrongly permit or block a `residency_memory` decrease) | — | | Per-index logical replication slot (`pg_replslot/vamana_*`) | — | Required | Required (WAL retention bound) | ### 9.2 Key Security Properties **Confidentiality.** Stored embeddings (on disk in `$PGDATA/vamana_indexes/` and in BGW memory) follow PostgreSQL's `$PGDATA` permissions posture, which `SetDataDirectoryCreatePerm` derives from the mode of `$PGDATA` itself; the umask the postmaster inherits is discarded (`postmaster.c` `umask(PG_MODE_MASK_OWNER)`, then `umask(pg_mode_mask)` in `miscinit.c`). Every mode the extension controls is stated at the point of creation rather than obtained by umask reduction: the three directories it creates itself, `vamana_indexes/`, `vamana_indexes//` and `vamana_indexes///`, via `MakePGDirectory` (`pg_dir_create_mode`, `0700` or `0750` with group access), and `tidmap.bin` via `OpenTransientFilePerm` plus an `fchmod` on the still-empty file (`pg_file_create_mode`, `0600` or `0640`) — see `VamanaEnsureSaveDir` and `VamanaSaveTidMapAtomically` in `vamanaio.c`. The `pg_*_create_mode` globals are used in preference to hardcoded modes so a cluster initialized with `initdb -g` retains group access. The remaining gap is the SVS payload, which the library creates in its entirety: `svs_index_save(index, directory, error)` accepts no mode, and the library creates the `config/`, `data/` and `graph/` subdirectories requesting `0777` and their files requesting `0666`. Nothing the extension does narrows those requests — `pg_mode_mask` alone is what brings them down to `0700`/`0600`, or `0750`/`0640` with group access. Like the assumption about SVS error strings below, this is a property of the library's I/O behavior and should be rechecked on each SVS version bump. The extension does not implement access control beyond the filesystem in this release. SVS error strings contain only metadata (operation names, error codes, dimensions), not vector values; this assumption is revalidated on each SVS library version bump. **Temporary file lifetime.** `tidmap.bin` is published atomically by writing `tidmap.bin.tmp` and renaming it. The temp file is unlinked on every failure path in `VamanaSaveTidMapAtomically`, and any pre-existing temp file is unlinked before the new one is created with `O_EXCL`. That delete-before-create is a correctness requirement as well as cleanup: `O_CREAT` does not alter the mode of a file that already exists, so a stale temp file at a wider mode would keep it and the rename would publish that mode onto `tidmap.bin`. A temp file orphaned by a crash between the write and the rename is swept at the top of `VamanaLoadTidMap`, gated on `st_mtime < PgStartTime` so a concurrent writer's temp file is never removed between its close and its rename. **Integrity.** The SVS wrapper trust boundary (`svs_wrapper.c`) validates output-buffer bounds before returning results to backends. Input-buffer sizing is a caller responsibility documented in `svs_wrapper.h`. The atomic 5-phase checkpoint sequence ensures on-disk state is never partial. The replication slot mechanism guarantees crash recovery replays only committed transactions (reorder buffer filtering). **Availability.** The per-transaction undo log is bounded to prevent memory-DoS from large transactions. BGW saturation from concurrent high-`search_window_size` queries is bounded by `svs.worker_timeout_ms` (converts hangs to ERRORs). The BGW restarts automatically after crashes with bounded replay cost from the replication slot. ### 9.3 Accepted Residuals The following risks are accepted with documented rationale: | Risk | Severity | Disposition | |---|---|---| | BGW saturation via `search_window_size = 10000` + connection flood | High | Accepted as bounded: `worker_timeout_ms` converts hangs to ERRORs; `search_window_size` GUC bounded 10-10000. | | CREATE INDEX OOM on oversized table | Medium | Backend-local failure; postmaster not impacted. Build memory is bounded cluster-wide by `svs.max_build_memory`. | | BGW crash amplification (reload time proportional to index size) | Medium | Accepted as known. Reload time is linear in total serialized index size; documented in deployment guidance. | | MKL CPU-feature environment variable forcing SIGILL on incompatible hardware | Medium | Accepted: operator environment is trusted configuration. See the User Guide for AVX-512 requirements. |