# MS MARCO vector benchmarks The MS MARCO workflow compares pg_search, pg_vectorsearch, and VectorChord on 113,520,750 rows with 1,024-dimensional vectors. Each engine restores the same PostgreSQL 18 heap backup (`20261102-224214F`) and runs on a `c8gd.metal-24xl` runner with a 8-CPU / 32-GiB database container on cores 0–7. All engines use 16 GiB shared memory, 8 GiB shared buffers, 8 GiB `maintenance_work_mem`, 24 GiB `effective_cache_size`, and 256 MiB `work_mem`. k6 runs on cores 64–67. CI overrides the pinned pdbench Compose configuration for every engine to use 8 CPUs, eight global parallel workers, and four workers per parallel query. The maintenance-worker limit is seven; VectorChord retains eight internal build threads. The worker-process pool remains 32. Query-session settings also use four workers for recall calibration and k6. Memory settings remain unchanged. Result dataset names include `-8cpu-32gb`, keeping these baselines separate from previous 64-CPU runs. The resolved Compose configuration is included in the run diagnostics as `runtime-compose.yml`. The RunsOn host remains unchanged; these limits apply to the database container. ## Run Apply a PR label to run one engine: | Engine | PR label | | ----------------------- | ----------------------------------------- | | `pg_search` | `benchmark-msmarco-pg_search` | | `pg_vectorsearch` | `benchmark-msmarco-pg_vectorsearch` | | `vectorchord` | `benchmark-msmarco-vectorchord` | | `vectorchord-prefilter` | `benchmark-msmarco-vectorchord-prefilter` | The label is removed when the run starts. Reapply it to rerun. For an on-demand runner, add `benchmark-nonspot` before the engine label. Remove it to use Spot. Daily runs benchmark all three engines with VectorChord prefilter off. Manual runs can also select `vectorchord-prefilter` to enable it; this variant has separate results. Both VectorChord modes use the same unfiltered MS MARCO queries. Manual runs select one engine: ```sh gh workflow run benchmark-pg_search-benchmarker-msmarco.yml \ --repo paradedb/paradedb --ref main -f engine=vectorchord ``` Add `-f runner_type=on-demand` for on-demand runners; Spot is the default. Set `commit` to benchmark another pg_search revision. Scheduled and manual runs on main update each engine's baseline unless `commit` is set. ## Measurements All engines use the same first 100 held-out vectors and published ground truth for unfiltered cosine top-10 and top-100 queries returning `_id, title`. `pdbench-load finalize` builds and vacuums the index. `pdbench search` tunes probes for 90%, 95%, and 99% recall. Reached targets run in k6 with one VU, 10 seconds of warmup, and 50 seconds of measurement. Targets that miss recall have no QPS or latency result. Plan checks require the expected vector index, and k6 checks the returned neighbor count. Results include QPS, latency, buffer counts, build time, index size, and peak build memory. Each engine has its own artifacts and baseline history. The TOML profiles contain index and recall settings. The workflow pins tool and extension revisions. Cluster setup, extension installation, build reporting, and k6 profiling use the same actions as HN. VectorChord's 100,000-list profile uses the memory-saving training settings described below; a completed 113M run with this configuration is still pending. ## Diagnostics Download the engine's diagnostics artifact for plans, raw reports, k6 dashboards, database logs, and memory traces: - Build samples: `benchmark-artifacts/load/*/*_load/points.jsonl`. The published peak uses `container_memory`, which excludes inactive page cache. Timestamps are UTC and values are bytes. - Query samples: `containers.paradedb.memory` in `benchmarker/results/.json`. Timestamps are Unix milliseconds and values are bytes. The collector subtracts cache when available, so these values use a different memory definition from the build peak. Run the shared Python checks locally: ```sh python3 -m unittest discover -s benchmarks/cohere -p 'test_*.py' ``` ## VectorChord comparison profiles The VectorChord profiles use 100,000 lists for MS MARCO, hierarchical k-means in 100 dimensions, sampling factor 64, eight build threads, `build.pin=2`, residual quantization, and spherical centroids. Probe sweeps are capped at the list count. Both build and search use the shared 32 GiB container configuration, with 8 GiB shared buffers and 8 GiB `maintenance_work_mem`.