# Detailed Benchmark Results The HNSW comparison uses pgvectorfast 0.2.1, upstream pgvector 0.8.7, Chroma 1.5.9, and Qdrant 1.20.1 on the same SIFT10M (128D) and DEEP10M (96D) vectors. The IVFFlat and SQL operator results below are from earlier runs. ## HNSW setup - Hardware: Intel Core i9-13900K, 62 GiB host RAM, Samsung 980 PRO NVMe. - Each service has one pinned server CPU, one client on a separate CPU, and one outstanding query. Both PostgreSQL variants use PostgreSQL 18.7. - Indexes use float32 vectors, L2 distance, `M=16`, and `ef_construction=64`, without quantization. The PostgreSQL variants share an identical heap/index snapshot. Chroma and Qdrant build their own HNSW graphs from the same vectors. - Total service memory, including file cache, is limited to **16 GiB** in memory and **2 GiB** for the larger-than-memory runs, with swap disabled. PostgreSQL uses 14 GiB and 1536 MiB of shared buffers, respectively, plus `io_uring` and direct data I/O. Files from the preceding service are evicted from its cache before starting the next cgroup. - A fixed random permutation (NumPy seed `20261115`) splits each dataset's 10,000 queries: 4,000 for calibration, 6,000 disjoint queries for in-memory timings, and the first 1,000 of those for larger-than-memory timings. The first 2,000 calibration queries warm each engine. Every engine uses the same order and split. The smallest `ef_search` reaching calibration Recall@10 of at least 0.9336 for SIFT10M or 0.8712 for DEEP10M is selected. The tables report actual timed-query recall. - Results are medians of three repeats; chart error bars show their full range. Latency includes the native client API call: prepared binary psycopg queries for PostgreSQL, gRPC for Qdrant, and HTTP for Chroma. QPS is `1000 / mean_latency_ms`; recall checks and result hashing are outside the timer. Index builds are paused during timings. - Larger-than-memory results require physical NVMe reads during the timed queries. The raw results include cgroup memory and I/O counters and Calico mapping checks. See the [runner and reproduction instructions](bench/hnsw/README.md) and [raw results](bench/hnsw/results) for calibration trials, all timing repeats, image identities, and environment details. ## HNSW summary - At 16 GiB per service, pgvectorfast delivers **5.32x** the throughput of upstream pgvector on SIFT10M and **5.88x** on DEEP10M. - At 2 GiB per service, pgvectorfast delivers **5.70x** the throughput of upstream pgvector on SIFT10M and **5.91x** on DEEP10M. ## Search calibration | Dataset | System | ef_search | Calibration Recall@10 | | --- | --- | ---: | ---: | | SIFT10M | pgvector | 40 | 0.93388 | | SIFT10M | pgvectorfast | 40 | 0.93388 | | SIFT10M | Chroma | 61 | 0.93415 | | SIFT10M | Qdrant | 35 | 0.93458 | | DEEP10M | pgvector | 39 | 0.87320 | | DEEP10M | pgvectorfast | 39 | 0.87315 | | DEEP10M | Chroma | 59 | 0.87275 | | DEEP10M | Qdrant | 34 | 0.87262 | ## In-Memory HNSW (16 GiB per Service) | Dataset | System | ef_search | Recall@10 | QPS | Mean (ms) | P95 (ms) | | --- | --- | ---: | ---: | ---: | ---: | ---: | | SIFT10M | pgvector | 40 | 0.9334 | 927.0 | 1.079 | 1.460 | | SIFT10M | pgvectorfast | 40 | 0.9334 | 4,933.1 | 0.203 | 0.250 | | SIFT10M | Chroma | 61 | 0.9335 | 847.4 | 1.180 | 1.296 | | SIFT10M | Qdrant | 35 | 0.9335 | 1,668.5 | 0.599 | 0.675 | | DEEP10M | pgvector | 39 | 0.8655 | 836.1 | 1.196 | 1.550 | | DEEP10M | pgvectorfast | 39 | 0.8656 | 4,913.3 | 0.204 | 0.245 | | DEEP10M | Chroma | 59 | 0.8664 | 862.9 | 1.159 | 1.282 | | DEEP10M | Qdrant | 34 | 0.8673 | 1,659.1 | 0.603 | 0.668 | ## Larger-Than-Memory HNSW (2 GiB per Service) | Dataset | System | ef_search | Recall@10 | QPS | Mean (ms) | P95 (ms) | | --- | --- | ---: | ---: | ---: | ---: | ---: | | SIFT10M | pgvector | 40 | 0.9356 | 11.4 | 87.668 | 129.250 | | SIFT10M | pgvectorfast | 40 | 0.9355 | 65.0 | 15.393 | 19.753 | | SIFT10M | Chroma | 61 | — | OOM at 2 GiB | — | — | | SIFT10M | Qdrant | 35 | 0.9372 | 21.2 | 47.240 | 74.524 | | DEEP10M | pgvector | 39 | 0.8753 | 10.4 | 96.219 | 127.511 | | DEEP10M | pgvectorfast | 39 | 0.8753 | 61.4 | 16.294 | 19.903 | | DEEP10M | Chroma | 59 | — | OOM at 2 GiB | — | — | | DEEP10M | Qdrant | 34 | 0.8779 | 17.9 | 55.917 | 74.860 | Chroma's process was OOM-killed while loading the 10M-vector HNSW index under the 2 GiB limit on SIFT10M and DEEP10M. Throughput and recall are unavailable for those runs. ## IVFFlat Vector Search (`shared_buffers = 32 GB`) Even without HNSW-specific optimistic read or graph prefetch paths, replacing PostgreSQL's shared buffer lookup hash table reduces query latency for IVFFlat scans: | Dataset | Dimensions | Recall | Stock `pgvector` Median (ms) | `pgvectorfast` Median (ms) | Latency Reduction | | --- | --- | --- | --- | --- | --- | | `DBPedia OpenAI-1M` | 1536 | `Recall@3 = 0.91` | 8.62 | **6.27** | **-27.3%** | | `GIST1M` | 960 | `Recall@3 = 0.87` | 18.63 | **15.37** | **-17.5%** | ## Analytical Scans & Joins (`shared_buffers = 128 GB`) Because `CALL calico.activate()` upgrades PostgreSQL's shared buffer pool for all relations in the instance, standard SQL analytical queries also benefit across row widths from `128 B` to `6144 B` (`1.03x`–`1.17x` at `128 B`, scaling up to `2.99x` at `6144 B`): | PostgreSQL Operator | Speedup vs. Stock PostgreSQL (`6144 B` Rows) | | --- | --- | | Sequential Scan | **2.99x** | | Hash Join | **2.34x** | | Merge Join | **1.88x** | | Nested Loop Join | **1.78x** | | Indexed Nested Loop Join | **1.52x** |