# mentat scale benchmark: ext-cache

## Environment (abridged; full detail in env.txt)

```
instance_type:   c7i.8xlarge
kernel:          6.12.103-129.197.amzn2023.x86_64
cpu_model:       Intel(R) Xeon(R) Platinum 8488C
cpus:            32
sockets/numa:    1 1 
mem_total_gib:   61.8
hugepages:       HugePages_Total:=0 HugePages_Free:=0 Hugepagesize:=2048 
data_fs:         /dev/nvme0n1p1 xfs   150G   13G  138G   9% /
mentat_git:      d34e0177
duckdb:          1.5.5
postgres:        n/a
```

## Dataset and PG size vs shared_buffers

```
s: 984803 datoms (1600 users, 130000 issues, 6492 history updates)
m: sqlite-ext/duckdb read the embedded-built store (loaded at 1.9.0, upgraded to v2 on first open: 16 s once)
```

## bulk_load

| backend | scale | datoms | wall s | datoms/s | ANALYZE s | VACUUM s | store size | note |
|---|---|---:|---:|---:|---:|---:|---:|---|
| sqlite-ext | s | 984,803 | 67.6 | 14,566 |  |  | 0.19 GiB |  |
| duckdb | s | 984,803 | 67.9 | 14,513 |  |  | 0.19 GiB |  |

## point_lookup

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 15574 | 0.625 | 0.713 | 0.745 | 1.502 | 1,557 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 8655 | 1.126 | 1.287 | 1.393 | 2.342 | 865.5 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 115297 | 0.083 | 0.098 | 0.107 | 0.487 | 11,530 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 18942 | 0.514 | 0.591 | 0.602 | 1.823 | 1,894 | 0 | 1 |

## ref_traversal

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 259 | 38.725 | 39.282 | 39.392 | 40.833 | 25.9 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 30 | 493.431 | 502.992 | 503.067 | 503.067 | 2.0 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 268 | 37.439 | 38.040 | 38.365 | 38.933 | 26.7 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 30 | 479.298 | 487.431 | 487.907 | 487.907 | 2.1 | 0 | 1 |

## aggregate

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 112 | 89.561 | 90.464 | 90.884 | 92.339 | 11.2 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 30 | 1,067 | 1,085 | 1,090 | 1,090 | 0.9 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 112 | 89.406 | 92.040 | 92.597 | 94.205 | 11.1 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 30 | 1,083 | 1,099 | 1,125 | 1,125 | 0.9 | 0 | 1 |

## predicate_scan

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 350 | 27.942 | 31.257 | 31.978 | 32.931 | 34.9 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 30 | 333.272 | 344.593 | 346.458 | 346.458 | 3.0 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 317 | 31.225 | 33.444 | 34.376 | 34.851 | 31.6 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 30 | 390.711 | 396.112 | 400.309 | 400.309 | 2.6 | 0 | 1 |

## pull

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 17318 | 0.559 | 0.640 | 0.671 | 42.579 | 1,732 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 4794 | 1.798 | 4.394 | 5.553 | 8.868 | 479.4 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 205169 | 0.047 | 0.055 | 0.068 | 0.473 | 20,517 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 6750 | 1.149 | 3.765 | 4.792 | 7.680 | 674.9 | 0 | 1 |

## as_of

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 118 | 85.256 | 85.996 | 87.037 | 87.267 | 11.7 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 30 | 1,096 | 1,110 | 1,119 | 1,119 | 0.9 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 118 | 84.768 | 85.500 | 86.394 | 86.401 | 11.8 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 30 | 1,097 | 1,112 | 1,122 | 1,122 | 0.9 | 0 | 1 |

## since

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 1933 | 5.123 | 5.258 | 7.924 | 9.373 | 193.3 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 200 | 48.264 | 53.064 | 90.671 | 156.318 | 19.9 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 2122 | 4.666 | 4.806 | 6.116 | 7.760 | 212.1 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 211 | 46.990 | 49.386 | 57.181 | 58.417 | 21.1 | 0 | 1 |

## input_bindings

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 9112 | 1.051 | 1.162 | 1.193 | 1.902 | 911.1 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 3731 | 2.660 | 2.758 | 2.826 | 3.529 | 373.1 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 19368 | 0.485 | 0.559 | 0.574 | 0.901 | 1,937 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 4570 | 2.149 | 2.258 | 2.346 | 13.933 | 456.9 | 0 | 1 |

## write_mixed

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | write | 14801 | 4.011 | 4.579 | 5.019 | 92.345 | 246.7 | 0 | 1 |
| duckdb | s | 984,803 | 8 | read | 19776 | 41.120 | 52.478 | 68.578 | 88.657 | 329.2 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | write | 11471 | 4.494 | 7.951 | 10.383 | 88.017 | 191.2 | 0 | 1 |
| duckdb | m | 9,850,258 | 8 | read | 1040 | 560.158 | 1,445 | 2,204 | 3,128 | 17.2 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | write | 17744 | 3.376 | 3.748 | 4.055 | 17.326 | 295.7 | 0 | 1 |
| sqlite-ext | s | 984,803 | 8 | read | 20934 | 40.001 | 49.107 | 63.524 | 81.062 | 348.7 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | write | 13011 | 3.828 | 7.389 | 9.725 | 25.896 | 216.8 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 8 | read | 950 | 559.156 | 1,759 | 2,297 | 2,382 | 15.7 | 0 | 1 |

## concurrency_sweep

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| duckdb | s | 984,803 | 1 | read | 749 | 0.694 | 39.149 | 39.366 | 40.230 | 74.8 | 0 | 1 |
| duckdb | s | 984,803 | 8 | read | 5806 | 0.717 | 40.356 | 40.652 | 63.589 | 578.8 | 0 | 1 |
| duckdb | s | 984,803 | 32 | read | 12901 | 1.305 | 70.622 | 79.311 | 177.967 | 1,282 | 0 | 1 |
| duckdb | s | 984,803 | 64 | read | 12713 | 2.914 | 166.626 | 197.772 | 269.112 | 1,261 | 0 | 1 |
| duckdb | s | 984,803 | 128 | read | 12255 | 9.997 | 338.355 | 385.633 | 565.661 | 1,206 | 0 | 1 |
| duckdb | m | 9,850,258 | 1 | read | 62 | 1.219 | 496.008 | 497.443 | 497.443 | 6.0 | 0 | 1 |
| duckdb | m | 9,850,258 | 8 | read | 472 | 1.588 | 506.754 | 509.463 | 517.167 | 46.5 | 0 | 1 |
| duckdb | m | 9,850,258 | 32 | read | 1120 | 2.859 | 838.959 | 846.750 | 901.919 | 109.9 | 0 | 1 |
| duckdb | m | 9,850,258 | 64 | read | 1150 | 9.844 | 2,063 | 2,501 | 2,603 | 104.0 | 0 | 1 |
| duckdb | m | 9,850,258 | 128 | read | 1180 | 33.738 | 3,672 | 4,156 | 4,218 | 101.6 | 0 | 1 |
| sqlite-ext | s | 984,803 | 1 | read | 797 | 0.147 | 37.908 | 38.116 | 39.247 | 79.4 | 0 | 1 |
| sqlite-ext | s | 984,803 | 8 | read | 6220 | 0.149 | 38.820 | 39.055 | 40.287 | 619.6 | 0 | 1 |
| sqlite-ext | s | 984,803 | 32 | read | 14137 | 0.236 | 67.877 | 68.661 | 139.083 | 1,405 | 0 | 1 |
| sqlite-ext | s | 984,803 | 64 | read | 14024 | 0.233 | 168.802 | 208.719 | 278.308 | 1,390 | 0 | 1 |
| sqlite-ext | s | 984,803 | 128 | read | 14048 | 0.241 | 348.606 | 417.555 | 488.853 | 1,382 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 1 | read | 62 | 0.606 | 484.710 | 488.314 | 488.314 | 6.1 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 8 | read | 496 | 0.713 | 496.333 | 498.205 | 507.335 | 47.9 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 32 | read | 1161 | 1.178 | 840.448 | 848.237 | 865.542 | 109.7 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 64 | read | 1157 | 1.187 | 2,083 | 2,559 | 2,826 | 105.7 | 0 | 1 |
| sqlite-ext | m | 9,850,258 | 128 | read | 1210 | 1.330 | 3,984 | 4,392 | 4,799 | 102.2 | 0 | 1 |

## Correctness checks

104 PASS, 0 FAIL, 0 SKIP; 8 backend check runs OK, 0 FAILED (all FAIL/FAILED lines, with annotations, in checks.txt)