# mentat scale benchmark: after

## Environment (abridged; full detail in env.txt)

```
instance_type:   c7i.8xlarge
kernel:          6.12.103-129.197.amzn2023.x86_64
cpu_model:       Intel(R) Xeon(R) Platinum 8488C
cpus:            24
sockets/numa:    1 1 
mem_total_gib:   61.8
hugepages:       HugePages_Total:=0 HugePages_Free:=0 Hugepagesize:=2048 
data_fs:         /dev/nvme0n1p1 xfs   150G   22G  129G  15% /
mentat_git:      6fed515a
duckdb:          n/a
postgres:        n/a
```

## Dataset and PG size vs shared_buffers

```
s: 984803 datoms (1600 users, 130000 issues, 6492 history updates)
m: 9850258 datoms (16000 users, 1300000 issues, 64986 history updates)
```

## bulk_load

| backend | scale | datoms | wall s | datoms/s | ANALYZE s | VACUUM s | store size | note |
|---|---|---:|---:|---:|---:|---:|---:|---|
| embedded | s | 984,803 | 86.6 | 11,373 |  |  | 0.20 GiB |  |
| embedded | m | 9,850,258 | 2,041 | 4,825 |  |  | 2.00 GiB |  |

## point_lookup

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 512356 | 0.019 | 0.021 | 0.025 | 0.381 | 51,236 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 508308 | 0.019 | 0.022 | 0.025 | 0.110 | 50,831 | 0 | 3 |

## ref_traversal

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 450 | 22.575 | 23.163 | 23.340 | 23.643 | 44.9 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 35 | 290.191 | 307.288 | 317.436 | 317.436 | 3.4 | 0 | 3 |

## aggregate

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 321 | 31.214 | 31.567 | 31.763 | 32.356 | 32.0 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 30 | 371.578 | 373.067 | 373.500 | 373.500 | 2.7 | 0 | 3 |

## predicate_scan

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 417 | 24.003 | 24.215 | 24.381 | 25.094 | 41.6 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 37 | 275.251 | 277.258 | 278.134 | 278.134 | 3.6 | 0 | 3 |

## pull

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 271614 | 0.036 | 0.041 | 0.050 | 0.396 | 27,161 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 226901 | 0.044 | 0.052 | 0.061 | 0.236 | 22,690 | 0 | 3 |

## as_of

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 114 | 88.058 | 88.805 | 89.195 | 89.366 | 11.4 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 30 | 1,040 | 1,046 | 1,048 | 1,048 | 1.0 | 0 | 3 |

## since

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 2403 | 4.159 | 4.259 | 4.303 | 4.661 | 240.2 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 191 | 52.509 | 52.943 | 54.645 | 55.074 | 19.0 | 0 | 3 |

## input_bindings

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 25631 | 0.383 | 0.429 | 0.438 | 0.509 | 2,563 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 4675 | 2.125 | 2.277 | 2.348 | 2.531 | 467.5 | 0 | 3 |

## write_mixed

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | write | 17647 | 3.392 | 3.815 | 4.040 | 22.434 | 294.1 | 0 | 3 |
| embedded | s | 984,803 | 8 | read | 34668 | 10.211 | 30.983 | 41.230 | 110.381 | 577.5 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | write | 15646 | 3.410 | 6.973 | 8.611 | 25.330 | 260.8 | 0 | 3 |
| embedded | m | 9,850,258 | 8 | read | 2234 | 6.323 | 585.528 | 1,149 | 2,076 | 36.9 | 0 | 3 |

## concurrency_sweep

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 1367 | 0.114 | 22.847 | 23.184 | 23.586 | 136.6 | 0 | 3 |
| embedded | s | 984,803 | 8 | read | 9986 | 0.081 | 24.433 | 24.735 | 26.542 | 996.3 | 0 | 3 |
| embedded | s | 984,803 | 32 | read | 21040 | 0.101 | 54.502 | 64.187 | 94.136 | 2,095 | 0 | 3 |
| embedded | s | 984,803 | 64 | read | 21084 | 0.103 | 115.099 | 144.955 | 175.393 | 2,093 | 0 | 3 |
| embedded | s | 984,803 | 128 | read | 21260 | 0.101 | 235.201 | 284.737 | 385.131 | 2,101 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 108 | 0.162 | 300.034 | 305.347 | 306.313 | 10.6 | 0 | 3 |
| embedded | m | 9,850,258 | 8 | read | 742 | 0.168 | 324.908 | 330.539 | 396.560 | 72.3 | 0 | 3 |
| embedded | m | 9,850,258 | 32 | read | 1639 | 0.176 | 661.280 | 701.176 | 765.435 | 157.1 | 0 | 3 |
| embedded | m | 9,850,258 | 64 | read | 1725 | 0.148 | 1,543 | 1,648 | 1,708 | 161.0 | 0 | 3 |
| embedded | m | 9,850,258 | 128 | read | 1792 | 0.158 | 2,814 | 3,005 | 3,279 | 160.4 | 0 | 3 |

## cold_vs_warm

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | cold_aggregate | 1 | 41.181 | 41.181 | 41.181 | 41.181 | 24.3 | 0 | 1 |
| embedded | s | 984,803 | 1 | cold_point_lookup | 1 | 8.894 | 8.894 | 8.894 | 8.894 | 112.4 | 0 | 1 |
| embedded | s | 984,803 | 1 | cold_pull | 1 | 4.608 | 4.608 | 4.608 | 4.608 | 217.0 | 0 | 1 |
| embedded | s | 984,803 | 1 | cold_ref_traversal | 1 | 523.707 | 523.707 | 523.707 | 523.707 | 1.9 | 0 | 1 |
| embedded | s | 984,803 | 1 | open | 1 | 44.886 | 44.886 | 44.886 | 44.886 | 22.3 | 0 | 1 |
| embedded | s | 984,803 | 1 | warm_aggregate | 30 | 31.196 | 31.599 | 31.667 | 31.667 | 32.0 | 0 | 1 |
| embedded | s | 984,803 | 1 | warm_point_lookup | 30 | 0.020 | 1.126 | 1.385 | 1.385 | 7,225 | 0 | 1 |
| embedded | s | 984,803 | 1 | warm_pull | 30 | 1.416 | 4.676 | 5.348 | 5.348 | 507.9 | 0 | 1 |
| embedded | s | 984,803 | 1 | warm_ref_traversal | 30 | 24.840 | 98.729 | 132.994 | 132.994 | 27.9 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | cold_aggregate | 1 | 4,984 | 4,984 | 4,984 | 4,984 | 0.2 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | cold_point_lookup | 1 | 12.646 | 12.646 | 12.646 | 12.646 | 79.1 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | cold_pull | 1 | 1.421 | 1.421 | 1.421 | 1.421 | 703.8 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | cold_ref_traversal | 1 | 4,941 | 4,941 | 4,941 | 4,941 | 0.2 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | open | 1 | 50.201 | 50.201 | 50.201 | 50.201 | 19.9 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | warm_aggregate | 30 | 374.378 | 375.532 | 375.533 | 375.533 | 2.7 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | warm_point_lookup | 30 | 1.080 | 2.573 | 2.592 | 2.592 | 928.5 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | warm_pull | 30 | 5.078 | 8.393 | 8.643 | 8.643 | 186.0 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | warm_ref_traversal | 30 | 392.530 | 477.170 | 520.620 | 520.620 | 2.5 | 0 | 1 |

## Correctness checks

52 PASS, 0 FAIL, 0 SKIP; 4 backend check runs OK, 0 FAILED (all FAIL/FAILED lines, with annotations, in checks.txt)