# mentat scale benchmark: before

## Environment (abridged; full detail in env.txt)

```
instance_type:   c7i.8xlarge
kernel:          6.12.103-129.197.amzn2023.x86_64
cpu_model:       Intel(R) Xeon(R) Platinum 8488C
cpus:            24
sockets/numa:    1 1 
mem_total_gib:   61.8
hugepages:       HugePages_Total:=0 HugePages_Free:=0 Hugepagesize:=2048 
data_fs:         /dev/nvme0n1p1 xfs   150G  7.6G  143G   6% /
mentat_git:      72ee3765
duckdb:          n/a
postgres:        n/a
```

## Dataset and PG size vs shared_buffers

```
s: 984803 datoms (1600 users, 130000 issues, 6492 history updates)
m: 9850258 datoms (16000 users, 1300000 issues, 64986 history updates)
```

## bulk_load

| backend | scale | datoms | wall s | datoms/s | ANALYZE s | VACUUM s | store size | note |
|---|---|---:|---:|---:|---:|---:|---:|---|
| embedded | s | 984,803 | 66.1 | 14,903 |  |  | 0.15 GiB |  |
| embedded | m | 9,850,258 | 1,482 | 6,646 |  |  | 1.57 GiB |  |

## point_lookup

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 139908 | 0.070 | 0.080 | 0.084 | 0.141 | 13,991 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 19058 | 0.515 | 0.572 | 0.586 | 0.684 | 1,906 | 0 | 3 |

## ref_traversal

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 234 | 42.842 | 43.410 | 43.561 | 43.768 | 23.3 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 30 | 556.256 | 562.578 | 562.993 | 562.993 | 1.8 | 0 | 3 |

## aggregate

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 99 | 101.769 | 102.549 | 104.314 | 104.314 | 9.8 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 30 | 1,352 | 1,370 | 1,373 | 1,373 | 0.8 | 0 | 3 |

## predicate_scan

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 391 | 25.595 | 25.829 | 26.009 | 26.451 | 39.0 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 36 | 279.930 | 281.160 | 281.895 | 281.895 | 3.6 | 0 | 3 |

## pull

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 236041 | 0.042 | 0.047 | 0.057 | 0.161 | 23,604 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 204778 | 0.048 | 0.055 | 0.065 | 0.173 | 20,478 | 0 | 3 |

## as_of

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 3 | 20,769 | 23,047 | 23,047 | 23,047 | 0.1 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | **CEILING** (first call) | 1 | 20,000 | 20,000 | 20,000 | 20,000 | 0.1 | 1 | 1 |

## since

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 207 | 48.442 | 48.657 | 49.328 | 49.654 | 20.6 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 30 | 484.891 | 486.573 | 487.344 | 487.344 | 2.1 | 0 | 3 |

## input_bindings

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 25965 | 0.378 | 0.423 | 0.432 | 0.507 | 2,596 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | read | 4918 | 2.026 | 2.111 | 2.153 | 2.252 | 491.7 | 0 | 3 |

## write_mixed

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | write | 15498 | 3.867 | 4.397 | 4.676 | 6.586 | 258.3 | 0 | 3 |
| embedded | s | 984,803 | 8 | read | 590 | 2.139 | 1,697 | 1,727 | 1,763 | 9.7 | 0 | 3 |
| embedded | m | 9,850,258 | 1 | write | 14219 | 4.170 | 5.007 | 5.586 | 22.136 | 237.0 | 0 | 3 |
| embedded | m | 9,850,258 | 8 | read | 48 | 4.225 | 23,940 | 24,083 | 24,083 | 0.7 | 0 | 3 |

## concurrency_sweep

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | read | 295 | 0.214 | 108.372 | 120.420 | 127.766 | 29.2 | 0 | 1 |
| embedded | s | 984,803 | 8 | read | 83 | 0.851 | 3,062 | 3,101 | 3,101 | 6.9 | 0 | 1 |
| embedded | s | 984,803 | 32 | read | 101 | 5.952 | 14,947 | 14,957 | 14,966 | 6.8 | 0 | 1 |
| embedded | s | 984,803 | 64 | **CEILING** (first call) | 1 | 20,002 | 20,002 | 20,002 | 20,002 | 0.1 | 20 | 1 |
| embedded | m | 9,850,258 | 1 | read | 30 | 0.631 | 1,938 | 1,946 | 1,946 | 1.6 | 0 | 1 |
| embedded | m | 9,850,258 | 8 | **CEILING** (first call) | 1 | 20,001 | 20,001 | 20,001 | 20,001 | 0.1 | 3 | 1 |

## cold_vs_warm

| backend | scale | datoms | clients | op | n | p50 ms | p95 ms | p99 ms | max ms | ops/s | errors | reps |
|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
| embedded | s | 984,803 | 1 | cold_aggregate | 1 | 123.480 | 123.480 | 123.480 | 123.480 | 8.1 | 0 | 1 |
| embedded | s | 984,803 | 1 | cold_point_lookup | 1 | 24.041 | 24.041 | 24.041 | 24.041 | 41.6 | 0 | 1 |
| embedded | s | 984,803 | 1 | cold_pull | 1 | 4.410 | 4.410 | 4.410 | 4.410 | 226.8 | 0 | 1 |
| embedded | s | 984,803 | 1 | cold_ref_traversal | 1 | 619.741 | 619.741 | 619.741 | 619.741 | 1.6 | 0 | 1 |
| embedded | s | 984,803 | 1 | open | 1 | 1,621 | 1,621 | 1,621 | 1,621 | 0.6 | 0 | 1 |
| embedded | s | 984,803 | 1 | warm_aggregate | 30 | 97.855 | 99.214 | 99.951 | 99.951 | 10.2 | 0 | 1 |
| embedded | s | 984,803 | 1 | warm_point_lookup | 30 | 0.140 | 1.014 | 1.065 | 1.065 | 2,406 | 0 | 1 |
| embedded | s | 984,803 | 1 | warm_pull | 30 | 2.168 | 3.212 | 4.113 | 4.113 | 483.6 | 0 | 1 |
| embedded | s | 984,803 | 1 | warm_ref_traversal | 30 | 47.580 | 60.582 | 61.416 | 61.416 | 20.3 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | cold_aggregate | 1 | 3,831 | 3,831 | 3,831 | 3,831 | 0.3 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | cold_point_lookup | 1 | 59.671 | 59.671 | 59.671 | 59.671 | 16.8 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | cold_pull | 1 | 5.785 | 5.785 | 5.785 | 5.785 | 172.9 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | cold_ref_traversal | 1 | 6,757 | 6,757 | 6,757 | 6,757 | 0.1 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | open | 1 | 16,499 | 16,499 | 16,499 | 16,499 | 0.1 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | warm_aggregate | 30 | 1,333 | 1,357 | 1,370 | 1,370 | 0.8 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | warm_point_lookup | 30 | 1.431 | 1.556 | 1.647 | 1.647 | 790.3 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | warm_pull | 30 | 4.189 | 6.211 | 6.615 | 6.615 | 230.7 | 0 | 1 |
| embedded | m | 9,850,258 | 1 | warm_ref_traversal | 30 | 576.792 | 592.832 | 608.399 | 608.399 | 1.7 | 0 | 1 |

## Correctness checks

46 PASS, 0 FAIL, 6 SKIP; 4 backend check runs OK, 0 FAILED (all FAIL/FAILED lines, with annotations, in checks.txt)