# Phase 2 Benchmark Results

Results from `benchmarks/phase2/run.sh` at commit `3b9a5d1` on
branch `feat/phase-0-honesty-and-build`.

**This is a developer-laptop run, not a hermetic benchmark.**
See [Caveats](#caveats) before citing any number.

## Environment

| Parameter | Value |
|-----------|-------|
| Date | 2026-05-10T02:06:05Z |
| Host | arnold |
| Kernel | 7.0.4-200.fc44.x86_64 |
| CPU | 12th Gen Intel Core i9-12900H (20 cores) |
| Memory | 31 GB |
| PostgreSQL | 16.13 (pgrx-managed, default config) |
| pg_mentat commit | 3b9a5d1 |
| Branch | feat/phase-0-honesty-and-build |

## Dataset

Issue-tracker workload generated by `gen_dataset.py` (seed=42):

| Scenario | Users | Issues | Labels | Datoms |
|----------|-------|--------|--------|--------|
| 100K | 200 | 16,000 | 50 | 120,638 |
| 300K | 600 | 49,000 | 100 | 368,871 |
| 1M | 2,000 | 162,000 | 200 | 1,219,419 |

## Queries

| ID | Description | Datalog |
|----|-------------|---------|
| Q1 | Point lookup: find user by email | `[:find ?e ?name :where [?e :user/email "..."] [?e :user/name ?name]]` |
| Q2 | Ref traversal: issues assigned to user | `[:find ?i ?title ?state :where [?u :user/email "..."] [?i :issue/assignee ?u] ...]` |
| Q3 | Aggregate: count issues per state | `[:find ?state (count ?i) :where [?i :issue/state ?state]]` |
| Q4 | Predicate: open high-priority issues | `[:find ?i ?title ?priority :where [?i :issue/state :state/open] [?i :issue/priority ?priority] [(>= ?priority 4)]]` |

## Results

Raw CSV: [`benchmarks/results/phase2-2026-05-10T020605Z/timings.csv`](../../benchmarks/results/phase2-2026-05-10T020605Z/timings.csv)

### Load Time

| Scenario | Datoms | mentat (ms) | EAV baseline (ms) | Ratio |
|----------|--------|-------------|-------------------|-------|
| 100K | 120,638 | 112,588 | 2,656 | 42x |
| 300K | 368,871 | 360,683 | 7,940 | 45x |
| 1M | 1,219,419 | 988,069 | 25,311 | 39x |

The ~40x load-time ratio reflects the cost of Mentat's transactor:
schema validation, tempid resolution, uniqueness enforcement, and
tx-log bookkeeping. Bulk-load is not the target workload; this
number sets a floor for future optimization.

### Query Latency (30 repetitions, warm cache)

#### 100K datoms (120,638)

| Query | Engine | p50 (ms) | p95 (ms) | p99 (ms) |
|-------|--------|----------|----------|----------|
| Q1 point lookup | mentat | 9.19 | 14.23 | 15.07 |
| Q1 point lookup | EAV baseline | 2.97 | 4.63 | 5.75 |
| Q2 ref traversal | mentat | 12.59 | 17.48 | 22.14 |
| Q2 ref traversal | EAV baseline | 5.46 | 8.70 | 9.75 |
| Q3 aggregate | mentat | 17.75 | 22.52 | 25.79 |
| Q3 aggregate | EAV baseline | 4.86 | 9.20 | 9.60 |
| Q4 predicate | mentat | 22.49 | 31.31 | 32.19 |
| Q4 predicate | EAV baseline | 11.89 | 14.80 | 18.57 |

#### 300K datoms (368,871)

| Query | Engine | p50 (ms) | p95 (ms) | p99 (ms) |
|-------|--------|----------|----------|----------|
| Q1 point lookup | mentat | 12.18 | 18.57 | 22.19 |
| Q1 point lookup | EAV baseline | 4.16 | 4.83 | 4.95 |
| Q2 ref traversal | mentat | 14.67 | 19.27 | 22.93 |
| Q2 ref traversal | EAV baseline | 5.80 | 8.98 | 9.96 |
| Q3 aggregate | mentat | 46.37 | 52.71 | 54.23 |
| Q3 aggregate | EAV baseline | 9.94 | 11.95 | 13.39 |
| Q4 predicate | mentat | 48.12 | 54.00 | 60.74 |
| Q4 predicate | EAV baseline | 25.54 | 32.47 | 33.68 |

#### 1M datoms (1,219,419)

| Query | Engine | p50 (ms) | p95 (ms) | p99 (ms) |
|-------|--------|----------|----------|----------|
| Q1 point lookup | mentat | 15.69 | 19.22 | 21.62 |
| Q1 point lookup | EAV baseline | 3.48 | 5.28 | 5.80 |
| Q2 ref traversal | mentat | 19.55 | 24.35 | 29.08 |
| Q2 ref traversal | EAV baseline | 5.31 | 8.13 | 8.71 |
| Q3 aggregate | mentat | 124.78 | 131.36 | 133.38 |
| Q3 aggregate | EAV baseline | 20.06 | 23.90 | 24.70 |
| Q4 predicate | mentat | 129.51 | 136.68 | 142.05 |
| Q4 predicate | EAV baseline | 55.77 | 61.96 | 64.32 |

### Overhead Ratios (mentat p50 / EAV p50)

| Query | 100K | 300K | 1M | Trend |
|-------|------|------|-----|-------|
| Q1 point lookup | 3.1x | 2.9x | 4.5x | Worsens due to seq scan growing with table size |
| Q2 ref traversal | 2.3x | 2.5x | 3.7x | Moderate growth; join cost dominates |
| Q3 aggregate | 3.7x | 4.7x | 6.2x | Linear scan — ratio worsens proportionally |
| Q4 predicate | 1.9x | 1.9x | 2.3x | Most stable; both engines scan similarly |

## Analysis

The overhead is dominated by two factors visible in the EXPLAIN plans:

1. **Schema-ident resolution (InitPlan):** Every attribute reference
   in a Datalog pattern becomes an `InitPlan` subquery against
   `mentat.schema` to resolve the keyword to an entid. The EAV
   baseline hardcodes integer attribute IDs, bypassing this. Each
   InitPlan adds ~0.01ms (negligible individually, but they
   serialize plan startup).

2. **Missing EAVT index on datoms_text_new for point lookups:**
   Q1's plan shows a **Seq Scan** with filter on `datoms_text_new`
   (cost=514.88) instead of using `idx_datoms_text_new_eavt`. This
   is because the filter combines `store_id`, `a`, and `v` but the
   AEVT index is `(store_id, a, e, v)` — the query needs `(a, v)`
   without `e`, which doesn't match the index prefix. A dedicated
   `(store_id, a, v)` index on text would collapse Q1 to an index
   scan.

3. **Function-call overhead:** The `mentat_query()` PL function
   parses EDN, compiles Datalog to SQL, and executes via SPI. The
   EAV baseline runs pre-written SQL directly. This fixed overhead
   (~5-8ms) dominates at small result sets.

## EXPLAIN Plans

Full plans are in
[`benchmarks/results/phase2-2026-05-10T010456Z/plans/`](../../benchmarks/results/phase2-2026-05-10T010456Z/plans/).

### Q1 Point Lookup (mentat) — abbreviated

```
Unique
  InitPlan 1: Index Scan schema_ident_key (':user/email' → entid)
  InitPlan 2: Index Scan schema_ident_key (':user/name' → entid)
  → Nested Loop
      → Seq Scan on datoms_text_new (Filter: store_id=0 AND a=$0 AND v='...')
      → Index Scan idx_datoms_text_new_aevt (e = outer.e AND a=$1)
```

### Q1 Point Lookup (EAV baseline)

```
Nested Loop (actual time=0.054..0.067 rows=1)
  → Bitmap Heap Scan on eav.text (a=1000, v='user100000@...')
      → Bitmap Index Scan on eav_text_aevt (0.028ms)
  → Index Scan on eav.text (a=1001, e=outer.e)
Planning Time: 1.399 ms
Execution Time: 0.109 ms
```

### Q4 Predicate (EAV baseline)

```
Nested Loop (actual time=1.696..7.494 rows=1313)
  → Nested Loop
      → Hash Join (s.e = s_1.e) — keyword state/open
          → Bitmap Heap Scan eav.keyword (v=':state/open', a=1003)
          → Hash: Bitmap Heap Scan eav.keyword (same filter, 3272 rows)
      → Index Scan eav.long (e=s.e, a=1004, Filter: v >= 4)
  → Index Scan eav.text (e=p.e, a=1002)
Planning Time: 6.339 ms
Execution Time: 7.713 ms
```

## Scaling Behavior

| Query | 100K→300K (3x data) | 300K→1M (3.3x data) |
|-------|---------------------|---------------------|
| Q1 mentat | 1.3x slower | 1.3x slower |
| Q2 mentat | 1.2x slower | 1.3x slower |
| Q3 mentat | 2.6x slower | 2.7x slower |
| Q4 mentat | 2.1x slower | 2.7x slower |

Q1/Q2 (point lookups) scale sub-linearly — the seq-scan cost grows
but the fixed function-call overhead dominates at these sizes. Q3/Q4
(full-table scans) scale nearly linearly with data volume, confirming
the scan-driven execution path.

## Caveats

1. **Laptop, not a benchmark box.** Intel i9-12900H with thermal
   throttling. Run-to-run variance is visible.
2. **pgrx-managed Postgres.** `shared_buffers`, `work_mem`,
   `effective_cache_size` are all defaults.
3. **Single connection, no concurrency.**
4. **One workload family.** Issue-tracker schema; YMMV.
5. **The EAV baseline is intentionally dumb.** A plain
   `(entity, attribute, value)` table with EAVT/AEVT/VAET indexes.
   It isolates the overhead pg_mentat adds on top of an equivalent
   storage layout; it is not an optimised Datalog implementation.
6. **Load time includes full transactor path.** The EAV baseline
   uses raw SQL INSERT; Mentat transacts with schema validation,
   tempid resolution, and uniqueness enforcement.

## Reproduce

```bash
CARGO_HOME=$HOME/.cargo bash benchmarks/phase2/run.sh
```

Results appear in `benchmarks/results/phase2-<timestamp>/`.

## CPU Flamegraph

A flamegraph captured via `benchmarks/phase2/perf_capture.sh` is
included for reference at:

[`benchmarks/results/phase2-perf-2026-05-13T124318Z/flame.svg`](../../benchmarks/results/phase2-perf-2026-05-13T124318Z/flame.svg)

The captured run is small-scale (7,574 datoms, 10s window, 300
outer-loop iterations) since it exists primarily to demonstrate
that the methodology works on this hybrid Alder Lake CPU
(`--call-graph fp -e cpu_core/cycles/` is required; `--call-graph
dwarf` produces zero samples). Hot frames in this capture, in
rough order:

1. `randomize_mem` (PostgreSQL's `palloc` / memory-context churn)
2. `set_var_from_str` (numeric / text input parsing in the
   value-decode CASE expression)
3. `ExecInterpExpr` (per-row expression evaluation for the
   value_type_tag CASE in the SELECT projection)
4. `pushJsonbValueScalar` / `serialize_str` (JSON encoding of the
   query result)
5. parking-lot `read_guard` on the schema-cache `HashMap`

None of these are the Datalog ↔ SQL translator itself; they are
all in PostgreSQL or in the result-encoding path. That tells the
same story the EAV-vs-mentat numbers tell at the macro level: the
Datalog compilation is cheap, the row-by-row CASE projection and
the JSON encoding are not. A larger-scale flamegraph (300K-1M
datoms) would surface the same hot frames at higher counts, not
different ones.

To regenerate at production scale (~360K datoms, 30s window, 1000
iters):

```
CARGO_HOME=$HOME/.cargo_pg_mentat bash benchmarks/phase2/perf_capture.sh
```

Expect ~10–12 minutes wall time (load is the long phase).

## What's Next

- Add `(store_id, a, v)` indexes on text/keyword narrow tables
  to eliminate the seq-scan bottleneck in Q1/Q3.
- Cache schema-ident → entid mappings in the query compiler to
  eliminate InitPlan subqueries.
- Run at 10M datoms to confirm scaling trends hold.
- Hermetic environment (dedicated host, pinned frequency, no
  background processes) for reproducible numbers.
- Production-scale CPU flamegraph (the small-scale one above
  proves the harness works; the 1M-datom one is dedicated-hardware
  work).