# Benchmark Results — multi-host, multi-run

This document records measured benchmark numbers with proper
statistical methodology.  Per-host: 11 outer runs, each binary
internally does 5 inner trials of 100,000 iterations.  CPU pinned
on FreeBSD via `cpuset -l 0`; M3 Pro runs unpinned because macOS
does not expose stable per-core pinning.

## Hosts

### nuc — Intel i7-8809G, FreeBSD 15.0-RELEASE-p1

  * **CPU:** Intel Core i7-8809G @ 3.10 GHz, 4 cores / 8 threads,
    32 KB L1d, 256 KB L2, 8 MB L3.  AVX2.
  * **OS:** FreeBSD 15.0-RELEASE-p1
  * **Compiler:** clang 19.1.7 (FreeBSD ports)
  * **simdjson:** 4.5.0 (FreeBSD ports)
  * **Allocator:** jemalloc (FreeBSD default)
  * **JIT:** enabled (LLVM 19)
  * **Load:** load avg < 1, 97 % idle, dedicated benchmark machine.
  * **Pinning:** `cpuset -l 0`

### m3pro — Apple M3 Pro, macOS 26.4.1

  * **CPU:** Apple M3 Pro (Mac15,7), NEON.
  * **OS:** macOS 26.4.1
  * **Compiler:** clang 21.1.8 (nix LLVM 21.1.8 toolchain)
  * **simdjson:** 4.6.4 (Homebrew)
  * **Allocator:** macOS libc malloc
  * **JIT:** enabled (LLVM 21.1.8)
  * **Pinning:** none (macOS doesn't expose stable per-core pinning)

### rv — Ky(R) X1, Ubuntu 24.04 / RISC-V

  * **CPU:** Ky(R) X1 @ 1.60 GHz max, 8 cores / 8 threads
    (no SMT), 32 KB L1d, 32 KB L1i, 1 MB L2 (per cluster).
    riscv64.  No SIMD ISA exposed.
  * **OS:** Linux 6.6.63-ky, Ubuntu 24.04.4 LTS
  * **Compiler:** gcc 13.3.0 (system) and clang 20.1.2 (Ubuntu)
  * **simdjson:** 3.6.4 (Ubuntu, fetched via `apt-get download`
    + dpkg-deb -x into a userspace prefix because we don't have
    sudo on this host)
  * **Allocator:** glibc malloc (ptmalloc2)
  * **JIT:** **disabled** at configure time (LLVM dev headers
    not available; `-Dllvm=disabled` selected).  All "lime+JIT"
    rows below fall through to the interpreter on this host.
  * **Pinning:** `taskset -c 0`

### icarus — Sun UltraSPARC, OpenIndiana 2026.04 / SPARC v9 (big-endian)

  * **CPU:** SPARC v9 @ 1.6 GHz, 2 virtual processors,
    SUNW,A70.  64-bit big-endian.  VIS / VIS2 SIMD instructions
    (not used by Lime; SIMD tokenizer falls back to scalar
    portable path).
  * **OS:** OpenIndiana Hipster 2026.04, illumos
    `illumos-31d3d510d0`.
  * **Compilers tested:** gcc 14.2 (system) and clang 21.1.8
    (OpenIndiana ports).  clang requires `--buildtype=release`
    on this host (debug DWARF triggers SPARC-specific
    relocation errors at link time).
  * **simdjson:** not installed; `bench_simdjson_compare`
    skipped at configure time.
  * **Allocator:** Solaris libc malloc.
  * **JIT:** disabled (LLVM dev not bound for the build); same
    skip semantics as rv.
  * **Pinning:** none (no per-core taskset analogue exposed in
    this OpenIndiana userland).

### santorini — Win11 ARM64 / MSVC

  * **CPU:** Qualcomm Snapdragon (ARMv8 Family 8 Model D4B, ARM64).
  * **OS:** Windows 11 (NT 10.0.26200.0).
  * **Compiler:** MSVC 14.50.35726 (Visual Studio 18 Community,
    Host=ARM64 / target=ARM64).
  * **simdjson:** not installed; bench_simdjson_compare is
    skipped at configure time.
  * **JIT:** disabled (`-Dllvm=disabled`); LLVM not installed
    on this host.
  * **lime_snapshot_create / lime_compile_grammar_text /
    lime_snapshot_extend** are stub-NULL on Windows in v0.2.x
    (the fork/exec/dlopen primitives have no Windows analog;
    the static-parser shape -- Shape A in
    docs/INTEGRATING_LIME.md -- works on every platform and is
    the right path on Windows for now).
  * **Tests:** 70 OK / 0 Fail / 1 Skipped (`jit_parse_equivalence`
    skipped because LLVM/JIT is disabled).  The 4 PBT files
    (`tests/test_*_pbt.c`) and the 2 lex-internals tests that
    use POSIX `pipe` / `open_memstream` are configure-skipped
    on this platform.
  * **Pinning:** none (no equivalent of cpuset on Windows).

The nuc and rv are the trustworthy hosts: dedicated, mostly idle,
predictable schedulers.  icarus is similarly dedicated.  The M3
Pro is a developer laptop; its per-run stddev is **5-17× larger**
than the nuc/rv on the same metric.  This is exactly why the user
routes long-running benchmarks elsewhere.

## Reproduce

```bash
# In a clone of the repo, on a quiet machine:
meson setup builddir -Dpg_grammar=false
ninja -C builddir
chmod +x bench-runner.sh
./bench-runner.sh 11 0 arith \
    ./builddir/bench/bench_flex_bison_compare/bench_flex_bison_compare
./bench-runner.sh 11 0 simdjson \
    ./builddir/bench/bench_simdjson_compare/bench_simdjson_compare
# The flex_bison_compare binary writes both arith and json sections
# to each run file; alias them so the stats parser reads both.
for f in bench-runs/arith-*.txt; do
    cp "$f" "bench-runs/jsonarith-${f##*arith-}"
done
python3 bench-stats.py bench-runs/
```

`bench-runner.sh` and `bench-stats.py` live at the project root.
The runner pins to the requested CPU via `cpuset` / `taskset`
when available and captures per-run output under `bench-runs/`.
The stats script parses the per-tool min/mean / per-mode (ms,
MB/s, docs/s) lines and emits min / median / mean / p95 / max /
stddev across the runs.

## Results — arithmetic grammar (parser only on Lime side)

11 outer runs of 5 inner trials of 100,000 parses of
`(1 + 2) * (3 + 4) - 5` (13 tokens).  Lime side is hand-fed
pre-tokenized input; Bison side runs through Flex.  This is
parser-only on the Lime side, so the absolute numbers
under-count Lime's full lex+parse cost; the JSON benchmark
below is the fair end-to-end comparison.

### nuc — Intel x86 / FreeBSD

| metric (ms)             |    min |    med |   mean |    p95 |    max |    std |
|-------------------------|-------:|-------:|-------:|-------:|-------:|-------:|
| bison min               |  35.25 |  35.30 |  35.48 |  36.08 |  36.48 |   0.34 |
| **lime min**            |  33.96 |  33.99 |  34.06 |  34.27 |  34.37 |   0.12 |
| lime+JIT min            |  37.79 |  37.85 |  37.94 |  38.31 |  38.42 |   0.20 |

Lime wins by 1.04× over Bison.  **JIT is 12 % slower than the
unrolled-switch interpreter on Intel x86 for this micro-grammar**
— stddev of 0.20 ms confirms this is real, not noise.

### m3pro — Apple Silicon / macOS

| metric (ms)             |    min |    med |   mean |    p95 |    max |    std |
|-------------------------|-------:|-------:|-------:|-------:|-------:|-------:|
| bison min               |  43.27 |  45.32 |  45.45 |  48.15 |  49.01 |   1.71 |
| lime min                |  37.79 |  41.26 |  41.36 |  45.12 |  47.45 |   2.47 |
| **lime+JIT min**        |  32.48 |  38.04 |  36.79 |  39.84 |  40.20 |   2.70 |

Lime wins by ~1.10× over Bison.  **JIT is 8 % faster than the
interpreter on Apple Silicon** — opposite of x86.  Stddev is 14×
larger here, but the JIT-faster effect is consistent across all
11 runs.

### rv — RISC-V / Ubuntu (no LLVM, JIT disabled)

| metric (ms)             |    min |    med |   mean |    p95 |    max |    std |
|-------------------------|-------:|-------:|-------:|-------:|-------:|-------:|
| bison min               | 260.83 | 261.14 | 261.27 | 261.95 | 262.11 |   0.39 |
| **lime min**            | 219.68 | 219.79 | 219.85 | 220.14 | 220.22 |   0.16 |

Lime wins by **1.19×** over Bison.  Tightest stddev of any host
(0.16 ms = 0.07 % relative).  No JIT row because LLVM dev
headers were not available on this host; the JIT path was
disabled at configure time.

## Results — JSON benchmark (lex + parse, fair on both sides)

The same 515-byte JSON document parsed 100,000 times per inner
trial, both sides running their own lexer end-to-end.  This is
the apples-to-apples number.

### nuc — Intel x86 / FreeBSD

| metric (ms)               |    min |    med |   mean |    p95 |    max |   std |
|---------------------------|-------:|-------:|-------:|-------:|-------:|------:|
| bison_json min            | 310.25 | 311.57 | 312.10 | 314.65 | 314.78 |  1.70 |
| **lime_json min**         | 256.30 | 256.45 | 257.07 | 259.20 | 260.12 |  1.16 |
| lime_json+JIT min         | 277.28 | 278.97 | 278.85 | 280.23 | 280.35 |  1.01 |

Lime+Flex-equivalent is **1.21× faster** than Bison+Flex
(median 256.45 vs 311.57 ms).  JIT again 8.6 % slower than the
interpreter on Intel.

### m3pro — Apple Silicon / macOS

| metric (ms)               |    min |    med |   mean |    p95 |    max |   std |
|---------------------------|-------:|-------:|-------:|-------:|-------:|------:|
| bison_json min            | 444.23 | 474.62 | 477.23 | 510.86 | 524.20 | 22.22 |
| lime_json min             | 205.37 | 258.62 | 252.79 | 290.84 | 292.21 | 26.11 |
| **lime_json+JIT min**     | 216.15 | 236.14 | 243.18 | 272.06 | 272.93 | 19.74 |

Lime+JIT is **2.01× faster** than Bison on M3 Pro (median 236.14
vs 474.62 ms).  Again the JIT helps on aarch64 and hurts on
Intel.

### rv — RISC-V / Ubuntu (no JIT)

| metric (ms)               |    min |    med |   mean |    p95 |    max |   std |
|---------------------------|-------:|-------:|-------:|-------:|-------:|------:|
| bison_json min            | 1976.85 | 1980.91 | 1980.35 | 1983.89 | 1984.49 |  2.64 |
| **lime_json min**         | 1425.02 | 1434.16 | 1434.34 | 1443.03 | 1445.85 |  6.16 |

Lime wins by **1.38× over Bison** — the largest gap of any host.
Likely cause: Bison's pull-parser-with-Flex pattern has more
function-call overhead than Lime's push parser, and on a
narrower-issue RISC-V core (1.6 GHz, no SIMD) that overhead
dominates more.

### icarus — SPARC v9 big-endian / OpenIndiana (no simdjson)

5-run sample on the arith parser-only benchmark:

| toolchain    | min (ms) | mean (ms) | std (ms) |
|--------------|---------:|----------:|---------:|
| gcc 14.2     |   365.14 |    374.15 |     ~10  |
| clang 21.1.8 |   313.74 |    317.43 |     ~2   |

This is the slowest host of the four (~10× the nuc on the same
benchmark) which is consistent with 1.6 GHz vs 3.1 GHz, narrower
SPARC issue width, and big-endian byte order forcing the action-
table loads through endianness-aware accessors.  No JIT and no
simdjson, but the FreeBSD/Linux unrolled-switch dispatch path
works correctly on the platform — exactly the portability check
the user wanted.

## Results — Lime+JIT vs simdjson (steady state, three Lime alloc modes)

11 outer runs, each with 50,000 iterations of warmup + 5 inner
trials of 100,000 parses on a 515-byte JSON document.  All three
Lime modes are JIT-armed; the only difference is the allocator.

### nuc

| mode                              |  median ms | median MB/s | median docs/s |
|-----------------------------------|-----------:|------------:|--------------:|
| lime+JIT malloc                   |     494.00 |        99.4 |       202,419 |
| lime+JIT malloc-leak (no free)    |     571.00 |        86.0 |       175,129 |
| **lime+JIT arena** (zero alloc)   |     381.40 |       128.8 |       262,179 |
| **simdjson ondemand**             |      29.10 |     1,690.1 |     3,441,208 |

Stddev is < 10 ms across all four modes.

### m3pro

| mode                              |  median ms | median MB/s | median docs/s |
|-----------------------------------|-----------:|------------:|--------------:|
| lime+JIT malloc                   |     693.70 |        70.8 |       144,154 |
| lime+JIT malloc-leak (no free)    |     573.60 |        85.6 |       174,329 |
| **lime+JIT arena** (zero alloc)   |     431.70 |       113.8 |       231,659 |
| **simdjson ondemand**             |      35.80 |     1,372.9 |     2,795,219 |

### rv (no JIT — interpreter only)

| mode                              |  median ms | median MB/s | median docs/s |
|-----------------------------------|-----------:|------------:|--------------:|
| lime malloc                       |    5438.00 |         9.0 |        18,389 |
| lime malloc-leak (no free)        |    6241.80 |         7.9 |        16,021 |
| **lime arena** (zero alloc)       |    2997.10 |        16.4 |        33,366 |
| **simdjson ondemand**             |     536.50 |        91.5 |       186,387 |

(JIT was disabled at configure time on this host since LLVM dev
headers weren't available; "lime+JIT" labels emitted by the bench
binary are misnomers here -- they all fall through to the
interpreter when `lime_jit_compile` returns non-zero.)

## The two surprising findings

### 1. JIT direction differs by architecture

  * **Apple Silicon (aarch64):** JIT is faster than the
    interpreter, both on the small arith grammar (1.04×) and
    the medium JSON grammar (1.10×).
  * **Intel (x86 / FreeBSD):** JIT is **slower** than the
    interpreter on both grammars (-12 % on arith, -8.6 % on
    JSON).  Stddev is tight (< 0.5 % relative); this is not
    measurement noise.

The likely cause is that Intel branch prediction is better at
walking the C interpreter's compact code than the LLVM ORC JIT's
generated dispatch stub.  Apple Silicon's wider front-end and
different branch predictor architecture benefits more from
LLVM's specialisation.  Whatever the cause, the practical
implication is that the JIT default behaviour should probably
gate on `__aarch64__` or do a startup micro-bench on first use,
rather than always-on.

### 2. Leak mode is slower than malloc on FreeBSD/jemalloc

| mode                | nuc (ms) | m3pro (ms) |
|---------------------|---------:|-----------:|
| malloc (free works) |    494   |    694     |
| malloc-leak (no free) | 571 (slower!) | 574 (faster) |

On FreeBSD's jemalloc, leaking memory is **15 % slower** than
calling `free()`.  On macOS libc malloc, leaking is **17 %
faster**.  The reason: jemalloc's free-list reuse — calling
`free` immediately recycles the slab into the allocator's
thread cache, so the next allocation hits a hot cache line.
Without `free()`, every iteration burns through fresh memory,
the working set grows unbounded across 100 K iterations, TLB
misses dominate.

The platform-portable answer is **arena allocation**, which is
fastest on both platforms.

## Cross-host shape comparison (median values)

|                      | nuc (Intel) | m3pro (Apple) | rv (RISC-V) |
|----------------------|------------:|--------------:|------------:|
| simdjson median      |   29.1 ms   |   35.8 ms     |  536.5 ms   |
| lime arena           |  381.4 ms (13.1×) | 431.7 ms (12.1×) | 2997 ms ( 5.6×) |
| lime malloc          |  494.0 ms (17.0×) | 693.7 ms (19.4×) | 5438 ms (10.1×) |
| lime malloc-leak     |  571.0 ms (19.6×) | 573.6 ms (16.0×) | 6242 ms (11.6×) |

Cross-host signal:

  * **simdjson is consistently 12-13× faster than Lime+arena on
    machines with SIMD** (Intel AVX2, Apple NEON).  On RISC-V
    where simdjson has no SIMD ISA to vectorise into (this build
    of simdjson 3.6.4 was compiled without RVV), the gap drops
    to **5.6×**.  That's our cleanest measurement of how much of
    simdjson's win is SIMD specifically: about 60 % of the gap
    on Intel/Apple is SIMD; the remaining 40 % is structural
    (lazy ondemand parsing, no per-value allocation, in-place
    iteration).
  * **Lime+arena is consistently the fastest of the Lime modes**
    on every platform tested.  The arena pattern (single buffer,
    reused, reset between parses) mirrors simdjson's own
    internal allocation model.
  * **Bison vs Lime ratio is consistent across all three hosts**
    (1.04-1.38× Lime advantage).  The win is robust across
    architectures, compilers, allocators, and SIMD availability.

## Methodology notes

  * **Statistical rigor:** 11 outer runs is enough to compute a
    stable median; we report min / med / mean / p95 / max /
    stddev so a reader can see distribution shape, not just one
    point.
  * **Pinning:** `cpuset -l 0` (FreeBSD) ensures every run hits
    the same physical core.  macOS doesn't expose stable
    pinning; on the M3 Pro we report higher stddev as honestly
    as we can.
  * **Warmup:** Each binary already does an internal warmup
    (50,000 iterations per side for the simdjson bench;
    5-trial aggregation in the bison-vs-lime bench).  The
    runner script additionally throws away one extra invocation
    up front to warm the page cache.
  * **What we did NOT do:** Disable Turbo Boost / SpeedStep,
    drop page caches between runs, or run with `nice -n -20`.
    Those would tighten the absolute numbers further but the
    stddev we see on the nuc (< 0.5 % on most metrics) suggests
    the marginal benefit is small for this workload.
  * **Why no GitHub Actions numbers:** CI runners are shared
    VMs with unpredictable neighbours.  We deliberately do not
    publish numbers from CI; the published numbers are from
    bare metal.

## Implications for the project

  * The user-visible default (Lime + malloc) is 1.04-1.21×
    faster than Bison on these workloads on x86 Intel and
    1.10-2.01× on Apple Silicon — same relative ordering across
    platforms.  The Bison comparison is robust.
  * The JIT helps on Apple Silicon but hurts on x86 Intel for
    grammars below the size threshold.  Consider a per-arch
    `IR_SIZE_THRESHOLD` in `src/jit_codegen.c` so x86 doesn't
    pay the JIT penalty on small grammars.  Tracking as a
    follow-up.
  * Switching the example's allocator from malloc-per-node to a
    single-arena bump allocator gives a 23-37 % speedup on
    end-to-end JSON parsing.  This matches simdjson's own
    design and is the right default for any production parser
    shape that processes many short documents.

## After the x86 perf round (commit `602c305`)

After landing the per-token hot-path optimisations (cached JIT
function pointer in `ParserSnapshot`, branch hints on the JIT
check, `static inline` on `find_shift_action` /
`find_reduce_action`, and architecture-conditional gate on
`lime_jit_compile` for x86 small grammars):

### nuc (x86 / FreeBSD), 11-run median min

| metric (ms)   | before | after  | delta   |
|---------------|-------:|-------:|--------:|
| lime arith    | 33.99  | 33.50  | -1.4 %  |
| lime JSON     | 256.45 | 243.53 | -5.0 %  |
| lime+JIT arith| 37.85  | (skipped) | -100 % cost (was a slowdown) |
| bison arith   | 35.30  | 36.80  | +4.2 %  (system noise; binary unchanged) |

The lime JSON improvement is significant: -5.0 % across 11 runs
(stddev 1.16 ms; the 13 ms delta is well outside noise).  The
JIT path is now skipped on x86 small grammars so the previous
+12 % penalty is gone -- code that always calls
`lime_jit_compile` now performs at the no-JIT level instead.

### m3pro (aarch64 / macOS), single-binary check

5-run median on the after-fix binary: lime arith 31.95 ms (min),
lime JSON 167.89 ms (min).  Earlier 11-run baseline was
lime arith 41.26 ms / lime JSON 258.62 ms; the M3 Pro is a
shared host so some of this is system-load drift, but the
direction is consistent and the arch-conditional gate is
inactive on aarch64 (JIT continues to run as before).