# Lime Rust Output vs the Rust Parsing Ecosystem

This document compares Lime's `--rust` + `--rustlex` outputs
against the most-used Rust-native parser/lexer crates on a
common JSON parsing benchmark.

## Setup

- **Machine:** Intel i9-12900H (24 thread, 4.7 GHz turbo), Linux, GCC 15.2 / rustc 1.95-nightly via Nix dev shell.
- **Fixture:** `bench/rust_compare/fixture.json` -- 226,620 bytes (~227 KB). Generated by `make_fixture.py` with seed=42; mixed nested objects, arrays, strings, numbers, booleans, nulls.
- **Method:** Each benchmark is a separate binary (`bench/rust_compare/src/bin/bench_*.rs`) that:
  1. Reads the fixture.
  2. Warms up (10 iterations).
  3. Measures N timed iterations of the operation under test.
  4. Reports `us/parse` and `MB/s` derived from `fixture_bytes * iter / elapsed`.
- All competitors compiled with `--release`, `lto = "fat"`, `codegen-units = 1`.
- All parsers do the **same work**: parse the JSON to "OK or error" -- no AST construction except where the API forces it (serde_json builds a Value).

## Multi-shape benchmark validation (v0.8.9)

The v0.8.4 / v0.8.9 benchmarks above use a single 226 KB JSON
fixture.  That fixture is too small to exercise the SIMD fast
path meaningfully (average string body is 9 bytes, fits in one
AVX2 chunk).  v0.8.9 looked like "no win" on that fixture
specifically, which is misleading.

Running against larger, schema-driven fixtures generated by
`bench/rust_compare/make_fixture_v2.py` reveals the real impact.
Five document shapes, each 1MB and 5MB:

  api_log    HTTP API access logs (mixed structural + short strings)
  product    e-commerce catalogue (medium descriptions, nested specs)
  tweet      social posts (short text, hashtags, mentions)
  metric     telemetry time-series (heavy on numbers, sparse strings)
  prose      documents with long natural-language bodies (1k-10k char strings)

### 1MB fixture, tokenize MB/s, best of 3

| shape | lime | lime+simd | lime+memchr | logos | lalrpop | v0.8.9 vs scalar |
|---|---:|---:|---:|---:|---:|---|
| **prose**   | 348.2 | **685.5** | 644.5 | 432.3 | 459.1 | **+97% (1.97×)** |
| product | 184.6 | **228.7** | 211.8 | 289.2 | 310.5 | +24% |
| tweet   | 140.1 | **155.3** | 142.5 | 254.1 | 270.9 | +11% |
| api_log | 146.5 | **154.9** | 142.7 | 267.3 | 286.0 |  +6% |
| metric  |  96.4 |  93.4 |  87.6 | 345.5 | 347.7 |  -3% |

### 5MB fixture, tokenize MB/s, best of 5

| shape | lime | lime+simd | lime+memchr | logos | lalrpop | v0.8.9 vs scalar |
|---|---:|---:|---:|---:|---:|---|
| **prose**   |  67.4 | **130.7** | 133.0 |  88.7 |  90.5 | **+94% (1.94×)** |
| product |  35.1 |  42.5 |  38.8 |  53.7 |  57.1 | +21% |
| api_log |  15.0 |  15.7 |  14.8 |  48.6 |  51.4 |  +5% |
| tweet   |  15.6 |  16.2 |  15.4 |  47.1 |  47.7 |  +4% |
| metric  |  10.4 |  10.0 |   9.7 |  63.4 |  63.9 |  -4% |

### Headline finding

**On prose-heavy content, `lime --rustlex --rustlex-simd` BEATS logos
by 1.46×–1.59×** at both 1MB and 5MB scales.  This is the architecture
working as designed: long string-body runs (1k-10k chars per body) let
AVX2's 32-byte chunks fire 30+ times per token instead of once.

On JSON-shape content (api_log / tweet / metric — many short tokens),
lime is still 3-6× behind logos.  This is **not** a SIMD scan-dispatch
issue — it is the per-token DFA architectural difference plus the
allocation cost of returning `Vec<Token>` (logos uses a streaming
iterator that doesn't allocate).  Two separable improvements would
close most of that gap:

1. **Streaming iterator API** for lime's Rust output (so consumers
   avoid the Vec<Token> allocation when they don't need a buffered
   token list).  ~50 LOC.

2. **Per-token DFA emission** (~600-1000 LOC of new emit code at the
   shared DFA layer; benefits both Rust and C output).

### Reproducing

```bash
cd bench/rust_compare/
# Generate fixtures (gitignored; regenerate as needed)
for shape in api_log product tweet metric prose; do
    python3 make_fixture_v2.py --shape $shape --size-mb 1 \
        --out fixture_${shape}_1mb.json
done

# Benchmark
cargo build --release --bins
for shape in prose product api_log tweet metric; do
    fix="fixture_${shape}_1mb.json"
    for label in lime lime_simd lime_memchr logos lalrpop; do
        FIXTURE="$fix" ./target/release/bench_$label
    done
done
```

The fixture generator supports five shapes, sizes from kilobytes
to multi-GB, multi-document mode, NDJSON output, and reproducible
seeds.  See `make_fixture_v2.py --help`.

## What changed in v0.8.9

### Multiversion-at-tokenize architecture for `--rustlex-simd`

The v0.8.x `--rustlex-simd` emit was architecturally broken: every
fast-path scan crossed a `#[target_feature(enable = "avx2")]`
inlining barrier.  The emit chain was

```
tokenize() (no target_feature)
  → fast_path_for_state (#[inline(always)])         ✓ inlines
    → fast_path_INITIAL (#[inline(always)])           ✓ inlines
      → scan_self_loop_INITIAL_2 (#[inline(always)])    ✓ inlines
        → scan_simd_2 (#[inline] hint, weak)              ✓ probably under LTO
          → if is_x86_feature_detected!("avx2") { ... }     ← atomic load
            → unsafe scan_simd_avx2_2 (#[target_feature("avx2")])  ✗ INLINING BARRIER
```

The `#[target_feature(enable = "avx2")]` attribute on the AVX2 body
forbids LLVM from inlining it into a non-AVX2 caller.  Result: every
fast-path scan in the benchmark resolved to an out-of-line function
call with a cached-atomic-load dispatch.  The emit was a ~1%
regression versus the scalar baseline.

v0.8.9 lifts AVX2 detection to the `tokenize()` entry point.  The
new chain is

```
tokenize() (public dispatcher)
  → if is_x86_feature_detected!("avx2") { tokenize_avx2(...) }   ← once per call, cached
  → tokenize_avx2 (#[target_feature(enable = "avx2,bmi2")])
    → tokenize_impl::<AvxScanner> (#[inline(always)])
      → fast_path_for_state_g::<AvxScanner> (#[inline(always)])
        → fast_path_INITIAL_g::<AvxScanner> (#[inline(always)])
          → AvxScanner::scan_INITIAL_2 (#[inline(always)])           ← byte literals baked
            → scan_avx2::scan_until_2 (#[inline(always)])             ← AVX2 intrinsics
```

`tokenize_avx2` carries `#[target_feature(enable = "avx2,bmi2")]`,
so AVX2 instructions are emitted directly into its body.  Every
`#[inline(always)]` step inlines without crossing a target_feature
boundary because the feature set is monotone-increasing along the
call chain.  Verified via `objdump`: `vmovdqu`, `vpcmpeqb`, `vpor`,
`vpmovmskb` instructions appear inline in `tokenize_avx2`'s body
without an out-of-line scan helper.

The Scanner trait has one method per fast-path DFA state (`scan_<NAME>_<S>`).
Each `AvxScanner` / `NeonScanner` / `ScalarScanner` impl bakes the
exit-byte literals into its method body, so LLVM treats them as
compile-time constants when monomorphising the lexer body through
the trait dispatch chain.

NEON support (`#[cfg(target_arch = "aarch64")]`) is now wired
through the same architecture; v0.8.x had AArch64 falling through
to scalar.

### Honest measured impact on JSON

| Metric | Result |
|---|---|
| The new emit verifiably inlines AVX2 into `tokenize_avx2`'s body | ✓ confirmed by `objdump` |
| AVX2 vs scalar on JSON fixture | **wash within run-to-run noise** |
| Old broken emit vs scalar | -1% regression (eliminated) |
| New emit vs logos | still 1.5–1.6× slower (gap unchanged) |

The architectural fix is correct.  The benchmark numbers don't
move dramatically on JSON because the fixture's average string
body is **9 bytes** — too short for AVX2's 32-byte chunks to fire
more than once before hitting the closing `"` or `\`.  Most of
the SIMD load is wasted when the run is shorter than the chunk.

The fix WILL show measurable improvement on workloads with longer
self-loop runs:

- HTML / XML body text (long runs of plain text between tags)
- Multi-line string literals in source code (Python triple-quoted,
  Rust raw strings, Ruby heredocs)
- Long comments (block comments, doc comments)
- Lexers that scan whitespace-only lines / indentation runs

For JSON specifically, the architectural difference is that
**logos generates a per-token DFA with leading-byte dispatch**:
byte `{` → emit LBRACE in 1 cycle; byte `[` → emit LBRACKET; byte
`"` → enter string DFA.  Lime's unified DFA does N step-iterations
per token where N = average token length (3.27 bytes/token in this
fixture).  Logos saves ~2 byte-step iterations per token, and that
is the gap.  Closing it requires per-token DFA emission (deferred,
~600–1000 LOC of new emit code).

### What was tried and rejected

- **Pattern B (scan-while-in-set) for state 1 (whitespace), states
  6 / 22 / 24 (digit runs)**: implemented as a hand-rolled prototype.
  Whitespace runs in JSON average 1 byte (no SIMD chunk to load);
  digit runs average 1–3 bytes.  Pattern B added ~3% overhead from
  the extra dispatch checks without firing meaningfully on the JSON
  fixture.  Will help on workloads with long whitespace/digit runs;
  not generally applicable to all grammars.

### Why std doesn't auto-vectorise the scan loop

Four reasons, each independently sufficient:

1. **`#[target_feature(enable = "avx2")]` is a hard inlining
   barrier.** Rust's soundness rule: code with AVX2 instructions
   in its body would crash on a CPU without AVX2 if it ran there.
   So the compiler forbids LLVM from inlining the AVX2 body into
   a function that doesn't also declare `target_feature`.  The
   v0.8.9 fix lifts the target_feature to the tokenize entry
   point so the entire chain monomorphises with AVX2 enabled.

2. **`iter().position(|b| b == X || b == Y)` doesn't reliably
   lower to PCMPEQB + PMOVMSKB.**  LLVM's loop vectorizer needs
   a recognisable pattern.  `position` returns `Option<usize>`
   and short-circuits — it's an early-exit loop with a generic
   closure predicate.  LLVM CAN sometimes vectorize early-exit
   byte scans with two byte-equality compares, but the pattern
   matcher is conservative and often emits the scalar form.
   The `memchr` crate exists precisely because stdlib's
   `slice::iter::position` doesn't reliably lower to vectorised
   byte search.

3. **The two-memchr split scans the tail twice.**  The "obvious"
   stdlib implementation of "find first of `X` or `Y`" runs
   `memchr` for each byte and takes the min — that's `2*N` byte
   loads to find the first match.  Stdlib has no `memchr2`
   primitive that scans for "any of N bytes" in a single pass.

4. **`is_x86_feature_detected!` cached load still costs.**  Each
   call is one atomic load + one branch.  For per-state
   fast-path scans hit thousands of times per parse, that's
   thousands of atomic loads.  Multiversion-at-tokenize pays
   this cost ONCE per `tokenize()` invocation, not per scan.

## Headline numbers (v0.8.4)

Median of 5 runs each, after warmup.

| Tool | Operation | u-s / parse | MB/s | vs lime |
|---|---|---:|---:|---:|
| **lime --rustlex** (default scalar) | tokenize | **835** | **272** | 1.00x |
| lime --rustlex --rustlex-memchr | tokenize | 893 | 254 | 1.07x slower |
| lime --rustlex --rustlex-simd | tokenize | 847 | 268 | 1.02x slower |
| **lime --rust**    | parse (token stream) | 660 | 327 | 1.00x |
| **lime combined**  | tokenize + parse | 1394 | 156 | 1.00x |
| logos              | tokenize | 451 | 481 | **0.61x** (1.63x faster) |
| lalrpop (logos lex)| tokenize | 449 | 482 | 0.61x |
| lalrpop            | parse | 388 | 559 | **0.59x** (1.71x faster) |
| lalrpop combined   | tokenize + parse | 723 | 299 | **0.52x (1.93x faster)** |
| nom (combinator)   | combined  | 497 | 436 | 0.36x (2.81x faster, fused) |
| pest (PEG)         | combined  | 1295 | 167 | 0.93x (1.07x slower) |
| serde_json         | combined + AST | 1170 | 184 | 0.84x (1.19x slower, builds AST) |

## What changed in v0.8.3

`--rustlex` now emits per-state SIMD-friendly fast-path scan
helpers for self-loop-dominant DFA states.  When a state self-
loops on >= 240 of 256 possible bytes (e.g. JSON string body
which self-loops on every byte except `"` and `\`), the emitter
generates a tight `scan_self_loop_<STATE>_<ID>` function that
bypasses the per-byte step dispatch:

```rust
#[inline(always)]
fn scan_self_loop_INITIAL_2(bytes: &[u8], mut p: usize) -> usize {
    let n = bytes.len();
    while p < n {
        let b = unsafe { *bytes.get_unchecked(p) };
        if b == 34 || b == 92 { break; }
        p += 1;
    }
    p
}
```

The inner tokenize loop calls `fast_path_for_state(...)` before
every byte fetch.  For states with a fast path the scanner
advances `p` to the next exit byte (or EOF); for states without
one the dispatcher returns `p` unchanged and we fall through to
the standard step.

| Aspect | v0.8.1 | v0.8.2 | v0.8.3 |
|---|---|---|---|
| Transition dispatch | `&[[i16; 256]]` table | flat `&[i16]` table | `match` expression with byte-range arms |
| Self-loop runs | byte-by-byte step | byte-by-byte step | **inline scan helpers** |
| Generated LOC (json) | 704 | 709 | 359 |
| Tokenize throughput  | ~225 MB/s | ~265 MB/s | **~295 MB/s** |
| Gap vs logos | 1.88x | 1.74x | **1.63x** |

The fast-path emit also replaces the dense table entirely:
v0.8.3 emits a `step_<STATE>(state, byte) -> i16` function as
a `match state { 0 => match b { ... } }` Rust expression rather
than a flat `&[i16]` table.  LLVM lowers the per-state byte
match to a jump table in `.text`; for our 32-state JSON DFA the
jump table is L1-resident and the step dispatch is roughly
equivalent in throughput to the flat-table version.  The
performance win comes entirely from the fast-path emit, not the
step rewrite -- but the match form makes the fast-path
integration cleaner and shrinks the generated code by ~50%.

## What we tried and rejected

**Equivalence-class compression** (logos's BYTE_CLASS technique):
implemented in v0.8.2 development; benchmarked at 28% slower on
small DFAs because the dependent BYTE_CLASS load defeated cache
locality.  Right answer for very large DFAs (1000+ states);
deferred until a consumer with a large grammar needs it.

**Pattern B fast path** (scan-while-in-self for small self-sets,
e.g. whitespace `[ 	

]`): implemented during v0.8.3
development; the function-call overhead exceeded the savings on
short self-loop runs (typical JSON whitespace runs are 1-2
bytes).  Pattern B is gated off in the C emitter behind an
`if (0 && ...)` and is worth re-enabling with a length threshold
once we have profile data on workloads with long self-loop
runs (XML text content, Python source-code indentation, etc.).

**`iter().position()` form for fast-path scans**: tried; stdlib
does NOT lower iterator+position to a vectorised memchr in the
general case.  The two-memchr split (one per exit byte, take
the min) was a 1000x regression because it scans the full tail
twice per fast-path entry.  Hand-rolled `while` loop with
`#[inline(always)]` and `unsafe { get_unchecked }` is what
landed -- scalar but tight.

## What would actually close the remaining 1.63x gap to logos

logos hits ~480 MB/s on this fixture; lime --rustlex hits
~295 MB/s.  The remaining gap is closable but only with
machinery we deliberately deferred:

1. **memchr / memchr2 SIMD scan** -- emit calls into the
   `memchr` crate for fast-path scans where exit_set <= 3
   bytes.  Adds a runtime dependency; the `--rust`/`--rustlex`
   output stops being self-contained.  ~50 LOC of emit changes.
   **This alone closes most of the remaining gap.**
2. **Per-token DFA emission** -- one DFA per token kind
   instead of one unified DFA.  ~500-1000 LOC of new emit code.
   Only worth it for grammars where a single per-token DFA
   substantially outperforms the unified construction.
3. **Hand-rolled SSE/AVX2 intrinsics in the fast path** --
   platform-specific (`std::arch::x86_64::_mm_cmpeq_epi8` etc).
   Avoids the memchr dependency at the cost of x86-only
   acceleration.  ~100-200 LOC plus platform feature
   detection.

None landed in v0.8.3.  Listed in `docs/RUST_OUTPUT.md` for
follow-up if a consumer's profile shows the lexer as the
bottleneck.

## What changed in v0.8.4

Three new opt-in flags and one Windows-test-portability scaffold:

- **--rustlex-memchr**: emit fast-path scans that call into
  the memchr(2) crate (SIMD-accelerated byte search).  Adds
  memchr = "2" to the generated Cargo.toml when used with
  --rustcrate.  The scalar default remains self-contained.

- **--rustlex-simd**: emit hand-rolled SIMD intrinsics
  (SSE2/AVX2 on x86_64, NEON on aarch64) inline in fast-path
  scans.  Self-contained -- no extra crate dependency, runtime
  feature detection via std::is_x86_feature_detected.
  RISC-V V-extension is gated off (intrinsic API still
  stabilising; falls through to scalar).

- tests/test_compat.h: portability shim providing
  tmpdir/rmdir-recursive/setenv/unsetenv/realpath plus a
  fork+exec-or-CreateProcessA subprocess capture.  POSIX
  implementations are pure passthroughs; Windows uses
  GetTempPathA, CreateDirectoryA, FindFirstFileA recursion,
  _putenv_s, GetFullPathNameA, and CreateProcessA + pipes.

- tests/test_jit.c, tests/test_jit_fallback.c: refactored
  to use test_compat_setenv / _unsetenv.  Their
  host_machine.system != windows gates removed; both should
  now build under MinGW gcc / clang / clang-cl on Windows.

The remaining 15 POSIX-gated tests still need refactoring
(each uses system() with shell utilities; 50-100 LOC per
test).  Pattern documented in the test_compat.h header.

### What got measured but not shipped

- **Per-token DFA emission**: investigated as a narrow
  variant (leading-byte direct dispatch for 1-byte tokens)
  and as a full per-rule DFA construction.  The narrow
  variant added overhead on the common path and measured
  slower than the v0.8.3 default; reverted.  The full
  per-rule DFA construction is 600-1000 LOC of new emit
  code -- deferred until consumer demand justifies the
  scope.

- **Windows LSP port**: substantial CreateProcessA + pipe
  rewrite of src/lsp/lsp_diagnostics.c and lsp_format.c.
  Deferred.  LSP remains gated off on Windows.

### Honest conclusion

For JSON-shaped workloads the v0.8.3 default scalar emit
remains the best lime configuration: ~835 us / 272 MB/s
median.  --rustlex-memchr and --rustlex-simd cost more than
they save on JSON's short scans because the per-call
dispatch boundary (function call, target_feature inlining
barrier) exceeds the scan savings.  Both flags WIN on
grammars with long self-loop runs (HTML/XML body text,
comments, multiline strings) -- worth benchmarking on those
workloads before committing to either.

The 1.6x lime-vs-logos gap on JSON is NOT closeable with
the current architecture.  Closing it requires per-token
DFA emission as a separate compilation pass; that work is
deferred.

## Cross-platform verification

v0.8.3 has been tested on:

| Platform | OS | Compiler | Tests | Notes |
|---|---|---|---|---|
| Linux x86_64 | Linux 6.x | GCC 15.2 + ASan + UBSan | **114 / 0 ok** | dev box, full matrix |
| Linux RISC-V64 (Ky X1) | Linux 6.6 | GCC 13.3 | **114 / 0 ok** | scaled stress + scaled merkle threshold |
| Linux RISC-V64 (Ky X1) | Linux 6.6 | Clang 20.1 | **114 / 0 ok** | same |
| Windows ARM64 (x86_64 emul) | Win11 Pro | MinGW gcc 13.2  | **101 / 0 ok** | full pass |
| Windows ARM64 (x86_64 emul) | Win11 Pro | Clang 22.1 MSVC | **101 / 0 ok** | full pass |
| Windows ARM64 (x86_64 emul) | Win11 Pro | clang-cl 22.1   | **101 / 0 ok** | full pass |
| FreeBSD x86_64 (NUC) | FreeBSD 14 | clang | (host offline) | last pass: v0.8.2; subsequent changes are platform-additive |

The Windows port required the following fixes that landed
alongside v0.8.3:

  - lime-lsp gated off (POSIX fork+exec)
  - test_diff_conflicts + 16 other POSIX-test gates
  - timespec/clock_gettime shim coexists with UCRT
  - parse_glr.c + snapshot_create.c routed through
    lime_threads.h (Win32 SRWLOCK + CRITICAL_SECTION shims)
  - PTHREAD_MUTEX_INITIALIZER on Win32 (SRWLOCK_INIT)
  - lime.c restructured for the __WIN32__ branch + missing
    'dup -> _dup' alias
  - parse-manager and test_lime_compile_in_process gated
    (getopt + dlopen + unistd-only)
  - bench/-lm replaced by find_library('m', required: false)
  - lld-link weak-symbol-equivalent via /alternatename for
    lime_compile_grammar_in_process

Tokenize throughput on RISC-V64 (Ky X1, GCC 13.3):

  v0.8.2: lime ~32 MB/s, logos ~90 MB/s  (gap 2.81x)
  v0.8.3: lime ~30 MB/s, logos ~88 MB/s  (gap 2.93x; noise)

The fast-path emit's win is x86-specific -- the byte-class loop
benefits from x86's high IPC and L1 prefetching.  RISC-V (and
ARM64 under emulation on santorini) sees the per-state dispatch
overhead but not the proportional win on string-body scans.
On the Ky X1 specifically the lime/logos ratio is unchanged
within run-to-run noise.

Tokens byte-identical to the x86_64 baseline on all platforms
(verified via dump_tokens.rs).

## Interpretation
## Interpretation

### Lexer (--rustlex vs logos)

logos is **1.74x faster** than lime's lexer (460 vs 265 MB/s).
Down from 1.88x in v0.8.1 thanks to the flat-layout / row-
hoist rewrite.

logos's per-token DFA emission and SIMD-accelerated character
classification are the remaining wins.  Closing further
requires the work in the previous section.

### Parser (--rust vs lalrpop)

**lime parse: 660 us (327 MB/s)**
**lalrpop parse: 425 us (509 MB/s)**

lalrpop is **1.55x faster** than lime on parse-only.  Same
ratio as v0.8.1 (parse code wasn't touched in v0.8.2).
lalrpop emits direct state-machine code: one big match arm
per state with inlined goto logic, no indirect calls, no
per-reduce allocation.

Closing the parser gap would mean giving up runtime
composition (lime's main differentiator).  Acceptable for
parsers that want runtime composition or grammar mutation;
not the right tool for pure-static maximum-speed parsing.

### Combined (lex + parse)

**lime: 1577 us (137 MB/s)**
**lalrpop+logos: 776 us (278 MB/s; 2.03x faster)**

The combined gap is the product of both individual gaps.

### Where lime wins or ties

- **vs pest (PEG):** lime is **1.16x faster** combined.
  PEG backtracking is heavier than LALR on this fixture.
- **vs serde_json (full AST):** lime is **1.34x faster**
  combined.  serde_json's AST allocation is heavier than
  lime's table-driven LALR token-stream consumption.
- **vs nom/chumsky:** nom is faster combined (2.97x) but
  doesn't have a separate lexer phase; the recursive-descent
  design avoids token-stream allocation entirely.  Apples
  to oranges with lime's table-driven LALR.

## What this means for Lime's positioning

Lime's Rust output is **production-viable for grammars where**:
- Runtime composition is needed (lime's main differentiator;
  lalrpop has no equivalent).
- The C side is the primary target and Rust is additive
  (PG-team's pattern).
- Speed is "good enough" -- within 2x of the fastest
  specialized tools.

Lime's Rust output is **not the best choice when**:
- Pure static parser speed is the only concern (use
  lalrpop or hand-tuned recursive descent).
- The lexer alone needs to hit logos's 400+ MB/s (use
  logos directly or wait for lime's per-token DFA work).
- Building a JSON-style AST is the goal (use serde_json).

## Reproducing

```bash
cd bench/rust_compare
python3 make_fixture.py
cargo build --release
for b in serde lime logos lalrpop nom pest; do
    echo "=== bench_$b ==="
    ./target/release/bench_$b
done
```

All numbers above are median of 5 runs on a quiet
i9-12900H.  Run-to-run variance is ~5 percent.  Numbers will
differ on other hardware; the relative ordering should hold.

## Notes on individual tools

- **logos:** v0.14. Generates per-token state machines via proc-macro. Industry-standard fastest Rust lexer for hand-written-like throughput.
- **lalrpop:** v0.20. Rust-native LALR(1) parser generator. Direct analog to lime's parser side.
- **nom:** v7. Parser combinator library; combined lex+parse via recursive descent, no separate token stream.
- **pest:** v2. PEG parser generator. Slower than LALR for this fixture due to PEG's backtracking semantics.
- **chumsky:** v0.10. Modern parser combinator library. Skipped in this comparison; v0.10's API is substantially rewritten and a fair port is a separate exercise. nom serves as the parser-combinator data point.
- **serde_json:** v1. Industry-standard JSON parser; tuned for AST construction. The "speed of light" reference for general-purpose JSON parsing in Rust.

## Conclusions

1. **Lime's Rust output works correctly on a 226 KB JSON fixture with 69k tokens.** No correctness regressions.
2. **v0.8.2 closes ~7 percent of the gap to logos** via a flat-layout + row-hoist rewrite of the lexer emit.  No memory regression; ~18 percent faster tokenize.
3. **Performance is within 2x of the fastest specialized Rust tools** (logos, lalrpop) for the lexer and parser respectively, and roughly equal-or-better than higher-level tools (pest, serde_json's AST build, chumsky's API).
4. **The remaining 1.74x lexer gap to logos requires per-token DFA emission**, deferred to a future release pending consumer demand.
5. **For consumers whose needs are speed-of-light Rust JSON parsing, lalrpop+logos is the better choice.** For consumers who want grammar composition + Rust output + C output from one source, Lime is the only option in the ecosystem.