# Crash-recovery and hot-standby tests

`pg_lion` writes generic WAL (`GenericXLogStart`/`GenericXLogFinish`,
DESIGN.md §4–§5), and the count path treats a standby specially: in recovery
the pin interlock of §9 does not hold, so the count trusts the visibility map
for nothing and rechecks every candidate TID. Neither of those had an
automated test. The only recovery check the project ever had was one manual
`pg_ctl -m immediate` early in development, before sparse segments (§13),
byte-sized bucket counts, the reserved NULL and key-less entries (§14, §17),
multi-key operator classes (§17) and the reworked two-pass VACUUM (§11)
existed. This directory is that test.

## Running it

```sh
make recovery-check PG_CONFIG=<prefix>/bin/pg_config RECOVERY_PREFIX=<prefix>

# or directly:
test/recovery/run.sh --prefix <prefix> [--iters N] [--keep]

# as root (a container, a CI image): initdb and postgres refuse to run as
# root, so name an unprivileged user to run the clusters as
RECOVERY_RUN_AS=postgres make recovery-check RECOVERY_PREFIX=<prefix>
```

`<prefix>` is a PostgreSQL *installation* (it needs `bin/initdb`,
`bin/pg_basebackup`, `bin/pgbench`, `bin/pg_waldump`, and `citext` in
`share/postgresql/extension`), and it is REQUIRED - `--prefix` or
`RECOVERY_PREFIX`; the run refuses to start without one. There is no default
on purpose: the extension is rebuilt from a clean copy of this tree — `src/`,
the Makefile, the control and SQL files, with the objects and the compiler's
`src/*.d` removed — and *installed* into that prefix, exactly as `bench/lib.sh`'s
`bench_build_extension` does, so a guessed prefix would be somebody else's
server. `make install` is never run in the source tree, and the prefix is
deliberately not derived from `PG_CONFIG`.

`RECOVERY_RUN_AS=<user>` runs `initdb`, `pg_ctl` and `pg_basebackup` through
`runuser -u <user>`; everything else (psql, pgbench, pg_waldump, the build)
runs as the caller. Run as root without it, the script stops before doing
anything and says so. That user must be able to read the prefix; it is given
ownership of the run's private directory.

`RECOVERY_SKIP_INSTALL=1` uses the pg_lion already installed in the prefix
instead of building and installing one, for an installation the caller cannot
write to: CI's packaged-server jobs install the extension with `sudo make
install` and then run the harness as an ordinary user. The run checks that
the control files and the library are there, and says so if the tree's
`pg_lion.so` is not the installed one.

Phases 1b-1e, 1g and 3 need the `injection_points` extension. On a server without
it they are skipped - the log and the summary say which - and phases 1, 1f,
1h, 2, 4 and 5 still run, phase 5 without its one case that parks an insert;
`--phases "1 1f 1h 2 4 5"` asks for exactly those (CI does, on the packaged
16-19 servers). Phase 5 needs pgvector and `pageinspect`, and is skipped, with a
line in the summary, without them; phase 1h needs `pageinspect` and
`pg_freespacemap`, and is skipped the same way. `INJECTION_POINTS=1` in the environment turns every
such skip into a failure, so a server that should have injection points and
does not cannot end in "ALL RECOVERY AND HOT-STANDBY CHECKS PASSED" having run
none of those phases; CI's source-build jobs set it. `PGVECTOR=1` does the
same for phase 5, and CI sets it wherever it built or installed pgvector.

The run creates everything it needs in ONE private directory that
`mktemp -d` makes for it under `$RECOVERY_TMPDIR` (default `$TMPDIR`, else
`/tmp`): the two clusters, their server logs, the socket directory and the
build copy. The clusters listen on ports 54340 (primary) and 54341 (standby)
of that private socket directory only, with `listen_addresses = ''`, so two
runs - two users, two worktrees - cannot meet at an endpoint. (A Unix socket
path has a length limit, so when the private directory is too deep for one
the socket directory is a second `mktemp -d` under `/tmp`.) The run checks
`SHOW data_directory` against the directory it asked for before trusting a
connection, and removes the directory it created, and nothing else, through
an exit trap (`--keep` suppresses only the removal and prints the path). It
never touches the dev cluster or the benchmark clusters.

One caveat about that prefix: it is shared. The run installs
`pg_lion.so` into it and then starts a cluster that loads it, so another
process running `make install` against the same prefix in that window will
have its build tested instead of yours. That shows up as `lion_index_verify`
failing in the *baseline* check, before any crash has happened — a failure
there is a reason to check who else is installing into the prefix, not a
recovery bug. Every failure after the baseline is about recovery.

Takes about 85-110 s with the default 8 iterations. Full output goes to
`test/recovery/log/run.log`; on failure the two server logs are copied next to
it. `make clean` removes the directory.

## Files

| file | what it is |
| --- | --- |
| `run.sh` | the whole thing: build, clusters, crash loop, standby, promotion, summary |
| `schema.sql` | the fixture and every check, as read-only SQL functions |
| `writer.sql` | the pgbench script the crashes interrupt |
| `standby_rr.sql` | the standby side of the recovery-conflict test |
| `lionvec.sql` | phase 5's fixture and checks: a lionvec index over vectors in TOAST |
| `lionvec_writer.sql` | the pgbench script phase 5's crashes interrupt |

The checks live in the database rather than in the shell because they have to
run unchanged on the primary, on a hot standby and on a promoted standby, and
a standby can be given neither new functions nor temp tables. So every helper
in `schema.sql` is read-only, and the multiset comparison that the SQL
regression tests spell as two `CREATE TEMP TABLE`s plus `EXCEPT ALL` is done
over two materialised text arrays — still with `EXCEPT ALL` in both
directions.

## What is covered

The fixture is a table with one column per operator class kind — `int4`,
`text`, `citext`, `bool`, a nullable `int4`, `int4[]` (with `NULL`, `'{}'`,
empty arrays with dimensions and all-`NULL`-element rows) and `tsvector` (with
`NULL` and empty rows) — so every reserved entry is populated. Four of the ten lion indexes
are created on the empty table and filled by `aminsert`, so their entries go
through INLINE→CHAIN spills, sparse-segment promotions and container-page
splits under WAL; six are built by `ambuild` after the load, among them a
multicolumn one (DESIGN.md §24) and one with summaries (§32).
`inline_limit = 64` on five of them makes spills and splits frequent rather
than rare, and the 3000 distinct values of `t` over ~800 heap pages give about
one member per container key, which is what puts sparse segments in an index at
all. One of the four filled by `aminsert`, `lion_rec_slc`, keeps bit slices
(DESIGN.md §51) of an `int8`, a nullable `int4` and an `int4` expression that
is negative one row in 97: every row the writer inserts goes into a few dozen
slices, the dense ones' in-place adds several to a record and the rare bits'
INLINE slices beside them (§51, "Insert cost, revisited"), which every crash
round replays and the standby replays under `wal_consistency_checking` in
rmgr mode.

**Phase 1, crash recovery under load.** Eight rounds of: ~2 s of mixed writes
(pgbench, 4 clients, `synchronous_commit = on`, ~430 transactions/s), then kill
the server with no chance to flush — `pg_ctl stop -m immediate` and `kill -9`
of the postmaster, alternating — restart, and check. Each round replays
30000–70000 generic WAL records.

Each writer transaction inserts three new rows, does a non-HOT update of every
indexed column of one of them (so every index gains a TID and loses one),
updates a short primary-key range of its own rows, and deletes its three
lowest-numbered rows; a second session `VACUUM`s every 0.6 s. Three rows in
and three out keeps the table at exactly 40000 rows while the live id range
slides upwards, which is the point: `VACUUM` frees low heap pages that later
inserts reuse, so container keys arrive out of order and the rarer placement
paths get exercised — a pair landing inside a *full* sparse segment (which
§13 notes only happens for out-of-order inserts) and a container inserted in
the middle of a chain page rather than appended at the tail. Each pgbench
client works only on rows with `id % 8 = client_id`, so no two clients contend
for a row and a deadlock cannot abort the load.

Rounds 5 and 6 crash 350 ms into a foreground `VACUUM`, so replay has to
finish a half-applied two-pass bulkdelete. After every restart:

* the restart really did recover: the log must say the system was not properly
  shut down and must print `redo starts at`, and `pg_waldump -r Generic` over
  the replayed range must have found generic records. A round that replayed
  nothing fails the run instead of passing silently.
* `lion_index_verify(idx, true)` on every index — structure plus
  every heap tuple present under every key it extracts to, and in every
  slice of its value's bits.
* `count`, `sum`, `min` and `max` of each sliced column from its slices
  (`lion_index_slice_agg()`) against a sequential scan - on a standby too,
  where the function's descent collects nothing (DESIGN.md §51.10).
* every single-key probe counted four ways — forced index path, forced
  sequential scan with no index path at all, `lion_index_count()`, and the
  count pushdown — must give the same number. 19 probes, including absent
  keys, a lower-case `citext` key against mixed-case data, and both `bool`
  values.
* 20 multi-row probes — `GROUP BY` on every column kind including the NULL
  group, the `IS NULL` and `IS NOT NULL` pushdown drivers, `@>`, `&&` and `@@`
  — compared between the index path and a forced sequential scan as multisets,
  `EXCEPT ALL` in both directions.
* `lion_index_stats().ntids` against the heap: `>=` immediately after
  recovery (the index may still hold TIDs of dead tuples), and exactly equal
  after a `VACUUM`. For the two multi-key indexes the expected total is the
  number of (key, row) pairs, with one pair for a row that extracted no keys.
  `null_tids`, `empty_tids` and `nullelem_tids` are checked separately,
  because a reserved entry that lost its flag would hide inside the `ntids`
  total.

**Phases 1b-1h, a crash or a restart at a chosen point.** Each builds a
table of its own. 1b parks a VACUUM between the two steps of a whole-chain
free (injection point `lion-vacuum-entries-deleted`), 1c and 1d an insert
between a directory or posting-tree split and its downlink, and each crashes
the server there and checks what comes back. Four more (DESIGN.md §18, §22,
§25):

* **1e**, a crash inside a MULTI-LEAF spill. VACUUM's filtering turns one
  key's INLINE payload into eleven NARROWs, three to a leaf, a set of four
  leaves with no internal level whose first leaf is its head; the spill parks
  at `lion-spill-leaves-written`, after its last leaf record and before the
  head and the entry, and the server dies. The entry must come back INLINE
  with every TID, `lion_index_verify()` must call the three other leaves and
  the head a leak and not corruption, and the next VACUUM must spill the set
  and free the orphans in its sweep. A second crash replays that VACUUM.
* **1f**, VACUUM of an rmgr-mode index on a server WITHOUT the resource
  manager. The partial index is built with the library preloaded, the server
  is restarted without it (and without `wal_consistency_checking`, which
  names the manager), and a VACUUM that removes nothing from the index runs
  with `pg_lion.vacuum_barrier_ranges = 1`. `pg_waldump` over its range must
  find no record of a custom manager - a `custom128` one, which no such server
  can replay - and a crash right after it must recover. With
  injection points, a second rmgr-mode index first gets phase 1b's crash
  under the manager, which leaves entries deleted and their chains' pages
  leaked; the VACUUM without the manager has nothing to remove from it and
  must go through and leave the leaks, which a VACUUM with the manager and
  dead rows to remove frees in rmgr mode (a VACUUM without dead rows calls no
  `ambulkdelete`, and only its walk sweeps for leaks). The primary is
  restarted in its `--mode` afterwards.
* **1g**, a crash between a PROMOTION and the split it was for. A posting
  set of up to 16 leaves has no internal level; the split that would make a
  17th leaf first pushes the head down under a root that links every leaf,
  in a record of its own. An append parks at `lion-posting-promoted`, after
  that record and before the split, and the server dies. The set must come
  back promoted with nothing flagged, verify() clean and both keys exact,
  the next inserts must split the full leaf as a tree's, and a second crash
  replays all of it.
* **1h**, the free space map after a crash (DESIGN.md §18, "The free space
  map holds only pages that may be handed out"). The map is not WAL-logged:
  with data checksums each of its pages reaches the WAL as one image of its
  first change after a checkpoint. A VACUUM records the pages a delete
  freed, a checkpoint follows, inserts take those pages, and the server
  dies, so recovery brings back map entries for pages in use again - the
  phase fails if it brings back none, since nothing would be tested. The
  next inserts are offered them, must refuse each without recording it
  again, and leave the map with no entry on a page in use; verify() is
  clean and the rows exact.

1b and 1e also read the page counts `VACUUM (VERBOSE)` reports for the index
(DESIGN.md §18, "Page counts"), which no SQL function can see: the pages
newly deleted must be exactly the ones the phase freed, and - since neither
index holds a page an earlier VACUUM freed - the free pages that are not newly
deleted must be exactly the reusable ones, all-zero pages: a page counted
twice, or one freed a moment ago called reusable, breaks the equation.

**Phase 2, hot standby.** `pg_basebackup -R -X stream` into a standby, then:

* the full phase-1 check battery again, in recovery.
* `lion_index_count_stats()` for every single-key probe must show
  `blocks_skipped = 0` and `tids_rechecked = count` — §9's rule that a standby
  never trusts the visibility map. On the primary the same probes must show
  the opposite (blocks skipped via the map), or the standby assertion would be
  vacuously true.
* a line-per-probe picture of everything the index answers — each key count
  through `lion_index_count()`, each multi-row query through the pushdown,
  with the plan node that answered it — must be byte-identical between primary
  and standby. Comparing the node too means a standby that quietly stopped
  using the pushdown is a failure rather than a silently weaker test.
* an index build replayed: two indexes built on the primary over 300000 rows
  that are already there (a few dense keys; 5000 text keys), then
  `lion_index_verify(idx, true)` and a key count through each on the standby.
  It is the one place a build's own WAL is replayed - the bulk writer's page
  images, which on 16 come from `lion_build.c`'s copy of 17's bulk-write API.
  The same two indexes then carry the key counts of DESIGN.md §33: the
  build's, an ANALYZE's after new keys and a VACUUM's after their rows are
  deleted - counted by ambulkdelete's walk, so the VACUUM is retried until
  the standby's feedback lets it remove them - each read on the standby
  through `lion_index_stats()` and compared with the primary's, and with the
  exact count, once it has replayed the META record that wrote it.
* a position tree damaged on the standby alone: verify.sql's seventeenth case
  built on the primary, checked on the standby while sound, then, with the
  standby stopped after a restartpoint, one byte of a position leaf changed in
  the standby's file, which leaves five TIDs without positions.
  `lion_index_verify()` on the standby must report them: a standby compares a
  position tree with its posting set from a walk that read every page as it
  was at one replay position (DESIGN.md §7, "Position trees").
* the recovery-conflict case, run twice. A standby session opens a
  `REPEATABLE READ` transaction and counts key K; the run waits (on
  `pg_stat_activity`, not on a timer) until that session really holds its
  snapshot, then deletes K's rows on the primary and vacuums. With
  `hot_standby_feedback = on` the primary must not have been able to remove
  the TIDs (asserted against `ntids`) and the session's next count of K must
  be the same number. With it off the removal must have happened (also
  asserted) and `max_standby_streaming_delay = 2s` makes the conflict
  reachable: the only legal outcomes are the same number again or a cancelled
  session with `conflict with recovery` on stderr. A *different* number fails
  the run. Afterwards, once the standby has caught up, a fresh session must
  see the new count, through `lion_index_count()` and through a sequential
  scan.
* `pg_ctl promote`, then `lion_index_verify(idx, true)` and the whole
  battery again on the promoted node, exact `ntids` after a `VACUUM`, and
  `lion_index_count_stats()` showing it trusting the visibility map again.

**Phase 3, a standby reader holding its pin** (DESIGN.md §9, §11, §25). A
standby of its own (phase 2 promoted the first one), with
`max_standby_streaming_delay = -1`, and three cases, each of which parks a
standby `lion_index_count()` at the injection point
`lion-count-containers-pinned` - containers copied, page pinned, visibility map
not yet consulted - and then has the primary move the TIDs it copied off that
page and VACUUM them there:

* **split**: the reader pins the leftmost leaf of a posting tree; inserts grow
  that leaf's first container until it splits, which moves the containers
  holding dead TIDs to the new right sibling, and the VACUUM removes them
  there. The primary's VACUUM cleanup-locks the pinned leaf but writes nothing
  for it. Runs with `pg_lion.vacuum_barrier_ranges = 1`, so the barrier goes
  out in stand-alone VACUUM_VISIT records.
* **pushdown**: the reader pins the head of a posting set of 16 leaves with no
  internal level; an insert needs a 17th leaf, pushes the head down, and the
  VACUUM cleans the child.
* **spill**: the reader pins a directory leaf for an INLINE set; the VACUUM's
  own filtering makes the payload outgrow the entry and spill.

The run then waits until replay is either BLOCKED (the startup process waiting
for a cleanup lock) or has replayed past the primary's last record, releases
the reader and compares its count with the one its snapshot must see. In rmgr
mode replay must have blocked; in generic mode it catches up and the count must
still be right (the standby rechecks every TID). A failing case is reported and
the other cases still run.

A fourth case, **setaba**, parks a standby `lion_index_verify()` instead, at
`lion-verify-set-leaf-read`, after the first of the two leaves of a posting
set (DESIGN.md §7, "Posting sets on a standby"). The primary then inserts a
row into the set's last container and VACUUMs away a row deleted before the
check began, from the first leaf: the entry's header - TIDs, containers,
leaves - is as it was, and a walk that read the first leaf before the VACUUM
and the second after the insert counts one TID more than the entry. The
check, released once the standby has caught up, must pass, and its DEBUG
line must show the walk kept for its pages and its totals settled by one
counting pass.

Eight more cases set up, one at a time, a state a standby's
`lion_index_verify()` meets and a primary's never does (DESIGN.md §7,
"verify() on a standby"): replay applies VACUUM beside the check, and every
operation the primary writes in more than one record shows the states between
its records to the standby's readers for as long as the primary takes. Each
check must pass, with no WARNING, except the last one of the split cases:

* **vunlink**, **vfree**: the check parks after the first leaf of a posting
  set of six; a VACUUM takes the emptied second leaf out of the set
  (UNLINKED), or frees the whole set and its entry, under it.
* **vreach**: the check parks before its reachability pass, and a directory
  leaf split takes a page out of the free space map that no walk reached
  (needs `pg_freespacemap`; skipped without it).
* **vdownlink**: the check parks after the directory's leaves, and a split's
  downlink names a leaf the walk did not reach.
* **vleftlink**: the check parks on the first directory leaf, and its split
  changes the left link of the page after it.
* **vsummary**: an INSERT parks on the primary between its row's key and its
  summary (`lion-insert-before-summary`), and the check runs, then again
  with heapallindexed.
* **vsplitdir**, **vsplitpost**: a directory leaf split, or a posting leaf
  split, parks on the primary between its first record and its downlink, and
  the check sees the flag - and, for the directory, parks before it looks at
  the flagged page again while the split finishes. Then a split that fails
  there (an error injected) is abandoned, and the check must warn of it.

`PHASE3_CASES="split spill"` in the environment runs a subset (the default is
every case), and `--phases 3` skips phases 1 and 2.

**Phase 4, `lion_index_verify()` on a standby beside replay** (DESIGN.md §7,
§11, §25). Replay takes a record's blocks in the order the writer registered
them and holds every one to the end of the record, and every record that
changes a posting set registers the set's pages before the entry's directory
leaf - container page, then leaf, the reverse of the primary's order. A
verify() that walked a set with that leaf held SHARE, or whose heapallindexed
lookups descended a set with it held, would lock leaf, then container page,
and the startup process and the backend would wait for each other on buffer
content locks, which have no deadlock detector and ignore cancel and
terminate: replay would stop for good, and only an immediate shutdown would
get the standby back. So on a standby verify() walks a set with nothing held,
and its lookups let the leaf go before they descend (DESIGN.md §7). The phase
makes a standby of its own, with a table of three keys
whose posting sets are trees of about twenty leaves, and for `PHASE4_SECS`
seconds (12) has pgbench insert into those keys on the primary while two loops
on the standby call `lion_index_verify(sv_k)` and
`lion_index_verify(sv_k, true)`. A watchdog asks the standby twice a second
where replay stands, and fails the phase as soon as replay has not moved for
`PHASE4_STALL` seconds (10) while WAL it received waits to be replayed - a
deadlock never ends, so the phase cannot hang, and `statement_timeout` would
not have ended it. Every verify() call must return without an error (a set
or position tree that replay kept changing through every counting pass is
kept with a WARNING, which the phase counts); then,
caught up, verify() with heapallindexed and every key's count must agree with
the primary. A deadlock shows within seconds of the load starting, in either
mode, as the startup process and both verify() backends waiting on
`LWLock/BufferContent`.

**Phase 5, lionvec's locator of vectors in TOAST** (DESIGN.md §44.13). A
lionvec index over vectors stored in TOAST keeps, for each row, where its
value's chunks lie: LOC entries in heap order under a LOCDIR, and the same
place in the row's REFINE item. The build writes them through
`log_newpage_range()`; an insert appends its row's entry in a generic WAL
record of its own, after its REFINE and CODE items, under the meta page's
lock; VACUUM removes the entries of the rows it removes. `lionvec.sql` builds
`lv_rec` (3,000 rows of `vector(768)`, two chunks each, and NULLs) and
`lv_rec_check()`, which reads the locator off the index's pages with
`pageinspect` and checks it against the heap - every tuple, whatever its
visibility - and the TOAST relation: no entry names another value than the
tuple at its TID stores; every row the snapshot sees whose vector is in TOAST
has an entry of its value, placed where its chunks are; a REFINE item's place
is its row's entry's; and queries by the exact path and by probing return the
same rows, order and distances with `pg_lion.vector_toast_direct` on and off,
and a sort's distances. Three rounds of `lionvec_writer.sql` (inserts, updates
HOT and not, of the vector and of other columns, deletes, rollbacks,
`INSERT ... SELECT`) beside VACUUMs of the table and of its TOAST relation
alone, killed as phase 1 kills, the last with a VACUUM in flight, each checked
after the restart and again after a VACUUM. The writers set
`wal_consistency_checking = 'generic'`, so each recovery compares every page
their records touched with the page the primary had. Then an insert parked at
`lionvec-insert-unlocated` - its items in, its entry not - is cut by a crash:
its row comes back aborted, with its REFINE item and without an entry, and the
checks pass before and after the VACUUM that removes it. Last, a hot standby
of its own under the same load must pass the same checks with the primary's
numbers, and again after a `VACUUM FULL` of the TOAST relation alone has made
every place stale.

## What is not covered

* **No torn-page test.** Nothing here interrupts a page write in the middle.
  Generic WAL records carry a delta against a page image and full-page writes
  are on, so a torn page is PostgreSQL's problem rather than the AM's, but
  that is an argument, not a test. Doing it properly wants a filesystem or
  device layer that can fail a write at an offset. The clusters do run with
  `--data-checksums`, so a page that replay corrupted structurally *inside*
  the page would be caught by the reads the checks do.
* **No long soak.** Eight rounds of two seconds is about 20000 transactions;
  it is a smoke test for the WAL records the write paths emit, not a
  statement about hours of churn. `--iters N` raises it.
* **Crash injection at a few chosen points only.** Phases 1b-1e and 1g stop a
  backend at an injection point and crash the server there; every other
  multi-record operation (a segment promotion, say) is crashed wherever the
  phase 1 workload happens to be, which reaches the frequent paths and not
  the rare ones.
* **No replica that is behind.** The standby is always caught up before it is
  read, except in the conflict test and in phase 4, where being behind is the
  point. Nothing tests a cascading standby, a restart from an archive,
  `pg_rewind`, or a timeline switch other than the one `pg_ctl promote` makes.
* **No unlogged or temporary indexes.** `test/sql/unlogged.sql` covers what an
  unlogged lion index does; recovery truncates it, so there is nothing to
  check here.
* **No concurrent readers during the crash.** The writers are killed; nobody
  is mid-scan. The interlock between readers and VACUUM is what
  `test/isolation/` is for.