# Testing And Release

The project has several validation layers. Use the narrowest test that covers
your change while preserving the release gates for cross-boundary behavior.

## Fast Local Checks

```bash
cd graph
cargo fmt --check
cargo test --features pg17
```

Run pgrx integration tests:

```bash
cd graph
cargo pgrx test pg17
```

Build docs:

```bash
cd graph
cargo doc --features pg17 --no-deps
```

Check docs against local references, the SQL-facing Rust API, registered GUCs,
and the Rust source map:

```bash
scripts/check_docs_drift.sh
```

The aggregate script runs `scripts/check_doc_references.py`,
`scripts/check_sql_api_drift.py`, and `scripts/check_rust_doc_map_drift.py`.
Use `python3 scripts/check_sql_api_drift.py --list-implemented` or
`python3 scripts/check_rust_doc_map_drift.py --list` when you need to inspect
the implementation-side inventory.

Check dependency freshness without updating manifests:

```bash
python3 scripts/check_dependency_updates.py
```

The dependency checker enforces the release policy that recommended dependency
updates must be at least 6 hours old. It flags newer releases for review and
requires explicit confirmation, for example
`--update cargo:serde --yes`, before it rewrites supported manifests. Any
package-manager fetch or install needed after a manifest change must go through
`sfw`.

## Rust Unit Tests

Module-local Rust tests cover:

| Module | Examples |
|---|---|
| `node_store.rs` | active bits, tombstones, mmap invariants |
| `edge_store.rs` | CSR construction, dedupe, reverse store, weights |
| `bfs.rs` | depth limits, edge filters, overlay traversal, paths |
| `filter_index.rs` | dense/sparse storage and typed filter checks |
| `persistence.rs` | roundtrip and corrupt artifact rejection |
| `types.rs` | parser helpers and filter operation boundaries |
| `config.rs` | GUC parser fallbacks |
| `safety.rs` | SQLSTATE mapping |

## pgrx SQL Tests

`src/pg_tests/` covers installed extension behavior inside PostgreSQL:

| File | Scope |
|---|---|
| `discovery.rs` | schema and table-set discovery |
| `gql.rs` | GQL SQL facade, mapped `CREATE`, ACL, tenant, and internal overlay proof tests |
| `cypher.rs` | openCypher compatibility facade and shared-plan SQL behavior |
| `traversal_api.rs` | traversal helpers, labels, weights, directions, strategies |
| `traversal_paths.rs` | paths, aggregation, strict specs |
| `filters.rs` | typed filter behavior |
| `registration_search.rs` | registration and source search |
| `sync_config_build.rs` | sync mode, GUC, OOM, build scanner, and schema drift behavior |
| `components_jobs.rs` | component functions and job APIs |
| `maintenance_admin.rs` | sync apply, vacuum, maintenance |
| `p8_transaction_labels.rs` | transaction-local relationship labels, savepoints, RLS, and replacement guards |
| `p9_open_types.rs` | open-type width boundaries, exact query output, ACL/RLS, transactions, and cancellation |
| `named_graphs.rs` | default-graph compatibility and named-graph policy guardrails |
| `workflow_search_api.rs` | `find`, `expand`, `find_related` |
| `workflow_relationship_api.rs` | `path`, `connection`, `neighborhood` |
| `workflow_validation.rs` | argument validation and unsupported modes |
| `synthetic_release.rs` | synthetic release-gate SQL fixture through pgrx |

The opt-in P9 high-cardinality query boundary uses a dedicated test database:

```bash
createdb pggraph_open_type_high_cardinality
cd graph
DBNAME=pggraph_open_type_high_cardinality \
  ./tests/heavy/open_type_high_cardinality_query.sh
```

It builds 65,536 exact relationship spellings and checks the 65,534, 65,535,
and 65,536 boundaries across traversal, shortest path, GQL, and Cypher. The
runner rejects database names outside the `pggraph_*` test namespace.

The focused open-type release matrix runs the routine PostgreSQL tests and then
packages, installs, unloads, reloads, and queries the extension on every
supported PostgreSQL major:

```bash
cd graph
./tests/heavy/open_type_release_matrix.sh
```

This Docker-backed gate covers PostgreSQL 14 through 18. The 65,536-label
runner remains a separate PG17 stress gate so the release matrix stays bounded.

See [SQL Tests](./sql-tests) for the full SQL test inventory and the boundary
each layer is meant to cover.

### Provision pgrx Test PostgreSQL

The pgrx SQL tests need a PostgreSQL installation for the target major version
and a pgrx test cluster. They do not need preloaded application tables or
external datasets. Test fixtures in `src/pg_tests/common.rs` create and drop
their own source tables, call `graph.reset(true)`, register graph metadata, and
build the graph inside the test database.

Install PostgreSQL and ensure the matching `pg_config` is available. Examples:

```bash
# macOS/Homebrew PostgreSQL 17
export PG_CONFIG=/opt/homebrew/opt/postgresql@17/bin/pg_config

# Debian/Ubuntu PostgreSQL 17
export PG_CONFIG=/usr/lib/postgresql/17/bin/pg_config
```

Install Nix on machines that own release dependency freshness remediation. The
dependency checker can report flake input updates, but `flake.lock` should be
regenerated and validated with `nix flake lock` rather than edited by hand. On
macOS, upstream Nix requires the daemon installer because single-user installs
are no longer supported.

Install and initialize pgrx once for that PostgreSQL version:

```bash
cargo install cargo-pgrx --version 0.19.1 --locked
cargo pgrx init --pg17 "$PG_CONFIG"
```

Run the SQL tests from the crate directory:

```bash
cd graph
cargo pgrx test pg17
```

Run a focused pgrx test by name when iterating:

```bash
cd graph
cargo pgrx test pg17 synthetic_fixture_exercises_release_gate_sql_shape
```

If pgrx reports an extension or SQL-definition mismatch after code changes,
reinstall the extension against the same PostgreSQL version and rerun:

```bash
cd graph
cargo pgrx install --pg-config "$PG_CONFIG" --features pg17 --no-default-features
cargo pgrx test pg17
```

## Heavy Tests

Some behavior must be tested outside `cargo pgrx test` because pgrx SQL tests
run inside generated SQL functions and do not reflect normal client/server
error boundaries.

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_boundary ./tests/heavy/run_sqlstate_acl_boundary.sh
```

Heavy suites:

| Script | Purpose |
|---|---|
| `run_sqlstate_acl_boundary.sh` | Client-visible SQLSTATEs and restricted-role ACL behavior |
| `tx_delta_lifecycle.sh` | Internal mutable-overlay transaction lifecycle proof |
| `gql_create_tx_lifecycle.sh` | Public mapped GQL `CREATE` rollback and commit lifecycle proof |
| `gql_set_tx_lifecycle.sh` | Public mapped GQL property `SET` rollback and filter-delta lifecycle proof |
| `gql_delete_tx_lifecycle.sh` | Public mapped GQL edge-row `DELETE` rollback and bidirectional tombstone lifecycle proof |
| `gql_write_recheck_race.sh` | Public mapped GQL write predicate recheck race proof |
| `gql_merge_race.sh` | Public mapped GQL `MERGE` same-key race proof |
| `gql_relationship_race.sh` | Public mapped GQL relationship `CREATE`/`DELETE` race proof |
| `tx_delta_crash_recovery.sh` | Internal mutable-overlay crash/reload proof |
| `run_pgbench_sync.sh` | Concurrent DML and trigger sync stress |
| `crash_recovery.sh` | Persisted graph recovery and corruption rejection after postmaster kill |
| `run_memory_sanitizers.sh` | pgrx suite plus optional ASan/Valgrind |
| `package_validate.sh` | Packaging checks |
| `fresh_install_smoke.sh` | Install and quick use smoke |
| `function_metadata_audit.sh` | SQL signature and metadata audit |
| `backup_restore_validate.sh` | Dump/restore state validation |
| `docker_smoke.sh` | Container smoke test |
| `pg_upgrade_validate.sh` | Upgrade validation |
| `run_pg_upgrade_matrix.sh` | Sentinel-owned adjacent PostgreSQL-major upgrade matrix |
| `run_crash_matrix.sh` | PostgreSQL 14–18 persisted and transaction-overlay crash matrix |
| `background_job_lock_regression.sh` | Background job lock/failure behavior |
| `build_lock_regression.sh` | Build/vacuum lock conflict behavior |
| `concurrency_stress.sh` | Mixed build/sync/vacuum/traversal stress |
| `synthetic_release_smoke.sh` | Deterministic synthetic release-gate graph |
| `run_synthetic_release_evidence.sh` | Synthetic release smoke with metadata and artifact report |
| `playground_release_gate.sh` | Docker-backed playground query catalog and result stability |
| `measure_build_rss.sh` | Peak RSS measurement |
| `measure_mmap_pss.sh` | Linux PSS measurement for mmap sharing |

See [SQL Tests](./sql-tests) for the full heavy-script inventory.

### SQLSTATE And ACL Boundary

`run_sqlstate_acl_boundary.sh` validates errors from client-side `psql` output
instead of wrapping calls in nested PL/pgSQL exception handlers. That is closer
to real application behavior and avoids confusing pgrx panic/error plumbing
with PL/pgSQL's nested exception machinery.

It covers representative public errors:

| SQLSTATE | Diagnostic | Boundary |
|---|---|---|
| `55000` | `PG003` | Traversal before the graph has been built. |
| `P0002` | `PG010` | Missing seed node. |
| `22023` | `PG005` | Invalid Unicode-heavy filter input. |
| `55000` | `PG020` | Graph queries disabled by configuration. |
| `42501` | `PG002` | Restricted-role access to global analytics. |

### Concurrent Sync Stress

`run_pgbench_sync.sh` creates a disposable schema, enables trigger-buffered
sync, runs concurrent insert/update/delete traffic through `pgbench`, applies
the sync buffer, and checks post-sync query behavior.

Useful knobs:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_pgbench CLIENTS=16 JOBS=4 TIME=120 \
  RATE=100 MAX_APPLY_MS=5000 MAX_QUERY_MS=250 MIN_ROWS_APPLIED=0 \
  ./tests/heavy/run_pgbench_sync.sh
```

`RATE` keeps the workload bounded so the release gate measures sync behavior
instead of turning `graph.apply_sync()` into an accidental soak test.

### Read Latency Under Sync Backlog

`read_latency_under_sync.sh` captures baseline read latency before changing
query-time sync semantics. It builds a line graph, enables trigger sync, then
records p50/p95/p99 latency for traversal, shortest path, and weighted shortest
path reads across four scenarios: no pending sync, a small pending backlog, a
large pending backlog, and concurrent writers appending to `graph._sync_log`.
For fixed-backlog scenarios, each query kind is measured from a rebuilt graph
with a fresh backlog, and backlog counts come from `graph.sync_health()`.

Useful knobs:

```bash
cd graph
DBNAME=pggraph_read_latency NODE_COUNT=10000 SAMPLES=25 \
  SMALL_BACKLOG=100 LARGE_BACKLOG=5000 CLIENTS=4 JOBS=2 TIME=30 RATE=100 \
  ./tests/heavy/read_latency_under_sync.sh
```

Set `QUERY_FRESHNESS=off` to compare the compatibility/manual catch-up path
against the default query-time catch-up behavior.

The script writes `read-latency-samples.csv` and
`read-latency-summary.csv` in the current directory. Keep those outputs with
milestone evidence when validating topology-read auto-sync behavior.

### Crash Recovery

`crash_recovery.sh` must run only against a disposable PostgreSQL instance. It
reads `PGDATA/postmaster.pid`, sends `kill -9` to that postmaster, restarts the
instance with `pg_ctl`, verifies graph auto-load, corrupts the persisted graph
file, and verifies the loader rejects the corruption cleanly.

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_crash PGDATA=/path/to/disposable/pgdata \
  ./tests/heavy/crash_recovery.sh
```

When the instance needs custom server options, pass the same restart options
with `POSTGRES_OPTS`, for example `POSTGRES_OPTS="-p 55433 -k /tmp"`.

`tx_delta_crash_recovery.sh` uses the same disposable-cluster pattern but
installs the extension with the internal `pg_test` surface. It opens an
uncommitted transaction-local edge overlay, kills the postmaster, restarts the
cluster, and verifies the persisted base graph reloads without the uncommitted
overlay.

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_tx_delta_crash \
  PGDATA=/path/to/disposable/pgdata ./tests/heavy/tx_delta_crash_recovery.sh
```

### Memory And FFI Boundary

`run_memory_sanitizers.sh` runs the pgrx SQL suite and a long-lived pgbench sync
smoke by default. ASan and PostgreSQL-process Valgrind are opt-in because they
need compatible toolchains. The Valgrind gate creates and removes its own data
cluster and instruments the postmaster and its child processes. Direct local
execution installs pgGraph into the PostgreSQL installation selected by
`PG_CONFIG`; use the Docker command below when that installation must remain
untouched.

```bash
cd graph
DBNAME=pggraph_pgbench ./tests/heavy/run_memory_sanitizers.sh
RUN_ASAN=1 ./tests/heavy/run_memory_sanitizers.sh
RUN_ASAN=1 ASAN_TEST_FILTER=mmap_ RUN_PGRX=0 RUN_PGBENCH=0 \
  ./tests/heavy/run_memory_sanitizers.sh
RUN_VALGRIND=1 PG_VERSION_FEATURE=pg17 \
  ./tests/heavy/run_memory_sanitizers.sh
# Containerized release environment:
PG_VERSIONS=17 RUN_PGRX_SQL=0 RUN_GQL_WRITE_MATRIX=0 \
  RUN_POSTGRES_SANITIZER=1 ./tests/heavy/run_pg_matrix_docker.sh
```

On macOS, the script locates and preloads the active nightly toolchain's ASan
runtime before compiling instrumented dependencies. This keeps sanitizer
interceptors available to compiler and test subprocesses without requiring a
machine-specific `DYLD_INSERT_LIBRARIES` value.
`ASAN_TEST_FILTER` can narrow an investigation to one Rust test name or prefix;
leave it unset for the release gate's complete Rust suite.

The PostgreSQL-process gate covers persisted mmap build, load, traversal,
reload, corrupt-artifact rejection, build-job execution, transaction callbacks,
and guarded error unwinding. Its suppression file contains only
stack-qualified PostgreSQL padding reads observed outside `graph.so`; every
remaining Valgrind process log must report zero errors. This is a memory-access
gate, not a leak-retention measurement. Client connections default to a
10-second connection timeout, SQL statements to five minutes, and lock waits to
30 seconds; `PGCONNECT_TIMEOUT`, `STATEMENT_TIMEOUT_MS`, and `LOCK_TIMEOUT_MS`
can adjust those bounds for slower release hosts.

The durable projection matrix reuses the cross-backend lifecycle and lock
regressions on every supported PostgreSQL major:

```bash
cd graph
RUN_RUST_TESTS=0 RUN_PGRX_SQL=0 RUN_GQL_WRITE_MATRIX=0 \
  RUN_DURABLE_PROJECTION_MATRIX=1 ./tests/heavy/run_pg_matrix_docker.sh
```

The gate covers committed and rolled-back writer horizons, competing
publishers, legacy trigger preflight, exact watermark advancement, active
artifact preservation, and build/vacuum exclusion. `RUN_RUST_TESTS=0` is
appropriate when the full supported-major Rust matrix has already passed in
the same release evidence set; leave its default value of `1` for a standalone
matrix run.
Each profile has a ten-minute outer deadline by default. Set
`PROFILE_TIMEOUT_SECONDS` when slower builders need a larger explicit bound.
The matrix also verifies that PostgreSQL relation locks serialize builds with
concurrent DDL: a rename preserves the registered relation OID, while a
drop/recreate operation cannot silently retarget the registration by name.

### Packaging, Docker, Upgrade, And Workers

```bash
cd graph
PG_VERSION_FEATURE=pg17 ./tests/heavy/package_validate.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_install ./tests/heavy/fresh_install_smoke.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_metadata ./tests/heavy/function_metadata_audit.sh
PG_VERSION_FEATURE=pg17 SOURCE_DB=pggraph_backup_src RESTORE_DB=pggraph_backup_dst ./tests/heavy/backup_restore_validate.sh
./tests/heavy/docker_smoke.sh
./tests/heavy/run_pg_upgrade_matrix.sh
```

Worker and memory sizing checks:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_background_lock ./tests/heavy/background_job_lock_regression.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_build_lock ./tests/heavy/build_lock_regression.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_concurrency ./tests/heavy/concurrency_stress.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_rss NODE_COUNT=200000 MAX_RSS_MB=0 ./tests/heavy/measure_build_rss.sh
PG_VERSION_FEATURE=pg17 DB_PREFIX=pggraph_memstress ./tests/heavy/build_memory_stress.sh
DBNAME=graph_test BACKENDS=50 ./tests/heavy/measure_mmap_pss.sh
```

The build-lock regression also verifies that discovery with `build => true`
fails closed with `55P03`/`PG006` when its caller already owns a mapping-catalog
write lock. The failed transaction must not leave a partially discovered table
registered.

`build_memory_stress.sh` composes the RSS harness into baseline, small-batch,
repeated rebuild, and low-memory rebuild profiles. Tune it with `NODE_COUNT`,
`EDGE_COUNT`, `BASELINE_BATCH_SIZE`, `SMALL_BATCH_SIZE`, `REBUILD_ROUNDS`, and
`MAX_RSS_MB`. Use `RUN_BASELINE=0`, `RUN_SMALL_BATCH=0`, `RUN_REBUILD=0`, or
`RUN_LOW_MEMORY_REBUILD=0` to narrow a local run.

`measure_mmap_pss.sh` is Linux-only. It keeps graph-query backends alive and
reads `/proc/<pid>/smaps_rollup` so release evidence can use proportional set
size instead of summed RSS. Current graph artifacts are private anonymous
snapshots per backend; the output must not be interpreted as proof of shared
file-backed graph pages.

`open_type_query_resources.sh` is the P9 Linux capacity harness. It builds the
predeclared high-cardinality fixture, synchronizes 1, 4, or 8 real PostgreSQL
backends across idle, loaded, and active-query barriers, and retains raw
per-backend RSS/PSS samples, aggregate and baseline-subtracted totals, and
projection artifact bytes.
Run it only with a `pggraph_*` disposable database name. P9 acceptance limits
and the exact OAT benchmark cases live in
`release/evidence/engine/2026-08-13-p9-open-type-query/` and are committed
before any result files.

`open_type_query_latency.sh` builds identical low- and high-cardinality
topologies on PostgreSQL 17, keeps warmup and measured transactions in one
pgbench backend, and records deterministic query-result oracles for traversal,
shortest path, GQL, and Cypher. `run_open_type_query_resources_docker.sh`
executes the 1/4/8-backend resource matrix in Linux and copies the raw files,
image metadata, and logs out of the container. Use the exact commands in the
measurement directory README; a green console log without retained raw files
is not P9 evidence.

### Synthetic Release Dataset

`synthetic_release_smoke.sh` creates a deterministic synthetic relational graph
for release gates. This is not a benchmark claim; it is a reproducible test
fixture that validates build, persistence, auto-load, traversal, search,
filters, shortest paths, weighted paths, and component summaries without
requiring a real dataset.

```bash
cd graph
DBNAME=pggraph_synthetic NODE_COUNT=50000 HUB_FANOUT=1000 \
  MAX_BUILD_MS=60000 MAX_QUERY_MS=1000 \
  ./tests/heavy/synthetic_release_smoke.sh
```

The release gate runs it by default. Tune with:

```bash
RUN_SYNTHETIC=1 SYNTHETIC_NODE_COUNT=50000 SYNTHETIC_HUB_FANOUT=1000 \
  SYNTHETIC_MAX_BUILD_MS=60000 SYNTHETIC_MAX_QUERY_MS=1000 \
  ./tests/heavy/run_release_gate.sh
```

Set `RUN_SYNTHETIC=0` only when the local environment cannot run disposable
PostgreSQL databases.

To capture metadata and inspect the persisted `.pggraph` artifact:

```bash
cd graph
NODE_COUNT=50000 HUB_FANOUT=1000 ./tests/heavy/run_synthetic_release_evidence.sh
```

The evidence wrapper writes JSON under `target/release-evidence/` and
uses `scripts/inspect_pggraph_artifact.py` when the artifact is present.

### Playground Release Gate

`playground_release_gate.sh` validates the Streamlit playground as a heavy
release gate. It prepares the Docker-backed Panama dataset, imports the same
shared query catalog used by the playground UI, and checks every sidebar example
against fixed result summaries. Stable examples compare row counts and result
hashes; operational examples with volatile job IDs or timings compare row
counts. Independent query failures do not stop the catalog sweep, so one run
reports every example that needs attention.

The gate verifies the tag-pinned 2026-07-29 Panama benchmark archive checksum
and uses the playground's documented 512 MiB query-memory profile. An archive
replacement therefore fails at preparation instead of silently rewriting
expected hashes.

```bash
cd graph
./tests/heavy/playground_release_gate.sh
PGGRAPH_PLAYGROUND_MODE=mutable ./tests/heavy/playground_release_gate.sh
```

By default the script uses the sandbox Docker container and PostgreSQL port
`55432` and validates the `csr_readonly` playground catalog. Set
`PGGRAPH_PLAYGROUND_MODE=mutable` to prepare `mutable_overlay` and validate the
mutable GQL examples. Set `PREPARE_PLAYGROUND=0` and provide `PGGRAPH_DSN` when
validating an already-prepared playground database. Set
`PGGRAPH_PLAYGROUND_YES=1` only when release automation is allowed to approve
the first Panama dataset download. The aggregate `run_release_gate.sh` is
non-interactive and sets this approval for its playground stage; run that gate
only where downloading and unpacking the documented dataset is acceptable.

## Release Gate

```bash
cd graph
PG_VERSION_FEATURE=pg17 ./tests/heavy/run_release_gate.sh
```

`run_release_gate.sh` is the release-evidence aggregator. It runs formatting,
clippy, docs, Rust tests, pgrx SQL tests, cargo-deny, fuzz binary compile,
projection manifest/segment fuzz seed corpora, package validation, install
smoke, metadata audit, SQLSTATE/ACL boundaries, backup/restore, lock
regressions, concurrency stress, synthetic release smoke, GQL write lifecycle
gates, pgbench sync stress, playground validation, optional crash gates, and
optional Docker smoke depending on its `RUN_*` environment variables.

The default command is not the full production release gate: Docker and crash
gates are off unless explicitly enabled. For staged release evidence, keep
skipped pieces explicit
in the command and record the reason:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DB_PREFIX=pggraph_release_m16 \
  RUN_PLAYGROUND=0 RUN_PGBENCH=0 RUN_DOCKER=0 \
  RUN_CRASH=0 RUN_TX_DELTA_CRASH=0 \
  ./tests/heavy/run_release_gate.sh
```

The named RC tier provisions its own temporary cluster and retains each gate's
combined log and artifact digest:

```bash
python3 scripts/run_release.py --tier rc
```

Use the named tier for reproducible release evidence. When diagnosing the
historical aggregate command, run it through the same disposable-cluster
wrapper:

Enable the heavy production-only pieces explicitly when the environment is
ready:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DB_PREFIX=pggraph_release \
  RUN_DOCKER=1 RUN_CRASH=1 RUN_TX_DELTA_CRASH=1 \
  ../scripts/with_disposable_postgres.sh ./tests/heavy/run_release_gate.sh
```

Run crash gates only through the disposable wrapper because they kill and
restart the postmaster. The wrapper creates a private data directory and socket;
the fixture verifies a per-run sentinel, the connected server's data directory,
and the postmaster command immediately before termination:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_release_crash \
  ../scripts/with_disposable_postgres.sh ./tests/heavy/crash_recovery.sh

PG_VERSION_FEATURE=pg17 \
  DBNAME=pggraph_release_tx_delta_crash \
  ../scripts/with_disposable_postgres.sh ./tests/heavy/tx_delta_crash_recovery.sh
```

`crash_recovery.sh` proves committed trigger rows survive postmaster death,
explicit sync replay works after restart, maintenance can rebuild/persist, and
corrupt persisted artifacts are rejected. `tx_delta_crash_recovery.sh` proves
transaction-local overlays are visible before crash but do not survive restart.

Use Docker for PostgreSQL major matrix validation when local headers are not
available:

```bash
cd graph
./tests/heavy/run_pg_matrix_docker.sh
```

The release-package matrix builds the install tree from a clean source archive
and verifies the control file, generated extension SQL, and shared library for
every supported major:

```bash
cd graph
./tests/heavy/source_archive_smoke.sh
```

The script derives the tag from `graph/Cargo.toml`, requires clean tracked files
and index state, and rejects an explicit tag that does not match the package
version. Ignored build output and untracked evidence cannot enter `git archive`.

Use `package_install_matrix.sh` only when diagnosing one already-extracted
source tree. The release gate uses `source_archive_smoke.sh` so untracked files
from a development checkout cannot satisfy package verification.

Create the reviewed release bundle from an explicit commit after package smoke:

```bash
commit="$(git rev-parse HEAD)"
scripts/prepare_release_bundle.py --ref "$commit" --out-dir dist/release-bundle
scripts/verify_release_bundle.py dist/release-bundle --commit "$commit"
```

The verifier checks the PGXN archive, SPDX dependency SBOM, provenance subject,
release manifest, complete file set, sizes, and SHA-256 digests without
rebuilding any payload.

The Docker matrix runs `run_gql_write_matrix.sh` by default after the Rust and
pgrx suites. This adds the full and persisted isolation profiles and the
deterministic concurrent-writer gates for every selected major. Set
`RUN_GQL_WRITE_MATRIX=0` only for a diagnostic build that is not release
evidence.

PostgreSQL-major Rust unit legs use Cargo's release profile. This keeps the
embedded release-latency contracts tied to optimized code while the pgrx legs
continue to verify the extension inside each PostgreSQL major.

Set `RUN_FULL_MATRIX=1` on `run_release_gate.sh` to run every configured
PostgreSQL major through `run_pg_matrix.sh`. Set `RUN_RSS=1` to include a
large-build peak RSS measurement; use `MAX_RSS_MB` when there is a recorded
release threshold. Set `RUN_BUILD_MEMORY_STRESS=1` to include the opt-in
baseline, small-batch, and rebuild RSS profiles.

External gates can also be run independently:

```bash
cd graph
DBNAME=pggraph_release_pgbench CLIENTS=4 JOBS=2 TIME=30 \
  ./tests/heavy/run_pgbench_sync.sh

IMAGE=pggraph:smoke CONTAINER=pggraph-smoke PG_MAJOR=17 \
  ./tests/heavy/docker_smoke.sh

PGGRAPH_PLAYGROUND_YES=1 ./tests/heavy/playground_release_gate.sh
```

Do not mark release verification complete when a gate is skipped or
interrupted. Either rerun the gate successfully or record an explicit
release-owner waiver with the measured evidence that justifies it.

## Package Release Workflow

The package workflow is manually dispatched in two phases so verified source
and multi-architecture Docker artifacts can be prepared before a signed tag or
public release exists. Run `Prepare Or Publish Packages` with:

```text
mode: prepare
tag: vX.Y.Z
source_ref: <full reviewed commit SHA>
```

The prepare phase validates the commit, builds PostgreSQL 14 through 18 Docker
images, pushes commit-addressed prepared GHCR tags, verifies those prepared
images, and uploads the source bundle as a workflow artifact.

```text
ghcr.io/<owner>/pggraph:pg17-vX.Y.Z-prepared
```

After preparation and artifact review pass, create and verify the signed
annotated tag at the exact commit, push it, and create the GitHub Release. Then
dispatch the workflow again with:

```text
mode: publish
tag: vX.Y.Z
prepare_run_id: <successful prepare run ID>
```

The publish phase verifies the GitHub Release exists, checks that the prepared
tag matches the prepared commit-sha tag, promotes prepared Docker manifests to
the public release tags, publishes PGXN, attaches the PGXN archive to the GitHub
Release, and verifies the final GHCR tags. It must not rebuild the Docker images.

### Separate Homebrew Tap Handoff

The protected package workflow does not publish the external Evokoa Homebrew
tap. After the signed GitHub release exists, the tap owner performs a separate
handoff:

1. Update the formula to the signed 1.2.0 source archive URL and exact SHA-256.
2. Run `brew audit --strict --online pggraph`, `brew style pggraph`, and
   `brew test pggraph` in the tap repository.
3. Install into PostgreSQL 17, query `pg_extension`, and verify the reported
   extension version is `1.2.0`.
4. Announce Homebrew availability only after the formula commit and checksum
   are reviewed. A delayed tap update does not change PGXN/GHCR release bytes.

## Fuzzing

Fuzz targets cover:

| Target | Boundary |
|---|---|
| `load_graph_file` | Artifact loader |
| `node_ref_string` | Node-ref parser |
| `traverse_options` | Traversal option parser |
| `sync_properties` | Sync JSON property parser |
| `structured_filter` | Structured filter shape validation |
| `gql_parser` | GQL parser frontend |
| `cypher_parser` | openCypher compatibility parser frontend |
| `load_projection_manifest` | Durable projection manifest loader |
| `load_projection_segment` | Durable projection segment loader |

Compile smoke:

```bash
cd graph/fuzz
cargo fuzz build
```

Projection release seed corpus:

```bash
cd graph
./fuzz/run_projection_seed_corpora.sh
```

## Benchmarking

See [Benchmarking](./benchmarking) for Criterion benchmark usage, synthetic
release-gate timing, and SQL-facing benchmark requirements.

## Documentation Checks

The repository compiles every MDX source and renders it to static HTML with
deterministic local stand-ins for the documentation host's application-shell
components. This catches MDX execution and markup failures without claiming to
reproduce the separately deployed site shell. External-link checks also verify
the published documentation routes. The docs-host deployment remains
responsible for its own application build and visual regression checks.

Rust public API docs are part of the contract. Before calling documentation
complete:

```bash
cd graph
RUSTDOCFLAGS='-D warnings -D missing-docs -D rustdoc::broken_intra_doc_links -D rustdoc::bare_urls' \
  cargo rustdoc --lib --no-default-features --features 'pg17 development'
cargo test --doc --no-default-features --features 'pg17 development'
```

If doctests require PostgreSQL, mark them `no_run` or keep examples in MDX user
docs instead of rustdoc.
