# Build And Persistence

`graph.build()` constructs a backend-local active engine from registered tables,
edges, and filter columns. When `graph.persist_on_build = true`, the base graph
is written atomically under the selected graph's UUID directory and later
backends can load it through read-only mmap.

The current artifact format is v6. Its 512-byte header describes 26 aligned
sections explicitly. Loaded node arrays, forward and inbound CSR arrays,
relationship-ID sidecars, resolution data, filter values and text dictionaries,
relationship identities, tenant metadata, and per-node tenant tokens remain in
the immutable snapshot. Post-load filter changes use sparse deltas, and newly
observed relationship identities use an owned suffix overlay.

Persisted fixed-width arrays use little-endian encoding. Zero-copy mmap loading
is supported on little-endian targets; an unsupported big-endian target fails
closed with compatibility and rebuild guidance instead of interpreting the
bytes with native typed loads. Loaded node and edge views retain ownership of
their read-only mapping, so replacing or dropping an engine cannot leave a
live view pointing at an unmapped artifact.

When persistence is requested, build, maintenance, and vacuum paths fail closed:
an artifact write error or immediate mmap reload error aborts the operation
instead of installing an unpersisted in-memory fallback.

<Callout type="info">
The `.pggraph` file is derived state. It is not PostgreSQL table storage and it
does not replace PostgreSQL's buffer pool, WAL, MVCC, or crash-recovery
machinery. If an artifact is missing, incompatible, or corrupt, rebuild it from
the source tables.
</Callout>

<Callout type="info">
Background workers are used for asynchronous build and maintenance jobs. They
rebuild/apply/persist graph state, but there is no always-running worker that
updates every backend's already-loaded `Engine` in place.
</Callout>

## Build Commands

Synchronous build:

```sql
SELECT * FROM graph.build();
SELECT * FROM graph.build(mode := 'csr_readonly');
```

Compatibility overload:

```sql
SELECT * FROM graph.build(concurrently := false);
```

Background build:

```sql
SELECT * FROM graph.build(concurrently := true);
SELECT * FROM graph.build_status('<build_id>');
```

Named graph build:

```sql
SELECT * FROM graph.build_graph('customer_360', graph_namespace := 'analytics');
SELECT * FROM graph.build_async_graph('customer_360', graph_namespace := 'analytics');
SELECT * FROM graph.build_status_for_graph('customer_360', graph_namespace := 'analytics');
```

The background path creates a row in `graph._build_jobs`, stores the selected
graph, stores the selected projection mode, and launches a dynamic background
worker. If worker launch fails, the job row is marked failed and the SQL call
reports the error.
If the worker starts and the build later fails, pgGraph records the failed
status and error detail from a fresh worker transaction so
`graph.build_status('<build_id>')` remains the operator source of truth.
Use `graph.build_status_for_graph(...)` to list recent jobs for one named graph.

## Return Columns

`graph.build()` returns:

| Column | Type | Meaning |
|---|---|---|
| `nodes_loaded` | `BIGINT` | Nodes loaded from registered source tables |
| `edges_loaded` | `BIGINT` | Directed edges loaded into the forward CSR store |
| `build_time_ms` | `DOUBLE PRECISION` | Wall-clock build duration |
| `memory_used_mb` | `DOUBLE PRECISION` | Engine memory estimate after build |
| `sync_mode` | `TEXT` | Parsed sync mode at build time |
| `projection_mode` | `TEXT` | Built projection mode: `csr_readonly` or `mutable_overlay` |

`graph.build(concurrently := ...)` returns:

| Column | Meaning |
|---|---|
| `build_id` | Durable job ID, or zero UUID for synchronous compatibility path |
| `status` | `queued`, `running`, `completed`, or `failed` |
| `nodes_loaded`, `edges_loaded`, `build_time_ms`, `memory_used_mb` | Populated after completion |
| `sync_mode` | Sync mode captured for the job |
| `projection_mode` | Projection mode captured for the job |

Named graph job helpers also return `graph_id` and `graph_name` so operators
can attribute queued, running, completed, and failed work to a graph without
joining internal catalog tables.

## Build Pipeline

<FlowDiagram
  trigger="graph.build()"
  steps={[
    {
      title: "Read graph catalog",
      description: "Load registered tables, edges, and filter columns from extension catalog tables."
    },
    {
      title: "Estimate memory",
      description: "Use pg_class.reltuples to estimate node and edge volume before allocating stores."
    },
    {
      title: "Load nodes",
      description: "Read source rows with SPI cursors in graph.build_batch_size batches.",
      substeps: [
        "assign node_idx",
        "write temp node lookup spool",
        "insert table+primary key into ResolutionIndex",
        "collect filter and tenant values"
      ]
    },
    {
      title: "Finalize resolution",
      description: "Convert build-time resolution entries into a compact sorted binary-search index."
    },
    {
      title: "Resolve edges",
      description: "Read edge source rows, resolve endpoints through the temp lookup spool, and batch unresolved edges into a temp edge spool."
    },
    {
      title: "Merge bounded runs",
      description: "When persistence is enabled, merge checksummed node, edge, filter, dictionary, tenant, identity, and resolution runs with a fixed fanout."
    },
    {
      title: "Persist when enabled",
      description: "Stream a generation-specific v6 candidate, fsync it, validate it through the production loader, and switch the current generation with compare-and-swap.",
      isResult: true
    }
  ]}
/>

## Build Memory Guard

Before loading rows, build orchestration estimates memory in checked integer
bytes with registered table and edge row estimates from
`pg_class.reltuples`. When a table has missing or zero statistics, pgGraph uses
an exact row-count preflight rather than treating the source as empty. Reused
source tables are counted from one cached estimate, so a table that contributes
both nodes and multiple registered edges does not require repeated catalog
reads. Rebuild and vacuum preflight reserves a conservative peak-build estimate
against the currently loaded engine, so large rebuilds fail before the backend
allocates the replacement graph.

During construction, pgGraph also holds live reservations for the serving
engine, a safety reserve, nodes, filter and tenant values, resolution entries,
endpoint-resolution work, both CSR directions, relationship identities, and
persistence serialization. The safety reserve is one sixteenth of the
configured limit, capped at 64 MiB. A stale positive row estimate therefore
cannot authorize unbounded construction: the build stops with `PG001` when its
measured phases consume the remaining budget.

`graph.build_batch_size` remains the compatibility row ceiling for cursor and
spool batches. Batches may flush earlier at a byte target derived from the
remaining build budget, bounded between 64 KiB and 8 MiB. A single row larger
than that target is handled alone when it still fits the operation budget.
`graph.build_resource_status()` reports the effective replacement budget after
serving residency and the safety reserve are subtracted, the observed
replacement peak, peak phase, and number of byte-pressure flushes from the most
recent successful build in the current backend.

Every over-budget build returns SQLSTATE `53200` with diagnostic `PG001`.
`graph.oom_action = 'readonly'` remains accepted as a deprecated 1.x
configuration spelling, but it no longer permits construction above the memory
limit. Read-only mode can still describe a serving graph state; it is not an
allocation fallback.

If the replacement graph fits by itself but the current backend cannot hold old
and new graphs together, set `graph.low_memory_build = on`. The build may
unload the current backend-local graph before constructing the replacement,
which lowers peak rebuild memory. This is a slower/lower-memory operational
path when combined with smaller `graph.build_batch_size`, but it cannot avoid
the memory needed by the final replacement graph.

Use:

```sql
SELECT * FROM graph.estimate();
```

to see estimated nodes, edges, memory, configured limit, and whether the graph
fits the current limit.

When persistence is enabled, pgGraph scans directly into bounded external-sort
runs and streams the mmap-ready artifact without constructing a complete owned
replacement. It drops run and scanner workspace before loading the validated
snapshot. Nonpersisted builds continue to use the backend-local owned builder.

## Build Locking

Build and vacuum/maintenance rebuild paths coordinate through a build lock. A
concurrent build or vacuum reports SQLSTATE `55P03` with diagnostic `PG006`.

```text
session A: graph.build() holds build/vacuum lock
session B: graph.build() -> 55P03 / PG006
session C: graph.vacuum() -> 55P03 / PG006
```

## Persistence Files

Default artifact-root layout:

```text
$PGDATA/graph/<graph_id>/main.pggraph
$PGDATA/graph/<graph_id>/main.pggraph.sync
$PGDATA/graph/<graph_id>/projection-generation-<generation>-base.pggraph
$PGDATA/graph/<graph_id>/projection-generation-<generation>.json
$PGDATA/graph/<graph_id>/projection-current.json
```

`graph.data_dir` changes the top-level subdirectory under `$PGDATA`. The graph
directory name is always the catalog `graph_id`, not `graph_name`.
`main.pggraph` is the logical compatibility path and the fallback when no
projection manifest exists. A current manifest takes precedence and selects
the immutable generation-specific base used for serving.

```sql
ALTER SYSTEM SET graph.data_dir = 'graph';
ALTER SYSTEM SET graph.persist_on_build = true;
ALTER SYSTEM SET graph.auto_load = true;
SELECT pg_reload_conf();
```

## Atomic Write Contract

Persisted build and vacuum publish a generation as:

```text
projection-generation-<generation>-base.pggraph
  create exclusively
  reserve and zero-fill exact length
  stream graph sections at checked offsets
  recompute and backpatch CRC/header
  fsync artifact and sidecars
projection-generation-<generation>.json
  record candidate checksum, source watermark, and predecessor
projection-current.json
  switch only if the previously observed generation is still current
```

Readers pin and resolve one immutable current manifest. They therefore see the
old complete generation or the validated new generation, never a partial
candidate. Failed publication preserves the old pointer and previous artifact.

Background build and maintenance jobs expose persistence phases through
`progress_phase` and `progress_message`: `persisting` covers artifact write and
fsync, and `validating_persistence` covers immediate mmap reload validation.

## Auto-Load

When a query needs the selected graph and the backend-local engine is empty:

1. `ensure_current_graph()` calls `maybe_auto_load()`.
2. If `graph.auto_load = false`, it does nothing.
3. If the selected graph has `cold` residency, it does nothing.
4. If the artifact does not exist, it does nothing.
5. If the artifact exists, `load_graph_file()` pins the current generation and
   validates the v6 header, both CRCs, all 26 descriptors, and section contents before exposing immutable
   sections read-only.
6. The engine is marked built and can answer queries.

Each backend has one loaded graph slot. If a different graph is selected, the
previous engine is cleared from that backend before another graph is loaded.
Use `graph.loaded_graphs()` to inspect the graph currently loaded in the active
backend, and use `graph.graph_runtime_status()` to inspect residency, artifact
presence, and loaded state for visible graphs.

For explicit operator-controlled loading, use:

```sql
SELECT * FROM graph.load_graph('customer_360', namespace := 'analytics');
SELECT * FROM graph.unload_graph('customer_360', namespace := 'analytics');
```

`graph.load_graph()` loads the persisted artifact immediately and does not
depend on `graph.auto_load`. It can load a `cold` graph explicitly. Missing,
corrupt, incompatible, or quota-blocked artifacts raise an error so they can be
rebuilt from the PostgreSQL source tables or handled by an operator.

<Callout type="warning">
Corrupt or incompatible artifacts are rejected. A corrupt file raises or logs a
SQLSTATE `XX001`/diagnostic `PG009` loader failure depending on the call site;
an incompatible file uses SQLSTATE `0A000`/diagnostic `PG011`. Rebuild from
source tables with `SELECT * FROM graph.build();`.
</Callout>

## Status Checks

```sql
SELECT * FROM graph.status();
```

Important build and persistence columns:

| Column | Meaning |
|---|---|
| `node_count` | Node slots in the active backend engine |
| `edge_count` | Directed edge rows in the forward CSR store |
| `memory_used_mb` | Engine memory estimate |
| `last_build` | Last build timestamp for this backend engine |
| `last_vacuum` | Last vacuum timestamp for this backend engine |
| `schema_status` | `current`, `stale`, or `invalid` |
| `needs_rebuild` | Catalog/schema drift requires rebuild |
| `invalid_reason` | Human-readable reason for invalid schema state |
| `read_only` | Engine has entered read-only mode |

## Rebuild Triggers

Rebuild with `graph.build()` when:

| Situation | Why |
|---|---|
| Registered tables or edges changed | Catalog fingerprint no longer matches the built base graph |
| Source topology changed without sync maintenance | CSR edges are immutable in the base graph |
| Table schema changed | Validation may mark schema state invalid |
| `.pggraph` format version changed | Version 5, earlier, and future artifacts fail closed and must be rebuilt |
| Corruption is detected | Loader rejects bad magic, CRC, offset, bounds, or section content |

## Reset

```sql
SELECT graph.reset();
```

`graph.reset()` clears the backend-local engine and removes persisted files for
the selected graph's UUID directory. It requires graph-admin privileges.

<Callout type="error">
`graph.reset()` removes the selected graph's persisted artifact root. The source
tables remain unchanged, but graph queries for that graph need a rebuild or a
new persisted artifact before they can run again.
</Callout>