# pg_vault_tde Technical Reference

**Version**: 1.7  
**PostgreSQL**: 17.x, 18.x (19.x planned)  
**License**: BSD (PostgreSQL License)  
**Copyright**: © 2026 Miriade S.r.l.

---

## Table of Contents

1. [Architecture](#architecture)
2. [Table Access Method (TAM)](#table-access-method)
3. [Crypto Layer](#crypto-layer)
4. [Performance](#performance)
5. [Index Access Method (IAM)](#index-access-method-iam)
6. [Logical Decoding and Replication](#logical-decoding-and-replication)
7. [PKCS#11 / HSM Provider](#pkcs11--hsm-provider)
8. [Known Limitations](#known-limitations)
9. [Security Considerations](#security-considerations)
10. [Wire Format Reference](#wire-format-reference)
11. [SQL API Reference](#sql-api-reference)
12. [Extension Initialization](#extension-initialization)
13. [Testing Strategy](#testing-strategy)
14. [Packaging](#packaging)
15. [Roadmap](#roadmap)
16. [Contributing](#contributing)

---

## Architecture

pg_vault_tde is a PostgreSQL extension that provides Transparent Data
Encryption at the Table Access Method layer. It operates entirely within
the extension API; zero modifications to PostgreSQL core are required.

### PostgreSQL Version Compatibility

| PG Major | Status | Notes |
|----------|--------|-------|
| 17 | ✅ Supported | Baseline API set |
| 18 | ✅ Supported | `scan_bitmap_next_tuple` signature change — guarded with `PG_VERSION_NUM` |
| 19 | 🔜 Planned | Infrastructure ready; audit at release |

This is the canonical PostgreSQL support statement for the extension; it says
nothing about operating systems or about which combinations are actually
exercised in CI — see [Support Matrix](#support-matrix) under Packaging for that.

### Version-Specific API Differences

| API / Struct | PG 17 | PG 18 | Guard Macro |
|-------------|-------|-------|-------------|
| `scan_bitmap_next_tuple` | `(scan, slot, recheck)` | `(scan, slot, recheck, lossy, exact)` | `PG_VERSION_NUM >= 180000` |
| `shmem_request_hook` | Available | Available | *(none needed)* |
| `GetHeapamTableAmRoutine()` | Returns `const *` | Returns `const *` | *(none needed)* |
| `tuplesort_begin_index_btree()` assert on `relam` | Asserts `rd_rel->relam == BTREE_AM_OID` | No such assert | `PG_VERSION_NUM < 180000` — `pg_vault_tde_ambuild()` temporarily impersonates `BTREE_AM_OID` on `index->rd_rel->relam` during the sort |
| `BuildSpeculativeIndexInfo()` assert on ON CONFLICT | Asserts against non-btree-looking `relam` for unique `tde_btree` indexes | No such assert | `PG_VERSION_NUM < 180000` — `tde_executor_start_hook()` swaps `relam` back to `BTREE_AM_OID` for the duration of the query, restored via a `MemoryContextCallback` |

### Design Goals

| Goal | Achieved | Notes |
|---|---|---|
| Plug-and-play | ✅ | `shared_preload_libraries` + `CREATE EXTENSION` only |
| Zero core patches | ✅ | Pure extension API (`tableam`, `indexam`) |
| AES-256-GCM per-tuple | ✅ | Authenticated encryption; integrity verified on read |
| Hardware acceleration | ✅ | OpenSSL 3.x EVP dispatch → AES-NI / ARM Crypto |
| Key rotation | ✅ | Lazy generation-epoch detection; no scan needed |
| MVCC compatibility | ✅ | HeapTupleHeader stays plaintext |
| pg_dump / pg_restore | ✅ | Decrypts transparently at TAM scan layer |
| Page checksums | ✅ | Checksums cover encrypted bytes |

### Component Map

```
PostgreSQL Core
 └── Extension API
      ├── TAM: encrypted_heap          src/tam/pg_vault_tde_tam.c
      │    ├─ Write path (encrypt)     tde_encrypt_heap_tuple()
      │    └─ Read paths (decrypt)     pg_vault_tde_decode_slot()
      ├── IAM: tde_btree               src/iam/pg_vault_tde_iam.c
      │    └─ AES-256-SIV key encrypt  tde_iam_encrypt_key()
      ├── Crypto                       src/crypto/pg_vault_tde_crypto.c
      │    ├─ tde_gcm_encrypt()        AES-256-GCM via OpenSSL 3.x EVP
      │    └─ tde_gcm_decrypt()        Authenticated decryption
      ├── Per-relation DEK cache       src/kms/pg_vault_tde_catalog.c
      │    ├─ DEK cache (shmem HTAB)    TdeRelDekMap (one shared LWLock)
      │    └─ get DEK for a relation    pg_vault_tde_kms_get_rel_dek(relid)
      ├── KMS provider vtable           src/kms/pg_vault_tde_kms.c
      │    └─ Vault provider (libcurl)  vault_provider_{wrap,unwrap,rewrap}_dek()
      ├── Local wallet provider        src/kms/pg_vault_tde_kms_local.c (PKCS#12)
      ├── DEK catalog (on-disk)        src/kms/pg_vault_tde_catalog.c (wrapped_dek)
      ├── Rotation background worker   src/kms/pg_vault_tde_rotation_bgw.c
      ├── HW acceleration              src/crypto/pg_vault_tde_hw_accel.c
      ├── Logical decoding plugin      src/logical/pg_vault_tde_pgoutput.c
      ├── Backup                       src/backup/pg_vault_tde_backup.c
      │                                (+ pg_dump_tde.c / pg_restore_tde.c)
      └── Entry point                  src/pg_vault_tde.c (_PG_init)
```

### Key Lifecycle

How each provider is *configured* — GUCs, credentials, per-database settings —
is covered in [Configure Key Access](../README.md#3-configure-key-access) for
Vault / OpenBao, the local wallet and PKCS#11. This chapter describes what
happens to key material once a provider is in place; the PKCS#11 provider gets
its own chapter below because it is the only one with meaningful implementation
surface of its own.

```
Vault / OpenBao (KEK owner)  ──or──  Local wallet (PKCS#12, KEK-on-disk)
        │
        │  unwrap wrapped_dek via active provider vtable (synchronous;
        │  libcurl HTTP(S) for Vault, AES-256-WRAP for local)
        ▼
pg_vault_tde_kms_get_rel_dek(relid)    [src/kms/pg_vault_tde_catalog.c]
        │   fast path: LW_SHARED hash_search of TdeRelDekMap (cache hit)
        │   slow path: read pg_vault_tde_catalog.wrapped_dek → provider
        │              unwrap → hash_search(HASH_ENTER) under LW_EXCLUSIVE
        ▼
TdeRelDekMap (shmem HTAB)              [one entry per (dbid, relid); single
        │                               shared LWLock; generation + prev_dek]
        │  DEK (32 bytes) copied into a stack buffer on every call; the crypto
        │  layer caches the AES key schedule keyed by (relid, generation) — no
        │  dbid there, those statics are per-backend and a backend is bound to
        │  one database
        ▼
tde_gcm_encrypt() / tde_gcm_decrypt()  [src/crypto/pg_vault_tde_crypto.c]
        │
        ▼
Disk: [HeapTupleHeader | attributes, value bytes encrypted | IV(12) | GCM-TAG(16) | VER(1) | GEN(8)]
```

### Shared Memory Layout

Since v1.7 the cache is a shared-memory **hash table** (`HTAB`), not a fixed
array scanned linearly. Each entry is one `TdeRelDekMap`, keyed by
`(dbid, relid)`:

```c
/*
 * Cache key.  relid is unique only WITHIN a database — never across
 * databases, never cluster-wide — while this HTAB is one segment read by the
 * backends of every database.  CREATE DATABASE physically copies the
 * template's directory, so a clone hands out pg_class OIDs identical to its
 * template's: colliding relids are normal, not a corner case.
 */
typedef struct TdeRelDekMapKey {
    Oid     dbid;                    /* always MyDatabaseId */
    Oid     relid;                   /* effective relid (TOAST → parent, etc.) */
} TdeRelDekMapKey;

/* Per-relation DEK entry — value type of the TdeRelDekMap HTAB (v1.5+) */
typedef struct TdeRelDekMap {
    TdeRelDekMapKey key;             /* hash key */
    char    dek[TDE_DEK_LEN];        /* current AES-256 DEK, 32 bytes */
    char    prev_dek[TDE_DEK_LEN];   /* previous DEK (valid during rotation) */
    uint64  generation;              /* rotation epoch for this relation */
    bool    dek_valid;               /* true iff dek[] holds a live key */
    bool    prev_dek_valid;          /* true iff prev_dek[] is populated */
} TdeRelDekMap;
```

- `dbid` is **not** a parameter of any public function in
  `pg_vault_tde_catalog.h`.  `tde_rel_dek_key()` fills it from `MyDatabaseId`
  when it builds the key, because every path that reaches the cache runs
  connected to the database owning the relation: a regular backend, the
  rotation BGW after `BackgroundWorkerInitializeConnectionByOid()`, or a
  walsender during logical decoding.  One derivation point instead of
  fourteen call sites that could each pass the wrong value.
- The **on-disk** catalog needs no dbid.  `pg_vault_tde_catalog` is an
  ordinary table created by `CREATE EXTENSION` in the extension's schema, so
  it exists once per database and its `relid` primary key is unambiguous
  there; every access (`tde_catalog_read_row`, `rewrap_all`,
  `read_all_wrapped`, `upsert_row`) goes through `table_open()` in the current
  backend's database.  The local wallet is per-database for the same reason,
  at `/var/lib/pg_vault_tde/<db_oid>/wallet.p12`.  Shared memory was the only
  place where per-database namespaces met.
- A relid-only key does **not** silently return wrong plaintext: the GCM AAD
  binds `MyDatabaseId` (`tde_compute_aad()`), so the victim database gets
  `AES-256-GCM authentication FAILED` on intact data.  A read outage, not a
  corruption — covered by `tap/21_cache_key_cross_db.t`.
- `TDE_DEK_LEN` is defined **only** in `src/include/pg_vault_tde_kms.h`.
- The HTAB lives in `src/kms/pg_vault_tde_catalog.c`, created with
  `ShmemInitHash("pg_vault_tde_rel_dek_map", capacity, capacity, &info, HASH_ELEM | HASH_BLOBS)`
  where `capacity = pg_vault_tde.max_encrypted_relations`. The segment is sized
  with `hash_estimate_size(capacity, sizeof(TdeRelDekMap))`.  `HASH_BLOBS`
  means the key is hashed as raw bytes, so `tde_rel_dek_key()` zeroes the
  struct before filling it — padding must not leak into the hash.  Capacity is
  cluster-wide: with encrypted tables in several databases, budget for the sum.
- There is **no per-entry lock**. A single `LWLock` (file-scope `rel_dek_lock`)
  from a **named** tranche guards the whole table:
  `RequestNamedLWLockTranche("pg_vault_tde_rel_dek_map", 1)` in the `shmem_request_hook`,
  then `&GetNamedLWLockTranche("pg_vault_tde_rel_dek_map")[0].lock` in the
  `shmem_startup_hook`. The lock is taken `LW_SHARED` for lookups and
  `LW_EXCLUSIVE` for insert/evict/rotate.
- A second, fixed-size shmem struct (`pg_vault_tde_kms_cache`, in
  `pg_vault_tde_kms.c`) holds the shared Vault token. Its lock uses a **dynamic**
  tranche (`LWLockNewTrancheId()`), which is why that call lives in the
  `shmem_startup_hook` and not `_PG_init` — see [Extension Initialization](#extension-initialization).

---

## Table Access Method

### Design Pattern: Mutable Copy of heapam

```c
static TableAmRoutine tde_methods;  /* zero-initialised at load time */

void pg_vault_tde_tam_init(void) {
    memcpy(&tde_methods, GetHeapamTableAmRoutine(), sizeof(TableAmRoutine));
    /* save originals, then install TDE wrappers */
}
```

All structural operations (VACUUM, CLUSTER, index build, truncate, scan state management) delegate to heapam unchanged. Only the five write paths, eight read paths, two visibility/build paths, and two rewrite path are overridden.

### Overridden Callbacks

#### Write Paths (encrypt before storing)

| Callback | Purpose |
|---|---|
| `tuple_insert` | Single-row INSERT |
| `tuple_insert_speculative` | Speculative INSERT (ON CONFLICT) |
| `multi_insert` | COPY FROM / bulk INSERT |
| `tuple_update` | UPDATE |
| `tuple_delete` | DELETE |

All write paths follow the same pattern:
1. Materialize the slot into a `HeapTuple` (plaintext)
2. Call `tde_encrypt_heap_tuple()` → returns palloc'd encrypted `HeapTuple`
3. Call the heapam storage function (`heap_insert`, `heap_update`)
4. Copy the physical TID back to the slot
5. `OPENSSL_cleanse` + `pfree` the plaintext copy

#### Read Paths (decrypt after fetching)

All read paths that cause heapam to fill a `TupleTableSlot` with a
buffer-backed `HeapTuple` MUST call `pg_vault_tde_decode_slot()`.

| Callback | Scan Type | Status |
|---|---|---|
| `scan_getnextslot` | SeqScan | ✅ Override |
| `scan_getnextslot_tidrange` | TidRangeScan | ✅ Override |
| `index_fetch_tuple` | Index Scan, Index Only Scan | ✅ Override |
| `scan_bitmap_next_tuple` | BitmapHeapScan | ✅ Override |
| `scan_analyze_next_tuple` | ANALYZE | ✅ Override |
| `scan_sample_next_tuple` | TABLESAMPLE | ✅ Override |
| `tuple_fetch_row_version` | TidScan, UPDATE recheck | ✅ Override |
| `tuple_lock` | SELECT FOR UPDATE/SHARE | ✅ Override |

#### Visibility & Index-Build Paths
 
| Callback | Purpose |
|---|---|
| `tuple_satisfies_snapshot` | Visibility recheck for RI foreign-key trigger (`RI_FKey_check`) — heapam's version asserts a live buffer pin, which our decrypt-into-palloc'd-tuple path doesn't hold |
| `index_build_range_scan` | CREATE INDEX / REINDEX — decrypts each tuple before `FormIndexDatum` extracts key values, otherwise indexes would be built over ciphertext|


#### Rewrite Paths (decrypt → process → re-encrypt)

| Callback | Trigger | Notes |
|---|---|---|
| `relation_copy_for_cluster` | `VACUUM FULL`, `CLUSTER` | Reads each tuple via `heap_getnext` (with `rd_tableam` impersonation), decrypts, re-encrypts into the new heap via `rewrite_heap_tuple`. Clears `HEAP_HASEXTERNAL` on the encrypted copy before writing; `tde_tuple_has_external_slow` (per-attribute varlena scan) is used on subsequent DELETE to locate TOAST chunks regardless of the infomask flag. |
| `relation_toast_am` | TOAST table creation | Always selects `encrypted_heap` as the TOAST AM (since 1.7.2 regardless of `pg_vault_tde.toast_encryption`, PSQLE-223), so the encrypted chunks are decrypted through the same `index_fetch_tuple`/`scan_getnextslot` hooks as the main table. |

### pg_vault_tde_decode_slot

This function is the core of the read path. It:

1. Casts the slot to `BufferHeapTupleTableSlot` (known buffer-backed after heapam)
2. **Guards against double-decode**: checks `bslot->buffer == InvalidBuffer`
3. Saves `bslot->base.tuple->t_self` (physical TID) and `t_tableOid` (relation OID) from the buffer page
4. Calls `tde_decrypt_heap_tuple(bslot->base.tuple, saved_tableoid)` — decrypts the
   buffer-backed tuple directly (verifies GCM tag via OpenSSL). **No `heap_copytuple`
   and no explicit `ExecClearTuple`**: the buffer pin must stay held until the
   force-store below. Releasing it early forces O(rows) buffer re-pins during a
   sequential scan instead of O(pages) (see performance note in the function header).
5. Stamps `plain->t_self = saved_tid`, `plain->t_tableOid = saved_tableoid`
6. Calls `ExecForceStoreHeapTuple(plain, slot, true)` — this performs the single
   internal `ExecClearTuple` that releases the buffer pin (the only release point)
7. **Manually sets `slot->tts_tid = saved_tid`** — `ExecForceStoreHeapTuple`
   does NOT restore `tts_tid` for buffer slots; it must be set explicitly

```c
static void pg_vault_tde_decode_slot(TupleTableSlot *slot)
{
    BufferHeapTupleTableSlot *bslot = (BufferHeapTupleTableSlot *) slot;
    HeapTuple   plain;
    ItemPointerData saved_tid;
    Oid         saved_tableoid;

    /*
     * Guard against double-decode: after ExecForceStoreHeapTuple the buffer
     * is released (buffer == InvalidBuffer) but base.tuple is still set.
     * Re-entering here would try to decrypt already-plain data.
     */
    if (bslot->buffer == InvalidBuffer)
        return;

    /*
     * Read TID + relation OID from the buffer-backed pointer directly, NOT from
     * ExecFetchSlotHeapTuple(slot, false, ...) which returns the tupdata
     * workspace with an uninitialized t_self.
     */
    ItemPointerCopy(&bslot->base.tuple->t_self, &saved_tid);
    saved_tableoid = bslot->base.tuple->t_tableOid;

    /*
     * Decrypt the buffer-backed tuple in place (verifies GCM tag; ereport(ERROR)
     * on tamper).  Do NOT call heap_copytuple/ExecClearTuple first: the pin must
     * stay held until ExecForceStoreHeapTuple, which releases it exactly once per
     * tuple.  Releasing early causes O(rows) buffer hits on sequential scans.
     */
    plain = tde_decrypt_heap_tuple(bslot->base.tuple, saved_tableoid);

    ItemPointerCopy(&saved_tid, &plain->t_self);
    plain->t_tableOid = saved_tableoid;

    ExecForceStoreHeapTuple(plain, slot, true);   /* internal ExecClearTuple releases pin */
    ItemPointerCopy(&saved_tid, &slot->tts_tid);  /* ExecForceStoreHeapTuple does not set this */
}
```

### rd_tableam Identity-Check Workaround

Several heapam internal functions protect themselves with:
```c
if (rel->rd_tableam != GetHeapamTableAmRoutine())
    ereport(ERROR, "only heap AM is supported");
```

Since our `&tde_methods` lives at a different address than heapam's static
struct, these checks fail when called on our tables. Two callbacks require
the workaround:

1. **`index_fetch_tuple`** — calls `heap_hot_search_buffer` on every index lookup
2. **`index_build_range_scan`** — calls `heap_getnext` internally during `CREATE INDEX`

**Fix pattern** (safe — `RelationData` is per-backend):
```c
const TableAmRoutine **rdam =
    (const TableAmRoutine **)(void *)&rel->rd_tableam;
const TableAmRoutine *saved_am = *rdam;
*rdam = GetHeapamTableAmRoutine();   /* impersonate heapam */
result = heapam_original_cb(rel, ...);
*rdam = saved_am;                    /* restore BEFORE any error path */
/* Then decrypt slot contents */
```

`RelationData` is per-backend (local relcache copy). The swap window is a
single function call. No signal/interrupt can preempt between the swap and
restore in a single-threaded backend.

### TOAST Table Override

```c
static Oid pg_vault_tde_toast_am(Relation rel)
{
    if (!pg_vault_tde_toast_encryption)
        ereport(WARNING, ... "has no effect" ...);   /* PSQLE-223 */
    encheap_oid = get_table_am_oid("encrypted_heap", true);
    if (OidIsValid(encheap_oid))
        return encheap_oid;
    return HEAP_TABLE_AM_OID;       /* only if the AM is missing */
}
```

TOAST tables are created with the `encrypted_heap` AM so that every TOAST chunk is
encrypted individually using the parent relation's DEK.  On PG 18 the `heap_getnext`
identity assertion inside the TOAST index build would reject `encrypted_heap`;
the `rd_tableam` impersonation workaround is applied during
`index_build_range_scan` to satisfy this assertion.

`pg_vault_tde.toast_encryption` has had no effect since 1.7.2 (PSQLE-223). The TOAST
pipeline encrypts every chunk it writes whatever the TOAST table's AM, so `off` never
gave plaintext TOAST: up to 1.7.1 it gave a heap TOAST table of encrypted chunks, read
back undecrypted, and every out-of-line value of the table failed. Setting it `off` now
only raises a `WARNING`; the parameter is removed in 1.8. A table left unreadable by
1.7.1 is repaired by giving its TOAST table the `encrypted_heap` access method — see
README.md › Upgrading to 1.7.2 › Tables created with `toast_encryption = off`.

### PG18-Specific API Notes

#### scan_bitmap_next_tuple

The signature changed in PG18:

```c
/* PG ≤ 17 */
bool (*scan_bitmap_next_tuple)(TableScanDesc, TBMIterateResult *,
                               TupleTableSlot *);
/* PG 18 */
bool (*scan_bitmap_next_tuple)(TableScanDesc, TupleTableSlot *,
                               bool *, uint64 *, uint64 *);
```

Always verify against `src/include/access/tableam.h` before implementing
or modifying this callback.

---

## Crypto Layer

### Algorithm

- **Encryption**: AES-256-GCM via OpenSSL 3.x `EVP_EncryptInit_ex2`
- **IV generation**: `pg_strong_random()` (PostgreSQL's `/dev/urandom` wrapper)
  — NOT `RAND_bytes()` because PostgreSQL processes can fork at any time;
  OpenSSL PRNG state is not fork-safe in all configurations.
- **Hardware acceleration**: OpenSSL 3.x EVP dispatch automatically selects
  the hardware provider (AES-NI on x86_64; ARM Crypto Extensions on aarch64).
- **Authentication**: 128-bit GCM tag appended to every encrypted region.
  Any bit-flip in ciphertext, IV, or associated data raises an `ERROR`
  (not a silent wrong result).

### Wire Format per Encrypted Region

**Version 5** is the format written today; **version 4** is still read, so a table
written before the upgrade keeps working. The legacy v1/v2/v3 formats were
**removed**. (The byte `0x02` still appears only in the `pg_dump_tde` *backup
block* format — a separate code path, see [Backup](../README.md#encrypted-backups).)

v5 is **structure preserving**: every attribute stays at its own offset with its
own length, and only the bytes of the VALUES are replaced by ciphertext.

```
+--------------------------------+----------+----------+-------+----------+
| ATTRIBUTES, values encrypted   | IV       | GCM TAG  | VER   | GEN      |
| D bytes (= plaintext data len) | 12 bytes | 16 bytes | 1 byte| 8 bytes  |
+--------------------------------+----------+----------+-------+----------+
                                   random               0x05   uint64 LE
```

Total overhead: `TDE_V4_OVERHEAD = 37` bytes
(`TDE_GCM_IV_LEN=12` + `TDE_GCM_TAG_LEN=16` + `1` version byte + `TDE_V4_GEN_LEN=8`),
identical to v4 — a v5 tuple is exactly as long as the v4 tuple for the same row.

v4 replaced the whole user-data region with one opaque blob
(`[IV | CIPHERTEXT | TAG | VER | GEN]`) while the tuple header, copied verbatim,
still advertised `natts` attributes laid out per the tuple descriptor. Core code
that deforms an on-disk tuple then walked ciphertext as if it were a tuple —
and `heap_update()` does exactly that, reading the indexed attributes straight
off the page to decide HOT and which indexes to maintain. Past the first
variable-length column the attribute offset is not cached, so the walk read a
varlena length header out of ciphertext, got a length of up to 1 GB and left the
page: **SIGSEGV** (PSQLE-165). Any index on such a column was enough, `tde_btree`
included — the trigger is the index attribute bitmap, not the access method.

What v5 gives up in exchange: the structural bytes stay in clear, because they
are what makes the walk possible. Concretely the **exact byte length of every
variable-length column** is visible in the heap file, along with whether the
value is compressed or held out of line. Fixed-length columns leak nothing (their
length is in the catalog), and the row length and null bitmap were already
visible in v4. Attribute values themselves are never in clear — regression test
143 reads the raw heap file and asserts it.

The AEAD is unchanged: the value bytes are gathered into one buffer, handed to
`tde_gcm_encrypt()` and scattered back, so ciphertext, tag and AAD are
bit-identical to what v4 produced for the same input.

**Upgrading an existing table.** v4 tuples are read transparently, but they keep
their old layout: `UPDATE` on a v4 row with an index behind a variable-length
column still crashes, because nothing can make that layout walkable after the
fact. `VACUUM FULL` (or `CLUSTER`) rewrites every row through the TAM and
migrates the table to v5.

v4 binds each tuple to its location by passing
`[MyDatabaseId(4) | relid(4) | generation(8)]` (little-endian, `TDE_V4_AAD_LEN = 16`
bytes) as GCM Additional Authenticated Data — zero wire overhead; prevents
cross-table ciphertext smuggling. The AAD is reconstructed on decrypt from the
**stored** generation in the wire trailer (not the current generation), so old-generation
rows still authenticate during the rotation window.

When `user_len == 0` (all-NULL tuple, or tuple with only system columns),
`tde_gcm_encrypt()` still produces a full `TDE_V4_OVERHEAD`-byte (37) block. This
exercises the GCM tag path on zero data; the decrypt path handles it symmetrically.
Test 16 covers this edge case.

### Memory Security

- DEK copies in per-backend memory are `OPENSSL_cleanse`d before `pfree`.
- Plaintext `HeapTuple` intermediates are `OPENSSL_cleanse`d after encryption.
- Per-backend EVP contexts are freed via `on_proc_exit()` callbacks
  (`tde_iam_ctx_cleanup` for AES-SIV; analogous cleanup for GCM contexts).
- Shared-memory DEK is `OPENSSL_cleanse`d during rotation before the new key
  is written.

---

## Performance

### Per-Backend EVP Contexts Keyed by (relid, generation)

Two costs hide on the per-tuple crypto path: allocating an `EVP_CIPHER_CTX`
(a heap malloc) and installing the AES-256 **key schedule**
(`EVP_EncryptInit_ex2` with the DEK). A naive implementation pays both on every
tuple. pg_vault_tde caches each context together with the key it is keyed for:

```c
typedef struct TdeCipherSlot {
    EVP_CIPHER_CTX *ctx;
    Oid             relid;
    uint64          generation;
} TdeCipherSlot;

static TdeCipherSlot tde_enc = { NULL, InvalidOid, 0 };  /* encrypt direction */
static TdeCipherSlot tde_dec = { NULL, InvalidOid, 0 };  /* decrypt direction */
```

Each context is allocated once per backend (`EVP_CIPHER_CTX_new()` on first
use). The expensive key-schedule install runs **only when the slot's cached
`(relid, generation)` differs** from the current operation — i.e. on the first
tuple of a relation and again after a key rotation. For every other tuple the
installed schedule is reused and only the per-tuple IV is rearmed with
`EVP_EncryptInit_ex2(ctx, NULL, NULL, iv, NULL)`. Consecutive tuples of the same
relation (the common bulk-INSERT / sequential-scan case) therefore skip the
key schedule entirely.

On decrypt the slot is keyed by the **stored** generation read from the wire
trailer, so old-generation rows decrypted via `prev_dek` during the rotation
window get their own cached schedule without thrashing the current-generation
one.

On any fatal OpenSSL error `tde_crypto_ctx_cleanup()` frees and NULL-outs both
contexts (and resets their `relid` to `InvalidOid`) so the next call
re-allocates and re-keys cleanly. Both contexts are freed in the
`on_proc_exit()` callback `tde_crypto_ctx_cleanup()`, which also wipes the IV
batch. The same allocate-once pattern is applied to the IAM: a single per-backend
AES-256-SIV context, re-keyed only when the `(idx_oid, generation)` pair changes
(`tde_iam_ctx_prepare`), freed by `tde_iam_ctx_cleanup()`.

### IV Batch Generation

`pg_strong_random()` is a syscall to `/dev/urandom` or `getrandom(2)`. A
non-batched implementation would pay one syscall per encrypted tuple. Instead,
pg_vault_tde batches 256 IVs per `pg_strong_random()` call:

```c
#define TDE_IV_BATCH_SIZE   256
#define TDE_IV_BATCH_BYTES  (TDE_IV_BATCH_SIZE * TDE_GCM_IV_LEN)
static char  iv_batch[TDE_IV_BATCH_BYTES];
static int   iv_batch_pos = TDE_IV_BATCH_SIZE;  /* start empty */
static int   iv_batch_pid = 0;                   /* the process that filled it */

static void tde_next_iv(unsigned char *iv_out)
{
    if (iv_batch_pos >= TDE_IV_BATCH_SIZE || iv_batch_pid != MyProcPid)
    {
        if (!pg_strong_random(iv_batch, TDE_IV_BATCH_BYTES))
            ereport(ERROR, (errmsg("[CRYPTO] Failed to generate IV batch")));
        iv_batch_pos = 0;
        iv_batch_pid = MyProcPid;
    }
    memcpy(iv_out, iv_batch + iv_batch_pos * TDE_GCM_IV_LEN, TDE_GCM_IV_LEN);
    iv_batch_pos++;
}
```

The buffer is wiped with `OPENSSL_cleanse()` in the backend-exit cleanup.

This amortises the syscall cost across 256 tuples.

A batch belongs to the process that filled it. A `fork()` copies it, and two
processes serving the same IVs under one DEK would void GCM for those tuples. No
PostgreSQL process forks after drawing an IV — the postmaster encrypts nothing — so
every process starts empty; the pid check keeps it so if that changes, and an Assert
on assertion-enabled builds compares `MyProcPid` with `getpid()` (PSQLE-178).
`tap/48_iv_uniqueness.t` reads every IV off the raw pages and checks that none repeats
under one DEK generation.

**Limit per key.** With random 96-bit IVs, NIST SP 800-38D allows at most 2^32
encryptions under one key — here one DEK generation: every tuple written, every row
rewritten, every TOAST chunk. `rotate_online()` starts a new generation; the README
("Routine administration") says how to estimate where a table stands.

### Benchmark

Run the included benchmark against a live container:

```bash
bash bench_tde.sh 100000
```

The script runs INSERT, SELECT, UPDATE, index scan, and TABLESAMPLE workloads
on `plain_heap` vs `encrypted_heap`, and prints a comparison table with
overhead percentages. Use `pg_vault_tde.enabled = off` (requires a server
restart — the GUC is `PGC_POSTMASTER`) to isolate pure TAM overhead (no
crypto) from actual encryption cost. See the warning in README.md before
toggling this on any database with existing `encrypted_heap` data.

```
┌────────────────────────────────────────────────────┐
│  Shared memory                                     │
│  ──────────────────────────────────────────────────│
│  LWLock (embedded by value)                        │
│  HTAB (TdeRelDekMap)                               │
│    ├─ relid: Oid         (key)                     │
│    ├─ dek[32]: char      (current DEK)             │
│    ├─ prev_dek[32]: char (rotation window)         │
│    ├─ generation: uint64                           │
│    └─ dek_valid / prev_dek_valid: bool             │
└────────────────────────────────────────────────────┘
               ▲ pg_vault_tde_kms_get_rel_dek(relid)
               │  fast path:  LW_SHARED cache hit
               │  slow path:  catalog read → KMS unwrap → cache insert
┌────────────────────────────────────────────────────┐
│  pg_vault_tde_catalog  (on-disk system table)      │
│  ──────────────────────────────────────────────────│
│  relid, generation,                                │
│  wrapped_dek, kms_provider, created_at, updated_at │
└────────────────────────────────────────────────────┘
```

1. Every encrypt/decrypt call fetches the DEK with
   `pg_vault_tde_kms_get_rel_dek()`: `hash_search(HASH_FIND)` under `LW_SHARED`,
   `memcpy` into a stack buffer, release. The buffer is `OPENSSL_cleanse`d after
   use (caller responsibility).
2. On a cache miss (first access after startup, or after the entry was evicted
   by rotation), the slow path reads `pg_vault_tde_catalog.wrapped_dek`, unwraps
   it via the active KMS provider, and inserts the entry under `LW_EXCLUSIVE`.
3. Cross-call key-schedule reuse lives in the **crypto layer**, not here: the
   `TdeCipherSlot` EVP contexts cache the installed AES schedule keyed by
   `(relid, generation)` (see [Per-Backend EVP Contexts](#per-backend-evp-contexts-keyed-by-relid-generation)),
   so re-fetching the DEK bytes per call is cheap and the expensive schedule
   install is amortised.

### Generation-Epoch Rotation

Key rotation is **per-relation** via `pg_vault_tde_rotate_online(relname, batch_size)`.
The rotation worker runs one transaction:
1. Takes `ShareRowExclusiveLock` on the heap — writers and a second rotation wait,
   `SELECT` continues — and only then its snapshot, so rows committed by the writers it
   waited for are re-encrypted too.
2. `pg_vault_tde_catalog_zero_rel_dek()` moves the current DEK to `prev_dek`, wipes
   `dek[32]`, sets `dek_valid = false` and `rotating = true`, and keeps the outgoing
   key in the worker's own memory. `generation` stays the outgoing key's.
3. `pg_vault_tde_catalog_update_rel_dek()` writes DEK N+1 to the catalog row and hands
   it to the worker's memory, never to the shared cache.
4. `pg_vault_tde_reencrypt_table()` rewrites every row with DEK N+1. Out-of-line values
   are fetched back from the TOAST relation first (still compressed), so the toaster
   stores them again under DEK N+1 and deletes the old chunks — reused as they were,
   they kept DEK N, which the catalog no longer holds (PSQLE-189).
5. A transaction callback moves the cache entry to DEK N+1 at commit (before the locks
   are released) or back to DEK N at abort, and clears `rotating`.

While `rotating` is set nobody installs a current key: every other session encrypts and
decrypts with the outgoing DEK, which is what its snapshot of the catalog shows. Every
ciphertext carries its generation, so decryption asks for the key of that generation
(`pg_vault_tde_kms_get_rel_dek_for_gen()`), and encryption reads DEK and generation in
one call (`pg_vault_tde_kms_get_rel_dek_gen()`).

- **Bounded staleness**: At most one LWLock pair per encrypt/decrypt call.
- **No signals**: Generation mismatch is detected lazily; no SIGUSR1/SIGHUP needed.
- **Fork safety**: fork() after `shmem_startup_hook` is safe because the shmem
  segment is mapped by all backends independently.

---

## Index Access Method (IAM)

### tde_btree

The `tde_btree` access method provides a B-Tree index with deterministic
(equality-preserving) key encryption using **AES-256-SIV**
(Synthetic IV — RFC 5297).

| Property | Value |
|---|---|
| Algorithm | AES-256-SIV (deterministic authenticated encryption) |
| Key length | 64 bytes (two 32-byte AES keys) |
| Equality | Preserved (same plaintext → same ciphertext under same DEK) |
| Ordering | **Not preserved** — the planner never uses `tde_btree` for ranges, `ORDER BY`, `min`/`max` or merge joins (sequential scan instead); a forced range is an error |
| Use case | Equality predicates only (`=`, `IN`, `= ANY`, `ON CONFLICT`) |
| Column support | Varlena `bytea`/`text` (`tde_*_ops`; since 1.7.2 no index creation path accepts `numeric` or a nondeterministic collation) and fixed-size `int4`/`int8`/`uuid`/`date`/`timestamptz` (`tde_*_enc_ops`, default since v1.7). All index keys are AES-256-SIV encrypted. |

AES-SIV is chosen over AES-GCM for index entries because:
- It produces a deterministic ciphertext (required for B-Tree comparisons).
- It provides authentication (misuse-resistant — no IV to manage).
- It prevents key reuse attacks that would be possible with AES-ECB.

### Implementation

The implementation uses the OpenSSL 3.x **provider API**:

```c
EVP_CIPHER *siv_cipher = EVP_CIPHER_fetch(NULL, "AES-256-SIV", NULL);
```

The DEK (32 bytes) is expanded to 64 bytes for AES-SIV's double-key
requirement via PBKDF2-SHA256:

```c
PKCS5_PBKDF2_HMAC(dek, TDE_DEK_LEN,
                  (unsigned char *)"tde-siv", 7,
                  1,        /* 1 iteration — determinism, not stretching */
                  EVP_sha256(), 64, siv_key);
```

### ambuild (sorted bulk-load)

The `ambuild` callback uses btree's internal sort layer
(`_bt_spoolinit` / `_bt_spool` / `_bt_leafbuild`) via forward-declared
prototypes in `pg_vault_tde_iam.c`. These symbols are available at
runtime from the postgres binary on all ELF platforms, even though
they are not declared in the installed extension dev headers.

For each live heap tuple, `tde_build_callback()` encrypts each non-null
indexed column datum via `tde_iam_encrypt_index_datum()` (which dispatches to
`tde_iam_encrypt_fixed_type_datum()` for fixed-size types and the varlena path
otherwise, both AES-256-SIV), then spools it into the btree sort buffer. After
the heap scan, `_bt_leafbuild()` writes all encrypted entries to the index pages
in sorted order.

### amrescan (query-time key encryption)

`pg_vault_tde_amrescan()` encrypts equality scan keys
(`sk_strategy == BTEqualStrategyNumber`) with AES-SIV before passing them
to the underlying btree scan. A range key (`sk_strategy != 3`) is an error:
AES-SIV does not preserve ordering, so walking one against the tree would
return wrong rows. The planner never builds such a scan on its own — a
`get_relation_info_hook` removes the index's sort order and
`pg_vault_tde_amcostestimate()` prices non-equality paths out — so only a
forced plan reaches that error. `amsearcharray` is off: the executor expands
`IN (…)` / `= ANY (…)` into one scalar lookup per element.

### Operator Class

```sql
-- Registered automatically by CREATE EXTENSION pg_vault_tde
CREATE OPERATOR CLASS tde_bytea_ops DEFAULT FOR TYPE bytea USING tde_btree AS
    OPERATOR 1 <  (bytea, bytea),
    OPERATOR 2 <= (bytea, bytea),
    OPERATOR 3 =  (bytea, bytea),
    OPERATOR 4 >= (bytea, bytea),
    OPERATOR 5 >  (bytea, bytea),
    FUNCTION 1 byteacmp(bytea, bytea);
```

`CREATE EXTENSION` registers two families of operator classes:

- **Varlena classes** — `tde_bytea_ops`, `tde_text_ops`, `tde_numeric_ops` (DEFAULT for
  their types). The varlena datum is encrypted with AES-256-SIV and stored as `bytea`.
- **Fixed-size `enc_ops` classes** (v1.7) — `tde_int4_enc_ops`, `tde_int8_enc_ops`,
  `tde_uuid_enc_ops`, `tde_date_enc_ops`, `tde_timestamptz_enc_ops`, all in the
  `tde_enc_ops_family` with `STORAGE bytea` and **DEFAULT** for their types. They expose
  only `OPERATOR 3 (=)` — equality is the only meaningful predicate on SIV ciphertext.
  The legacy classes (`tde_int4_ops`, `tde_int8_ops`, `tde_uuid_ops`, `tde_date_ops`,
  `tde_timestamptz_ops`) store their keys **in plaintext** and are retained only for
  indexes already built on them: since 1.7.2 a new index cannot use them unless
  `pg_vault_tde.allow_plaintext_index = on`, and 1.8 removes them.

### Index-Only Scans

Index-only scans are **not supported** on `tde_btree` indexes by design. PostgreSQL
index-only scans return column values directly from the index pages without visiting
the heap. Since `tde_btree` stores AES-256-SIV ciphertexts as index keys, returning
those values directly would expose raw ciphertext to the client with no decryption.

All decryption happens in the TAM layer (`decode_slot`) when the heap tuple is
fetched. The planner is prevented from choosing an index-only scan path on
`tde_btree` indexes; it always fetches the tuple from the `encrypted_heap` table.

Range scans, `ORDER BY` and `min()`/`max()` are never served from a `tde_btree`
index — AES-256-SIV does not preserve ordering regardless of column type — and run
as sequential scans instead.

### Usage Example

```sql
-- Create an encrypted table
CREATE TABLE employees (
    id        int4,
    username  text,
    salary    numeric
) USING encrypted_heap;

-- Create tde_btree indexes on multiple column types. The opclass is optional:
-- the encrypted-key classes are the DEFAULT for each type since v1.7, so
-- `USING tde_btree (id)` picks tde_int4_enc_ops automatically.
CREATE INDEX employees_id_idx       ON employees USING tde_btree (id tde_int4_enc_ops);
CREATE INDEX employees_username_idx ON employees USING tde_btree (username tde_text_ops);

-- Equality lookups use the encrypted index
INSERT INTO employees VALUES (1, 'alice', 90000);
INSERT INTO employees VALUES (2, 'bob',   85000);

SELECT salary FROM employees WHERE id = 1;       -- uses index
SELECT id     FROM employees WHERE username = 'alice';  -- uses index

-- Range predicates run as a sequential scan: tde_btree answers equality only
SELECT * FROM employees WHERE id > 1;            -- seq scan, not index scan

-- Index-only scans are not supported and never chosen by the planner;
-- the heap tuple is always fetched to decrypt column values.
```

---

## Logical Decoding and Replication

`pg_vault_tde` ships a logical decoding output plugin (`pg_vault_tde_pgoutput`)
so that `encrypted_heap` tables can be published to logical replication
subscribers **in plaintext**, even though their on-disk tuples — and their WAL —
are ciphertext.

### Why a plugin is needed

The TAM decrypt-on-read callbacks run in the query executor, not in the WAL
sender. A logical decoder reads raw WAL records whose tuple bodies are
ciphertext, so without intervention a subscriber receives encrypted garbage.

### Non-TOAST tables — pgoutput wrapper

`_PG_output_plugin_init` loads the built-in `pgoutput` via
`load_external_function()`, lets it populate every callback, then overrides the
change callbacks with thin wrappers that decrypt the tuple **in place** (via
`tde_decrypt_heap_tuple`) before delegating back to `pgoutput` for the actual
serialization. Because the emitted wire format is exactly the `pgoutput`
protocol, this works with `pg_recvlogical` and with a native `CREATE
SUBSCRIPTION` pointed at a slot created with this plugin. (Same wrapping
technique as Citus's CDC decoder.)

### TOAST columns — custom WAL resource manager

Externally-TOASTed columns need more than in-place decryption: the core reorder
buffer reassembles a TOAST value by `heap_deform_tuple()`-ing the chunks and the
main tuple **before any output-plugin callback runs**, and on an encrypted tuple
that crashes (`got sequence entry … for toast chunk`). There is no extension
hook earlier than that point.

The lever that does exist is a **custom WAL resource manager**, gated by the GUC
`pg_vault_tde.toast_custom_rmgr` (PGC_POSTMASTER, default **off**; requires
`pg_vault_tde` in `shared_preload_libraries`). When enabled:

1. **Write path** — `tde_toast_wal_insert()` (a faithful clone of `heap_insert`)
   logs encrypted TOAST chunks under `TDE_RMGR_ID` (161, registered for
   pg_vault_tde on the PostgreSQL *Custom WAL Resource Managers* wiki) instead of
   `RM_HEAP_ID`. The
   WAL record is byte-identical to heap's except for the resource manager id, so
   crash recovery is unaffected (`rm_redo` delegates to `heap_redo`).
2. **Decode** — routing the chunks to our `rm_decode` keeps them out of the
   reorder buffer's `toast_hash`, so the core never deforms the still-encrypted
   main tuple. `rm_decode` captures the raw encrypted chunks per transaction
   (no catalog access during decode).
3. **Stitch** — `tde_toast_stitch()`, called from the plugin's change callback
   after the main tuple has been decrypted, decrypts the captured chunks,
   reconstructs the plaintext value, and rewrites the external on-disk TOAST
   pointers into in-memory indirect pointers — a faithful analogue of core's
   `ReorderBufferToastReplace()`. `pgoutput` then serializes the full plaintext.

### Server configuration

PostgreSQL 17.11 / 18.x — and the matching minors of the older back branches —
only load a library as a logical decoding output plugin if it is listed in the
`output_plugin_libraries` GUC (default `pgoutput, test_decoding`). On those
versions the publisher must be told to accept this plugin, otherwise slot
creation fails with:

```
ERROR:  library "pg_vault_tde" may not be used as an output plugin
HINT:   ... add it to "output_plugin_libraries" and reload the server configuration.
```

```conf
# postgresql.conf on the publisher (PGC_SUSET — a reload is enough)
output_plugin_libraries = 'pgoutput, pg_vault_tde'
```

It must be set in the server configuration, not in a session: the process that
loads the plugin is the walsender, not the client backend. Earlier minors have
no such GUC — and an unrecognised parameter in `postgresql.conf` is fatal at
startup — so add the line only where `pg_settings` reports it.

### Requirements and supported operations

| Operation | Requirement |
|-----------|-------------|
| INSERT (inline, TOAST, bursts) | `toast_custom_rmgr = on` for TOAST columns |
| Initial table sync (COPY) | works via the TAM read path (decrypt-on-read) |
| UPDATE / DELETE | **`REPLICA IDENTITY FULL` + a primary key** |

`REPLICA IDENTITY FULL` is mandatory for UPDATE/DELETE: with `DEFAULT` the core
derives the replica identity by reading the **encrypted** old tuple as if it
were the key, producing a constant garbage key — the subscriber then silently
targets the wrong row. Tables without a primary key are likewise unsupported for
UPDATE/DELETE (no key to match on). These are documented limitations, not bugs:
they follow from the tuple being an opaque ciphertext blob to the core.

### Structural limitations

- **`heap_insert` clone maintenance** — `tde_toast_wal_insert()` mirrors
  `heap_insert()` and must be re-synced on each major PostgreSQL release; it is
  version-audited against the upstream function (see the comment in
  `src/logical/pg_vault_tde_rmgr.c`).
- **Reorder-buffer coupling** — the stitch path mirrors internal contracts of
  `ReorderBufferToastReplace` (buffer copy-back, memory context) that are not a
  stable public API.
- **Slots behind a key rotation** — a rotation is decoded as one UPDATE per row,
  and WAL written under the previous DEK generation decodes only while that key is
  still in shared memory (the catalog keeps the current one only).  A slot that has
  not decoded it when the publisher restarts, or when the same table is rotated
  again, fails with `pg_vault_tde: decryption failed` at the same LSN on every
  retry — every slot of the database, since the plugin decrypts before pgoutput
  filters by publication.  Let slots confirm past a rotation first; persisting
  previous keys is the 1.8 key ring.
- **Aborted-transaction capture** — a TOAST-writing transaction that reaches a
  full snapshot and then aborts *without being streamed* leaves its captured
  chunks in memory until the decoding process exits (there is no output-plugin
  hook for non-streamed aborts; it is a slow, per-abort leak, not per-row).

This ciphertext-as-opaque-blob conflict — every place the core reads a single
column (e.g. replica identity) sees ciphertext — is the motivation for the
column-level encryption alternative on the v1.8 roadmap.

---

## PKCS#11 / HSM Provider

The `pkcs11` KMS provider (v1.7, `src/kms/pg_vault_tde_kms_pkcs11.c`) keeps
the KEK inside a hardware security module. It talks the Cryptoki API
directly: the vendor's PKCS#11 module (`pg_vault_tde.pkcs11_library`) is
`dlopen()`ed at runtime and every DEK is wrapped/unwrapped with
`C_WrapKey`/`C_UnwrapKey` using `CKM_AES_KEY_WRAP` (RFC 3394, the same
algorithm the local wallet provider uses in software), with a runtime
fallback to `CKM_AES_KEY_WRAP_PAD`. No OpenSSL involvement and no
build/runtime dependency: the OASIS interface headers are vendored under
`src/include/pkcs11/` (include them only through
`src/include/pg_vault_tde_cryptoki.h`).

### Key Hierarchy and Threat Model

```
HSM token (user PIN via env var)
  └── KEK: AES-256, CKO_SECRET_KEY, CKA_SENSITIVE, CKA_EXTRACTABLE=FALSE
        └── C_WrapKey (CKM_AES_KEY_WRAP) → per-table DEK  (40-byte blob
              │                             in pg_vault_tde_catalog)
              └── encrypts tuple data (AES-256-GCM, in-process)
```

Only the **KEK** is confined to the HSM: tuple crypto runs in-process, so
the plaintext DEK necessarily transits backend memory (stack buffers,
`OPENSSL_cleanse`d after use) — the same model as the Vault Transit
provider. An attacker with the disk (or a catalog dump) holds only
DEKs wrapped by a key that exists exclusively inside the device.

### Setup

1. `pg_vault_tde.kms_provider = 'pkcs11'`, `pkcs11_library`, and
   `pkcs11_token_label` (preferred; `pkcs11_slot_id` is the fallback —
   slot IDs are not stable across restarts on some modules).
2. Export the token user PIN in the environment variable named by
   `pkcs11_pin_env` (default `PG_TDE_PKCS11_PIN`) before starting
   PostgreSQL. The GUC holds the env var *name* — never put the PIN in
   `postgresql.conf`.
3. `SELECT pg_vault_tde_pkcs11_keygen();` (superuser, once) generates the
   AES-256 KEK on the token under `pkcs11_key_label`. It refuses to
   overwrite an existing key. Alternatively provision the key with the HSM
   tooling (`CKA_WRAP`, `CKA_UNWRAP`, `CKA_EXTRACTABLE=FALSE`).

Per-database HSM isolation works like every other provider: all `pkcs11_*`
GUCs are `PGC_SUSET`, so different databases can use different tokens or
key labels via `ALTER DATABASE ... SET`.

### Process Model and Fork Safety

PKCS#11 (§6.6 of the spec) makes Cryptoki state unusable across `fork()`.
Because every PostgreSQL backend is forked from the postmaster:

- `C_Initialize` is **never** called in the postmaster — `init()` there
  only validates the GUCs;
- each backend attaches lazily on first use (dlopen → `C_Initialize` →
  slot discovery → `C_OpenSession` → `C_Login` → KEK lookup), and a
  `getpid()` guard discards any state inherited across fork without
  calling into the module;
- on session/device loss (`CKR_SESSION_HANDLE_INVALID`,
  `CKR_DEVICE_ERROR`, ...) operations retry exactly once through a fresh
  session;
- the vendor module is never `dlclose()`d (many modules crash on unload).

### KEK Rotation

Each KEK generation lives forever under its own immutable token label
`<label>.v<N>` (N monotonically increasing) — rotation never renames or
destroys a key. "Current" is simply the highest `N` found on the token, and
every wrapped DEK stored in the catalog carries a 4-byte version tag
identifying exactly which `<label>.v<N>` produced it. Unwrap always looks
up that exact version, regardless of which one is "current" at the time.

`SELECT pg_vault_tde_rotate_kek();` drives:

1. **prepare** — generates a fresh KEK as `<label>.v<current+1>`;
2. **rewrap** — every catalog DEK is unwrapped with the KEK version tagged
   in its own blob and re-wrapped with the new version, tagging the new
   blob accordingly (transactional catalog UPDATEs);
3. **commit** — purely an in-backend cache update (the new version is
   already durably on the token and every rewrapped row already carries its
   own version tag), so there is nothing left to make durable and no
   crash window: whatever the catalog transaction ends up committing is
   self-describing and always resolves to the right KEK.

Because no KEK generation is ever renamed or destroyed, a crash or a rolled
back rotation at any point simply leaves an unused `<label>.v<N+1>` key on
the token (harmless — the next rotation attempt reuses or supersedes it)
with the catalog untouched, still tagged with the old version and still
fully readable.

**Cross-backend propagation.** `commit_kek_rotation` only updates the
cache of the ONE backend that ran `pg_vault_tde_rotate_kek()`. Without more,
every OTHER already-connected backend would keep wrapping new DEKs under
the pre-rotation KEK indefinitely. A small shared-memory beacon (one
`LWLock` + a `uint32` KEK version, same dynamic-tranche pattern as the
Vault token cache) fixes this: `commit_kek_rotation` (and the initial
keygen) publish the new version there; every backend checks it
opportunistically on its own next wrap/unwrap/rewrap call and, if stale,
resolves its own `CK_OBJECT_HANDLE` locally via a label lookup. Only the
version NUMBER crosses the process boundary — never the object handle
itself, which PKCS#11 only guarantees meaningful within the session that
resolved it. Staleness for an already-attached backend is therefore bounded
by "its own next operation", not by wall-clock time or a reconnect.

### Testing with SoftHSM2

`tap/16_pkcs11.t` is fully self-contained: it provisions a throwaway
SoftHSM2 token in a tempdir (`SOFTHSM2_CONF` + `softhsm2-util
--init-token`, no root needed) and exercises keygen, round-trip, on-disk
ciphertext, restart, health check, KEK rotation, cross-backend rotation
propagation (a long-lived session picking up a rotation committed by a
different connection, via `background_psql`), and clean failure paths
(wrong token label, wrong PIN, unprovisioned key label, invalid
`pkcs11_library` path) — 19 assertions in total.
Run it via `make ci-pkcs11` (containerized) or `prove tap/16_pkcs11.t`
where the `softhsm2` package is installed; it skips itself otherwise.
`pkcs11-tool` (package `opensc`) is handy for inspecting the token:
`pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --list-objects`.

### Limitations

- The standalone backup tools (`pg_dump_tde`/`pg_restore_tde`) do not
  support `kms_provider = 'pkcs11'` yet; they exit with a clear error.
- `health_check()` may exceed its usual latency budget on first touch of
  a *network* HSM (the lazy attach performs the full login sequence).
- PKCS#11 labels are not unique: keep exactly one KEK under
  `pkcs11_key_label` (the provider warns and picks the first match).

---

## Known Limitations

### Current Limitations (v1.7 — Current Release)

| # | Limitation | Fix Version |
|---|-----------|-------------|
| 1 | **TOAST chunk-level storage encryption** — ✅ **Resolved in v1.6**: large values round-trip fully encrypted via `pg_vault_tde_toast_am` returning `encrypted_heap` AM. `pg_vault_tde.toast_encryption` has no effect since 1.7.2 (PSQLE-223). | v1.6 ✅ |
| 2 | **tde_btree fixed-size types plaintext index keys** — ✅ **Resolved in v1.7**: `int4`, `int8`, `uuid`, `date`, `timestamptz` btree index keys are now encrypted with AES-256-SIV, matching varlena type behaviour. | v1.7 ✅ |
| 3 | **Logical replication of TOAST columns** — ✅ **Resolved in v1.7** via the custom WAL resource manager (enable `pg_vault_tde.toast_custom_rmgr`). UPDATE/DELETE require `REPLICA IDENTITY FULL` + a primary key; `REPLICA IDENTITY DEFAULT` and PK-less tables remain unsupported. See [Logical Decoding and Replication](#logical-decoding-and-replication). | v1.7 ✅ |
| 4 | **WAL unencrypted** — requires `XLogInsert()` hook unavailable in extension API | Permanently deferred |
| 5 | **All-or-nothing table encryption** — no per-column granularity | v1.8 |
| 6 | **tde_btree answers equality only** — ranges, `ORDER BY`, `min`/`max` run as sequential scans; `numeric` and nondeterministic collations refused (AES-SIV not order-preserving) | By design, permanent |
| 7 | **BRIN on encrypted columns** — min/max of AES-SIV ciphertexts is meaningless | By design, permanent |
| 8 | **HOT updates disabled** — `heap_update` decides HOT by comparing the indexed columns' on-disk bytes between old and new tuple. A changed indexed value could re-encrypt to the same bytes (once in 256^L for L bytes), so `pg_vault_tde_tuple_update()` re-encrypts a changed indexed column under a fresh IV (PSQLE-219); `heap_update` then always sees it as modified and skips HOT, keeping the index coherent. See the section below. | By design, permanent |
| 9 | **`WITH HOLD` cursor plaintext temp file** — a held cursor's result set is materialized into a tuplestore at `COMMIT` and spills to a plain temp file on disk past `work_mem`, bypassing the TAM entirely; no extension hook exists anywhere in the `WITH HOLD` cursor lifecycle to intercept it. See README.md § Limitations item 6. | Permanently deferred |
| 10 | **Plain `COPY <table> TO` / `pg_dump` produce a plaintext dump, with no warning** — encryption lives entirely in the TAM's read callbacks (`scan_getnextslot` and friends), which decrypt unconditionally and cannot distinguish a `COPY TO` from a `SELECT`; `pg_dump`'s default table-data path is exactly this form of `COPY`. No `ProcessUtility_hook` guard or GUC-gated `WARNING` exists yet (designed, never implemented). Use `pg_dump_tde`/`pg_restore_tde` instead. See README.md § Limitations item 10. | v1.8 |
| 11 | **Three index paths store plaintext keys past the `allow_plaintext_index` guard, with no check and no warning — not supported in 1.7.2**: an `EXCLUDE` constraint on a native access method (it is a constraint, so it skips the `CREATE INDEX` guard); a native index cloned onto an encrypted partition (`PARTITION OF`/`ATTACH PARTITION`, created internally with `is_internal`); a native index carried over by `ALTER TABLE … SET ACCESS METHOD encrypted_heap`. Do not use them on encrypted tables; index with `tde_btree`, and create native indexes before converting a table or attaching a partition. `INCLUDE` columns on a `tde_btree` index had the same effect and are rejected outright since v1.7.2. See README.md § Limitations item 11. | v1.8 |

### HOT updates are disabled by design

On an `encrypted_heap` table, `heap_update` never chooses a HOT (heap-only tuple)
update: every UPDATE writes new index entries. This is **intentional** and is what
keeps `tde_btree` indexes coherent across UPDATEs of indexed columns — the index always
follows the row to its new key, with no `REINDEX` needed.

**Mechanism.** `heap_update` decides whether an update can be HOT by comparing the
indexed columns between the old and the new tuple image, on disk. On an
`encrypted_heap` table both images are encrypted, and every version of a row is
encrypted under a fresh random GCM IV, so an attribute is byte-stable across an
update only by chance — once in 256^L for a value of L bytes. When the chance hits a
value that changed, `pg_vault_tde_tuple_update()` encrypts the row again under
another IV (PSQLE-219), so `heap_update` sees every changed indexed column as modified
and skips the HOT path. The constant `[VERSION | GENERATION]` bytes sit at the **end** of the
region, outside every attribute, so they cannot create a byte-stable window.

> **Historical note (v3 bug, fixed in v4).** The v3 format placed a constant
> `[VERSION(1)=0x03 | GENERATION(8)]` prefix *first*. An indexed column whose datum landed
> inside that 9-byte prefix — typically a leading fixed-width `int4`/`int8` key — looked
> *unchanged* to `heap_update`, which then chose a HOT update and silently skipped the
> index maintenance, leaving the index pointing at the old key. Moving the constant bytes
> to the trailer removed the byte-stable region; the workarounds v3 required (`REINDEX`,
> or arranging the indexed column past the first 9 bytes) are no longer needed.

> **Historical note (v4 bug, fixed in v5 — PSQLE-165).** v4 made the comparison
> read a region that was not laid out as a tuple at all, so past the first
> variable-length column `heap_update` walked ciphertext looking for attribute
> boundaries. Usually that left the page and the backend died; when it stayed on
> the page it compared garbage, and an updated indexed column could come out
> *unchanged* — the v3 failure mode again, by a different route. v5 keeps the
> attribute layout intact, so the comparison reads real per-attribute ciphertext.
>
> **Short values, fixed in 1.7.2 (PSQLE-219).** Real per-attribute ciphertext is short
> when the value is: a `bool` or a one-character text repeated its old ciphertext on one
> `UPDATE` in 256, and when its value had changed that `UPDATE` went HOT — the index kept
> the old key, lookups missed the row and UNIQUE let a duplicate in. v4 had it too for
> short fixed-length columns, read at a fixed offset inside its random blob. Hence the
> re-encryption above; `tap/49_hot_update_short_indexed.t` covers it.

### Historical Limitations (v1.0) — Many Resolved Since

1. **TOAST encryption** (ticket #1) — ✅ **Resolved in 6**  
   `pg_vault_tde_toast_am` now returns `encrypted_heap` AM when
   `pg_vault_tde.toast_encryption = on` (default). Every TOAST chunk is
   encrypted individually using the parent relation's DEK.  `toast_encryption = off`
   no longer restores the v1.0 behaviour: it has no effect since 1.7.2 (PSQLE-223).

2. **Row re-encryption after rotation** (ticket #2) — ✅ **Resolved**  
   `pg_vault_tde_rotate_online(relname, batch_size)` promotes the current DEK to
   `prev_dek` and moves the relation to the next generation; old-generation rows stay
   readable via `prev_dek` (see Generation-Epoch Rotation). `pg_vault_tde_reencrypt_table(regclass
   [, batch_size])` (implemented in `src/tam/pg_vault_tde_tam.c`) then rewrites every
   row to the new generation in batches, closing the window. A rotation background
   worker (`src/kms/pg_vault_tde_rotation_bgw.c`) can drive this automatically.

3. **Vault HTTP connector** (ticket #3) — ✅ **Resolved**  
   The libcurl-based Vault/OpenBao Transit integration is fully implemented in
   `src/kms/pg_vault_tde_kms.c` (`vault_provider_wrap_dek()` /
   `vault_provider_unwrap_dek()` / `vault_provider_rewrap_dek()`, synchronous via
   `vault_transit_request()`). Supports token, AppRole, and Kubernetes JWT auth.

4. **multi_insert / COPY throughput** (ticket #4)
   The `multi_insert` callback encrypts per-slot and calls `heap_insert`
   in a loop, forfeiting WAL-batching optimizations. This degrades COPY
   workloads by roughly the single-insert overhead multiplied by batch size.
   Batch-encrypted `heap_multi_insert` is targeted for v1.1.

5. **Logical replication** (ticket #5) — ✅ **Resolved** (non-TOAST in v1.2,
   TOAST columns in v1.7)  
   The `pg_vault_tde_pgoutput` plugin decrypts tuples before publishing to
   subscribers; the custom WAL resource manager
   (`pg_vault_tde.toast_custom_rmgr`) extends this to externally-TOASTed
   columns. See [Logical Decoding and Replication](#logical-decoding-and-replication).

6. **IAM range scans** (ticket #6)  
   `WHERE col > 'x'` on a column with a `tde_btree` index always returns
   empty: AES-SIV does not preserve ordering. Users requiring range
   predicates on encrypted columns must use sequential scans.

---

## Security Considerations

### Threat Model

pg_vault_tde encrypts **data at rest** (relation files, TOAST files, backup
media, standby base backups). It does **not** protect:

- In-memory tuple data during query execution.
- WAL structural metadata (tuple payloads in WAL are ciphertext; LSN,
  block numbers, and relation OIDs are plaintext).
- `pg_statistic` rows written by ANALYZE (see below).
- Network connections between backends and clients (use `ssl = on` in
  `pg_hba.conf`).

### Planner Statistics

`pg_statistic` is populated by ANALYZE after decryption through the TAM
read path. Statistics are stored **as plaintext** in the system catalog.
Protect via:
```sql
REVOKE SELECT ON TABLE pg_statistic FROM PUBLIC;
```

### WAL

Tuple payload bytes in WAL are the encrypted bytes written to disk —
a WAL stream viewer sees ciphertext in DATA positions. Structural
metadata (LSN, block numbers, relation OID, MVCC fields) is plaintext.

### What the authentication tag does not cover

The GCM tag of a tuple authenticates the bytes of its attribute values,
concatenated in attribute order, and an AAD of `[MyDatabaseId | relid | generation]`.
A ciphertext therefore does not verify in another database, another relation, or
under another DEK generation: moved there, it is refused. Damage to the value bytes
is refused too. The tag does not bind:

- **the tuple's position** — its block and line pointer. heapam chooses where a
  tuple goes after it has been formed, ciphertext included, so the position cannot
  be part of the AAD. Within one relation and one generation, a tuple image written
  back to another place, or an older image of the same table, still verifies.
- **the tuple header** — `xmin`, `xmax`, the infomask. Core rewrites them (hint
  bits, `xmax`, freezing) without the key. Damage there can make a row version
  visible or invisible (`tap/20_ondisk_fuzz.t` counts that outcome apart).
- **the layout of the values** — the null bitmap and the length headers of
  variable-length attributes, which v5 keeps in clear (see
  [Wire Format per Encrypted Region](#wire-format-per-encrypted-region)). They say
  where one value ends and the next begins, and they are not part of the tag: whoever
  can write the data files can change how a row's bytes are divided among its
  variable-length attributes without failing it. No plaintext byte can be changed or
  added that way. Authenticating the layout is a change of tuple format, planned
  for 1.8 (PSQLE-218).

Detecting a replayed or moved tuple needs integrity over pages or relations, which
an extension cannot add; data checksums detect accidental damage only. All three are
outside the threat model: the attacker it defends against reads files, and does not
write to the data directory (see `doc/SECURITY-REVIEW.md`).

### Superuser Bypass

A PostgreSQL superuser executing SQL sees plaintext (decrypted through
the TAM layer). A superuser with OS-level file access sees encrypted content.
Row-level security and column-level privileges complement TDE for
access control but do not replace it.

### Key Material Lifecycle

| Event | Action |
|---|---|
| Backend start | Local DEK copy in `TopMemoryContext` |
| Query end | DEK remains in `TopMemoryContext` (not wiped per-query) |
| `pg_vault_tde_rotate_online()` | Per-relation shmem DEK `OPENSSL_cleanse`d; generation incremented |
| Backend exit | `on_proc_exit` hook calls `OPENSSL_cleanse` on local copy |
| OS crash | Shmem lost; DEK must be re-injected from Vault on restart |

---

## Wire Format Reference

### Per-Page Layout

```
┌────────────────────────────────────────────────────────────────────┐
│  PageHeaderData (24 bytes, plain)                                  │
│  ItemId array (4 bytes per slot, plain)                            │
├────────────────────────────────────────────────────────────────────┤
│  Free space                                                        │
├────────────────────────────────────────────────────────────────────┤
│  ... tuples grow downward from end of page ...                     │
│                                                                    │
│  ┌─────────────────────────────┬─────────────────────────────────┐ │
│  │  HeapTupleHeaderData        │ attrs (values enc.) │IV│TAG│V│G│ │
│  │  (t_hoff bytes, PLAINTEXT)  │                                 │ │
│  └─────────────────────────────┴─────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
```

### Constants

| Constant | Value | Defined in |
|---|---|---|
| `TDE_GCM_IV_LEN` | 12 | `pg_vault_tde_crypto.h` |
| `TDE_GCM_TAG_LEN` | 16 | `pg_vault_tde_crypto.h` |
| `TDE_V4_VERSION_BYTE` | `0x04` | `pg_vault_tde_crypto.h` |
| `TDE_V4_GEN_LEN` | 8 | `pg_vault_tde_crypto.h` |
| `TDE_V4_AAD_LEN` | 16 | `pg_vault_tde_crypto.h` (`dboid` + `relid` + `generation`) |
| `TDE_V4_OVERHEAD` | 37 | `pg_vault_tde_crypto.h` (= 12 + 16 + 1 + 8) |
| `TDE_DEK_LEN` | 32 | `pg_vault_tde_kms.h` **only** |

`TDE_DEK_LEN` MUST NOT be redefined in any `.c` file or other header
(header hygiene rule).

---

## SQL API Reference

### Access Methods

```sql
-- Table AM: encrypts all column data per tuple
CREATE TABLE t (...) USING encrypted_heap;

-- Index AM: deterministic AES-SIV for B-Tree key equality
CREATE INDEX ON t USING tde_btree (col);
```

### GUC Parameters

All parameters are in the `pg_vault_tde` namespace and are registered in
`_PG_init` via `DefineCustomXxxVariable`.

**Context**: all KMS-related parameters are `PGC_SUSET` — superusers can set
  them at session level or scope them to individual databases with
  `ALTER DATABASE SET` / `ALTER ROLE SET`.  No server restart is needed.
  The exceptions are `max_encrypted_relations` (controls shared-memory
  sizing, `PGC_POSTMASTER`), `crypto_provider` (OpenSSL provider selection,
  `PGC_POSTMASTER`), and `enabled` (master crypto switch, `PGC_POSTMASTER` —
  its value is baked into the on-disk wire format, so it cannot be toggled
  without risking silent plaintext/ciphertext mismatches; see the `enabled`
  row below).


#### Per-Database KMS Configuration

Because all GUCs are `PGC_SUSET`, each database in the same PostgreSQL cluster
can independently select its KMS backend and credentials.  This is the primary
mechanism for multi-tenant key isolation:

```sql
-- cluster-wide default (postgresql.conf or ALTER SYSTEM)
-- pg_vault_tde.kms_provider = 'vault'

-- tenant_a: dedicated Transit key, no change to other settings
ALTER DATABASE tenant_a SET pg_vault_tde.vault_key_name     = 'tde-dek-a';
ALTER DATABASE tenant_a SET pg_vault_tde.vault_transit_mount = 'transit-tenants';

-- tenant_b: offline local wallet, completely different backend
ALTER DATABASE tenant_b SET pg_vault_tde.kms_provider          = 'local';
ALTER DATABASE tenant_b SET pg_vault_tde.wallet_passphrase_env  = 'TDE_WALLET_B';

-- verify effective configuration
\connect tenant_b
SHOW pg_vault_tde.kms_provider;   -- 'local'
SELECT * FROM pg_vault_tde_health_check();
```

Settings applied with `ALTER DATABASE SET` take effect for new connections to
that database.  The provider is selected per-connection from the effective GUC
value; no shared state is changed.

| Parameter | Type | Default | Context | Description |
|---|---|---|---|---|
| `kms_provider` | string | `''` (unset — must be configured) | suset | Active KMS backend: `vault`, `local` (v1.6). Settable per-database. |
| `vault_url` | string | `''` | suset | Vault / OpenBao base URL |
| `vault_namespace` | string | `''` | suset | Vault namespace (enterprise; empty for community) |
| `vault_auth_method` | string | `token` | suset | Vault auth method: `token`, `approle`, or `kubernetes` |
| `vault_token` | string | `''` | suset | Auth token — shown only to a superuser (others read `********`), not in `pg_settings` (v1.7.2, PSQLE-224) |
| `vault_role_id` | string | `''` | suset | AppRole role_id UUID — shown only to a superuser (others read `********`), not in `pg_settings` (v1.7.2, PSQLE-224) |
| `vault_secret_id` | string | `''` | suset | AppRole secret_id — shown only to a superuser (others read `********`), not in `pg_settings` (v1.7.2, PSQLE-224) |
| `vault_role_name` | string | `''` | suset | AppRole role name for secret_id rotation **(v1.4)** — calls `secret-id/destroy` after login |
| `vault_k8s_role` | string | `''` | suset | Kubernetes JWT auth role name |
| `vault_k8s_mount` | string | `kubernetes` | suset | Kubernetes auth engine mount path |
| `vault_transit_mount` | string | `transit` | suset | Transit secrets engine mount path |
| `vault_key_name` | string | `pg-tde-dek` | suset | Transit key name for DEK wrapping. Override per-database to isolate tenant keys. |
| `vault_ca_cert` | string | `''` | suset | Path to CA bundle for Vault TLS (`CURLOPT_CAINFO`) |
| `vault_timeout_ms` | integer | `5000` | suset | Vault HTTP timeout in ms (0 = no timeout; range 0–300000) |
| `wallet_path` | string | `/var/lib/pg_vault_tde/<OID>/wallet.p12` | suset | Local wallet PKCS#12 path (`kms_provider = 'local'`). Cluster-wide value = one KEK for every database; rotating it is then destructive (see Shared Memory Layout) |
| `wallet_passphrase_env` | string | `''` | suset | Env var NAME holding the wallet passphrase |
| `wallet_passphrase_file` | string | `''` | suset | File path containing the wallet passphrase (mode 0400 enforced) |
| `wallet_passphrase_command` | string | `''` | suset | Shell command whose stdout is the passphrase (highest priority) |
| `wallet_auto_open` | boolean | `on` | suset | Auto-open wallet at startup if passphrase env var is set |
| `dev_mode` | boolean | `off` | suset | Enable development-only conveniences (insecure in production) |
| `wallet_dev_mode_passphrase` | string | `''` | suset | Inline dev passphrase, used only when `dev_mode = on`; emits a `WARNING` on every use — shown only to a superuser (others read `********`), not in `pg_settings` (v1.7.2, PSQLE-224) |
| `dek_cache_ttl` | integer | `0` | suset | Per-backend DEK cache TTL in seconds (0 = no expiry; range 0–86400). When > 0, each backend re-reads the DEK from shmem after this interval even without rotation |
| `toast_encryption` | boolean | `on` | suset | **No effect since 1.7.2** (PSQLE-223): the TOAST table of an `encrypted_heap` table is always `encrypted_heap` and its chunks always encrypted with the parent relation's DEK; setting it `off` only raises a `WARNING` when a TOAST table is created. Removed in 1.8 |
| `allow_plaintext_index` | boolean | `off` | suset | When `off` (default), `CREATE INDEX`/`CREATE UNIQUE INDEX` with a non-`tde_btree` access method on an `encrypted_heap` table is rejected with `ERROR`. When `on`, the same statement is allowed after a `WARNING` — the indexed column's plaintext value is then stored unencrypted on disk. Does not affect `PRIMARY KEY`/`UNIQUE` table constraints, which PostgreSQL core always backs with a native btree index regardless of this setting (that case already warns unconditionally). |
| `bgw_enabled` | boolean | `off` | suset | Enable background worker for automatic token renewal. **Requires cluster restart**: the BGW is registered via `RegisterBackgroundWorker()` at postmaster startup; changing via `pg_reload_conf()` updates the value but does not start/stop the worker dynamically. |
| `token_renewal_interval` | integer | `3600` | suset | Token renewal interval in seconds (60–86400) |
| `enabled` | boolean | `on` | postmaster | Master switch: `off` disables crypto for benchmarking overhead. Fixed at server startup |
| `max_encrypted_relations` | integer | `1024` | postmaster | Max per-table DEK entries in shmem (64–65536), **cluster-wide** — entries are keyed by `(dbid, relid)`, so budget for the sum across all databases. Enforced since 1.7.2 — before that the cache silently grew past it (ShmemInitHash's size is not a cap), so count your encrypted relations across all databases before upgrading. ~112 bytes per relation, reserved at startup. Max 1048576. **Requires restart**. |
| `crypto_provider` | string | `''` | postmaster | OpenSSL 3.x provider name (`qatprovider`, `fips`; empty = built-in dispatch). **Requires restart**. |

All variables are declared `extern` in `src/include/pg_vault_tde_guc.h`
and included by any translation unit that needs them (`tam.c`, `kms.c`).

---

## Extension Initialization

### _PG_init Sequence

```
_PG_init()
  ├── DefineCustomStringVariable("pg_vault_tde.vault_url", ...)      [PGC_SUSET]
  ├── DefineCustomStringVariable("pg_vault_tde.vault_token", ...)    [PGC_SUSET, GUC_SUPERUSER_ONLY | GUC_NOT_IN_SAMPLE | GUC_NO_SHOW_ALL, show hook]
  ├── ... ~20 more GUC parameters (all PGC_SUSET except max_encrypted_relations/crypto_provider/enabled) ...
  ├── install shmem_request_hook  → pg_vault_tde_shmem_request()
  │       ├── pg_vault_tde_kms_shmem_request()
  │       │       └── RequestAddinShmemSpace(sizeof(pg_vault_tde_kms_cache))
  │       └── pg_vault_tde_catalog_shmem_request()
  │               ├── RequestAddinShmemSpace(tde_rel_dek_cache_size(capacity))
  │               └── RequestNamedLWLockTranche("pg_vault_tde_rel_dek_map", 1)
  ├── install shmem_startup_hook  → pg_vault_tde_shmem_startup()
  │       ├── pg_vault_tde_kms_shmem_init()       (Vault-token cache)
  │       │       ├── ShmemInitStruct("pg_vault_tde_kms_cache", ..., &found)
  │       │       └── if !found: LWLockNewTrancheId() + LWLockInitialize()
  │       │                      ← dynamic tranche; requires shmem to be up!
  │       ├── pg_vault_tde_catalog_shmem_init()   (per-relation DEK cache)
  │       │       ├── ShmemInitHash("pg_vault_tde_rel_dek_map", capacity, capacity, ...)
  │       │       └── rel_dek_lock = &GetNamedLWLockTranche("pg_vault_tde_rel_dek_map")[0].lock
  │       └── tde_shmem_started = true; tde_active_kms_provider->init()
  ├── pg_vault_tde_tam_init()
  │       └── memcpy(&tde_methods, GetHeapamTableAmRoutine(), sizeof(TableAmRoutine))
  │           + install 5 write + 8 read + 2 visibility/build + 2 rewrite AM wrappers
  └── register on_proc_exit(tde_backend_cleanup)
          ├── tde_crypto_ctx_cleanup()   ← EVP_CIPHER_CTX_free + OPENSSL_cleanse iv_batch
          └── tde_iam_ctx_cleanup()      ← EVP_CIPHER_CTX_free SIV enc/dec context
```

#### Critical: Why LWLockNewTrancheId cannot be called from _PG_init

In PostgreSQL 17+, `LWLockNewTrancheId()` acquires
`WaitEventCustomCounterLock`, a spinlock stored **in shared memory**.
Shared memory does not exist at `_PG_init` time, so calling
`LWLockNewTrancheId()` from `_PG_init` causes an immediate segfault. The two
shmem structs use two different (both correct) tranche strategies:

- **`pg_vault_tde_kms_cache`** (dynamic tranche): `RequestAddinShmemSpace()`
  in `shmem_request_hook`; `LWLockNewTrancheId()` + `LWLockInitialize()` in the
  `shmem_startup_hook` `!found` branch.
- **`TdeRelDekMap`** (named tranche): `RequestAddinShmemSpace()` **and**
  `RequestNamedLWLockTranche("pg_vault_tde_rel_dek_map", 1)` in `shmem_request_hook`
  (named-tranche *requests* are allowed there — only `LWLockNewTrancheId()` is
  not), then `GetNamedLWLockTranche("pg_vault_tde_rel_dek_map")` in `shmem_startup_hook`
  (the HTAB and its lock are created by `ShmemInitHash` / picked up from the
  tranche; no `LWLockInitialize()` needed for a named-tranche lock).

This is the correct pattern for all PG17+ / PG18 extensions that need a
dynamic LWLock tranche.

---

## Testing Strategy

### Standalone Smoke Test

```bash
make install
make check-standalone
```

`make check-standalone` drives `pg_regress` directly: it initialises a
throwaway cluster under `tmp_check/` on a free port, appends
`test/regress.conf` to its configuration, runs the `pg_vault_tde_init` test and
removes the cluster. No existing installation is started or modified, no KMS
service is contacted, and nothing is edited by hand.

Two constraints explain its shape. PGXS declines `make check` for out-of-tree
extensions, so the target cannot simply defer to the standard rule; and plain
`make installcheck` fails against a stock cluster, because the extension
registers its Table Access Method and requests shared memory from `_PG_init`
and therefore has to be preloaded. `test/regress.conf` carries exactly that one
setting — deliberately not a copy of the CI configuration.

It verifies that the extension builds, installs and loads. It does **not**
exercise key management: that is the job of the suites below and of the
providers listed under [KMS Provider Coverage](#kms-provider-coverage).

### Regression Tests

`make ci-regress` (driven by `ci/scripts/run-regress.sh`) runs four SQL files in
sequence, gated on the live extension version:

| File | Tests | Scope |
|---|---|---|
| `sql/regression_test.sql` | 1–52 | v1.0–v1.4 baseline: crypto, TAM, TOAST, tde_btree |
| `sql/regression_test_v15.sql` | 53–72 | v1.5: per-table DEK, online rotation, AAD |
| `sql/regression_test_v16.sql` | 73–109 | v1.6: local wallet KMS |
| `sql/regression_test_v17.sql` | 111–140, 154–164 | v1.7: `enc_ops` indexes, partition trees, HOT/REINDEX, FK lifecycle, TidRangeScan, on-disk tuple layout |
| `sql/regression_test_errorpath.sql` | 141–153 | error paths (`make ci-errorpath`) |

Test numbers are one sequence shared by every suite, which is why 141–153 are missing
from the v1.7 file rather than being a gap. The real gaps are 5–11 and 49 (no
longer exist), 80 (removed in v1.7) and 110 (`WITH HOLD` cursor spill, disabled: a
permanent limitation), so `make ci-regress` runs **141 tests**. The table below details the v1.0–v1.4 baseline file:

| Range | Area |
|---|---|
| 1–4 | Extension, access methods and SQL functions registered; wallet unlock |
| 12 | TAM INSERT + SELECT basic round-trip |
| 13 | On-disk plaintext absence (raw file scan) |
| 14 | TAM UPDATE (ctid preservation, tuple refetch, HOT chains) |
| 15 | DELETE (no crash, row unreachable) |
| 16 | All-NULL rows (`user_len == 0` edge case) |
| 17 | Index scan (`index_fetch_tuple` + `rd_tableam` workaround) |
| 18 | COPY / bulk insert (`multi_insert` path) |
| 19 | Multi-column types (int, text, bool, numeric, timestamptz) |
| 20 | Key rotation isolation (DEK-A rows rejected after rotating to DEK-B) |
| 21 | ANALYZE (statistics computed on decrypted values) |
| 22 | SELECT FOR UPDATE (`tuple_lock` path) |
| 23 | BitmapHeapScan (`scan_bitmap_next_tuple` via forced bitmap scan) |
| 24 | TABLESAMPLE (`scan_sample_next_tuple` via SYSTEM(100)) |
| 25–48 | UPSERT, MERGE, TRUNCATE, REINDEX, ALTER, JOINs, CTEs, HW accel, Vault, logical decoding |
| 50 | tde_btree CREATE INDEX + equality index scan **(v1.4)** |
| 51 | health_check() `kms_provider` GUC coherence **(v1.4)** |
| 52 | tde_btree UNIQUE constraint **(v1.4)** |
### Page Checksum Test (`make ci-checksums`)

Starts PostgreSQL with `initdb -k` (`--data-checksums`). Verifies that:
1. Page checksums are enabled (`SHOW data_checksums = 'on'`).
2. Encrypted pages pass checksum validation (checksums cover encrypted bytes).
3. The relation file is physically non-zero (encryption actually ran).

### TAP Tests (`tap/`)

49 files, run together by `make ci-tap` (which also starts the Vault container
the Vault-dependent files need; they `skip_all` when `VAULT_ADDR` is unset).
`tap/43_soak.t` skips unless `PG_VAULT_TDE_SOAK=1`; `make ci-soak` runs it alone.

| File | Coverage |
|---|---|
| `tap/01_load.t` | Extension load, AM registration, basic SQL round-trip |
| `tap/02_backup_local.t` | `pg_dump_tde` / `pg_restore_tde` round-trip, local wallet KMS |
| `tap/03_backup_vault.t` | Same round-trip against a real Vault Transit backend |
| `tap/04_backup_format.t` | Binary dump format, multi-object dump/restore |
| `tap/05_backup_data_variety.t` | TOAST, Unicode, bytea, bulk rows, sequences |
| `tap/06_backup_corruption.t` | Corrupted ciphertext, truncation, bad magic, IV randomness |
| `tap/07_backup_cli.t` | CLI argument validation of the backup wrappers |
| `tap/08_backup_security.t` | No plaintext leakage, IV uniqueness, opaque output |
| `tap/09_backup_wrong_key.t` | Wrong passphrase and missing-wallet scenarios |
| `tap/10_backup_cross_db.t` | Restore into a database other than the source |
| `tap/11_multi_kms_cluster.t` | Different KMS providers per database in one cluster |
| `tap/12_logical_repl_toast.t` | Logical replication of encrypted_heap TOAST values |
| `tap/13_index_concurrently.t` | `CREATE INDEX CONCURRENTLY` / `REINDEX CONCURRENTLY` |
| `tap/14_seal_keys.t` | Physical-backup key sealing bundle round-trip |
| `tap/15_basebackup_tde.t` | `pg_basebackup_tde` wrapper, multi-database bundles |
| `tap/16_pkcs11.t` | pkcs11 provider against a throwaway SoftHSM2 token |
| `tap/17_index_constraints.t` | Index AM whitelist, PRIMARY KEY / UNIQUE behaviour |
| `tap/18_guc_order_independence.t` | KMS GUCs are order- and scope-independent (see below) |
| `tap/20_ondisk_fuzz.t` | Arbitrary on-disk bit flips must yield correct data or a refusal, never a forged value (checksums disabled so the GCM tag is the line under test) |
| `tap/19_crash_recovery_rmgr.t` | Encrypted TOAST chunks survive WAL replay after an unclean shutdown — the only test that executes the custom resource manager's `rm_redo` (see below) |
| `tap/21_cache_key_cross_db.t` | Two databases holding the same relid (a `CREATE DATABASE ... TEMPLATE` clone) with different DEKs must not share a shmem cache entry |
| `tap/22_rotate_cold_cache.t` | `pg_vault_tde_rotate_online()` must preserve the outgoing DEK when the shmem entry is cold (after a restart, or a wallet lock/unlock) |
| `tap/23_rotation_generation_drift.t` | An aborted rotation must not leave the shmem cache a generation ahead of the catalog (fault injection: the catalog row is removed mid-rotation) |
| `tap/24_shared_wallet_warning.t` | KEK rotation warns when the wallet is not this database's own file, and stays quiet on the per-database default |
| `tap/25_cache_full_degrades.t` | `max_encrypted_relations` is honoured, and relations past it keep reading and writing |
| `tap/26_preload_keys.t` | The startup warm-up loads a database's DEKs when it asks, skips the databases that did not, keeps their keys apart, and honours `preload_max_failures` |
| `tap/27_preload_providers.t` | The warm-up works with the KEK outside the server: Vault/OpenBao and PKCS#11 (each half skips when its backend is absent) |
| `tap/28_dml_memory_scaling.t` | Per-row memory is released on every DML path: a second session samples `pg_log_backend_memory_contexts()` mid-statement and compares `encrypted_heap` with a plain heap under the same workload — the only stage that sees a lifetime bug |
| `tap/29_rotate_online_concurrent_access.t` | A table read or written while `pg_vault_tde_rotate_online()` runs stays readable after the rotation, after a second rotation and after a restart; a row lock on `pg_vault_tde_catalog` holds the worker inside the window (PSQLE-184). Runs once per available provider: local, Vault (`VAULT_ADDR`), PKCS#11 (SoftHSM2) |
| `tap/30_rotate_kek_atomicity.t` | A KEK rotation that rolls back, fails later in its statement, or dies in a crash leaves every table readable; so do a session that read the tables (local: unlocked the wallet) before another session rotated the KEK, and a `CREATE TABLE` that waited on an aborted rotation; a rotation cancelled by `statement_timeout` after the key store changed can be run again, from the same session and from another — each checked right after and after a restart, once per provider: local, Vault (`VAULT_ADDR`), PKCS#11 (SoftHSM2) (PSQLE-185, PSQLE-209) |
| `tap/31_migrate_vault_to_wallet.t` | `pg_vault_tde_migrate_vault_to_wallet()` refuses a passphrase that does not open the wallet and leaves every migrated table readable under the local wallet — in a session opened before the migration, a new one and after a restart; the real-Vault half runs when `VAULT_ADDR` is set (PSQLE-188) |
| `tap/32_rotate_online_toast.t` | Out-of-line values — stored uncompressed and compressed — stay readable after `pg_vault_tde_rotate_online()`, after a restart, after a second rotation and after another restart, and the TOAST relation keeps the same number of live chunks (PSQLE-189). Runs once per available provider, as `tap/29` |
| `tap/33_toast_lifecycle.t` | Out-of-line values go through every write path as on a plain heap twin put through the same statements — UPDATEs that keep, replace, inline or drop them, DELETE, VACUUM, `DROP COLUMN`, VACUUM FULL, two rotations and a restart; after each step the contents, a read of every whole row and the number of values left in the TOAST relation must match (PSQLE-189, 191, 192) |
| `tap/34_standby_rotation.t` | A streaming standby across `rotate_online()` on the primary: tables it had cached before the rotation read after it without an unwrap per row, and after a promotion new rows — including those of a table first touched by an `INSERT` — survive the promoted node's restart; one table rotated twice, one first read after the rotation as the control (PSQLE-190) |
| `tap/35_rotate_online_indexes.t` | Every index keeps finding every row across `pg_vault_tde_rotate_online()` — PRIMARY KEY, UNIQUE, plain, partial and `tde_btree`: a lookup through each, a full range, amcheck `heapallindexed`, and duplicate keys refused, after two rotations, a VACUUM and a restart (PSQLE-194) |
| `tap/36_verify_integrity_toast.t` | `pg_vault_tde_verify_integrity()` counts a row whose out-of-line value no longer decrypts: one byte flipped in a TOAST chunk's ciphertext (checksums off, as `tap/20`), the row counted once, the total still a row count, an untouched table clean, no resource left behind (PSQLE-196) |
| `tap/37_speculative_abort_toast.t` | An `INSERT ... ON CONFLICT` that loses the race to a concurrent insert leaves no out-of-line value behind, for DO NOTHING and DO UPDATE, on `encrypted_heap` and on a plain heap: the race is made deterministic with an expression index that blocks on an advisory lock, no injection points needed (PSQLE-197) |
| `tap/38_partial_index_build.t` | A partial index on `encrypted_heap` holds only the rows its predicate admits, against a plain heap twin: a valid `UNIQUE ... WHERE` builds and enforces, a partial btree has the heap twin's size after CREATE INDEX and REINDEX, amcheck `heapallindexed` passes, a `tde_btree` partial index answers, `reltuples` still counts every row (PSQLE-198) |
| `tap/39_index_build_old_snapshot.t` | An index built while an older REPEATABLE READ snapshot is open gives that snapshot what a plain heap twin gives it — rows deleted after it, HOT-updated rows by their old values only; after two rotations under an open snapshot `CREATE INDEX` still builds and marks the index `indcheckxmin`; a parallel build passes amcheck (PSQLE-201) |
| `tap/40_cluster_order.t` | `CLUSTER` on `encrypted_heap` puts the rows in index order through both paths core can choose (index scan, sort), against a plain heap twin with out-of-line values and a dropped column; `CLUSTER` on a `tde_btree` index is refused; `VACUUM FULL` still works (PSQLE-204) |
| `tap/41_reencrypt_table_privileges.t` | `pg_vault_tde_reencrypt_table()` rewrites a table only for a role holding `MAINTAIN` on it: a `pg_monitor` member (both overloads, one `SECURITY DEFINER`) and a role granted `EXECUTE` alone are refused; the owner, a `pg_maintain` member and a superuser succeed; `rotate_online()` is unaffected (PSQLE-205) |
| `tap/42_security_definer_callers.t` | Every key-management function refuses any caller but a superuser, whoever holds `EXECUTE`: a `pg_monitor` member cannot create a database's wallet, and a role granted `EXECUTE` on all ten is refused by each; a superuser still succeeds (PSQLE-206) |
| `tap/43_soak.t` | Soak test, skipped by default (`make ci-soak`, `SOAK_MINUTES`, `SOAK_SEED`): random `INSERT ... ON CONFLICT`, `UPDATE` and `DELETE` on an encrypted table and a heap twin, with random VACUUM, VACUUM FULL, CLUSTER, REINDEX, `rotate_online()`, `rotate_kek()` and immediate stops; after every round contents, whole rows, TOAST values, `verify_integrity()`, amcheck and index lookups (PSQLE-207) |
| `tap/44_damaged_wallet.t` | A local wallet truncated, empty, overwritten with random bytes, with one byte flipped, missing or unreadable: after a restart the server starts; reading, writing and creating an encrypted table fail with an ERROR; `wallet_unlock()`, `rotate_kek()`, `change_passphrase()` and `wallet_init()` fail and leave the file as it was; putting the file back restores every row. Leftover `wallet.p12.new` and `.lock` files are harmless; with the tables dropped, `wallet_init()` starts over (PSQLE-208) |
| `tap/45_verify_integrity_relcache_inval.t` | `pg_vault_tde_verify_integrity()` under a relcache invalidation of the table it scans: stopped on its first tuple (the DEK load waits on a lock), the table's `pg_class` row is updated as autovacuum does, and every tuple must still verify (PSQLE-207) |
| `tap/46_rotate_online_interrupted.t` | `rotate_online()` held halfway through its rewrite (an expression index waits on an advisory lock at row 500 of 1000), then cancelled, terminated, or the server stopped immediately: every tag verifies, the table equals its heap twin (whole rows, TOAST values), amcheck finds every row, the DEK generation is unchanged, right after and after a restart; the progress row says `failed` (still `running` after an immediate stop, as documented); a new rotation completes — once per provider (PSQLE-211) |
| `tap/47_basebackup_tde_search_path.t` | `pg_basebackup_tde` calls the extension's own `pg_vault_tde_seal_keys_bytea()` even when a database's owner puts a schema with a function of the same name first in the database's `search_path`: that function never runs, and the bundle written is the real one; a database with the extension in a schema off its `search_path` is sealed too (PSQLE-178) |
| `tap/48_iv_uniqueness.t` | No IV is used twice under one DEK: four sessions writing in turn, a restart and a rotation; every tuple of the heap and of its TOAST relation, dead versions included, is read off the raw pages and its (generation, IV) pair must be unique (PSQLE-178) |
| `tap/49_hot_update_short_indexed.t` | An UPDATE that changes a 1-byte indexed value (tde_btree on a 1-character text, btree on a bool) never goes HOT: 4000 UPDATEs, one per transaction, and after each the index finds the row by its new value and not by its old one, and UNIQUE refuses a duplicate; heap_update() compares ciphertext, which for a changed value of L bytes repeats with probability 256^-L (PSQLE-219) |

#### `tap/19_crash_recovery_rmgr.t`

Replays WAL written by the custom resource manager after an unclean shutdown.

`pg_vault_tde_rmgr.c` registers a resource manager whose `rm_redo` callback,
`tde_rmgr_redo()`, executes in exactly one situation: **WAL replay** — crash
recovery, PITR, or a standby applying the stream. Encrypted TOAST chunks are
routed through it by `tde_toast_wal_insert()` when
`pg_vault_tde.toast_custom_rmgr` is on.

Until this test existed, nothing in the suite ever replayed that WAL. The
`$node->restart` calls in other files are *clean* shutdowns, which checkpoint on
the way down and therefore replay nothing, so the redo path only ever ran on a
production system during recovery.

Three details are load-bearing, and each has an assertion guarding it:

- **`stop('immediate')`** — SIGQUIT, no shutdown checkpoint, so everything since
  the last checkpoint must be replayed. The test asserts
  `database system was not properly shut down` appears in the log after the
  restart; without it the test would keep passing if it silently stopped
  exercising recovery.
- **`STORAGE EXTERNAL` + incompressible payload** — with the default `EXTENDED`
  storage PostgreSQL compresses these values inline, no TOAST chunks are
  written, and the custom rmgr is never reached. The test asserts the TOAST
  relation exceeds 8 kB so it cannot pass vacuously.
- **A per-row payload seed** — the md5 over the whole column would not notice a
  row being replayed as a copy of its neighbour if every row held the same
  bytes.

Both branches of the `toast_custom_rmgr` GUC run: `on` is the path under test,
`off` is the control through `heap_insert`/`RM_HEAP`. If both fail, the fault is
in encryption or TOAST rather than in the resource manager.

#### `tap/18_guc_order_independence.t`

Pins the contract that the KMS GUCs behave as plain independent settings: a
value set at database level overrides the cluster-level one, and **nothing
depends on the order or the scope in which they were set**.

PostgreSQL applies a database's `pg_db_role_setting` entries one at a time —
`ProcessGUCArray()` walks the `setconfig` array in order, and `GUCArrayAdd()`
replaces an existing name *in place*, so re-issuing an `ALTER DATABASE SET`
does not move it to the end — while `process_settings()` applies the
`DATABASE_USER` scope before the `DATABASE` one.  Any provider `init()`
performed from the `kms_provider` assign hook therefore ran against a
half-applied configuration.  The test covers the four cases that broke:

1. `kms_provider` set *before* the wallet GUCs — no spurious passphrase
   WARNING, and the encrypted round-trip works.
2. A custom `wallet_path` set *after* `kms_provider` is honoured, and the
   per-database default path is not silently used instead.
3. `kms_provider` at `ALTER ROLE … IN DATABASE` scope with the wallet GUCs at
   `ALTER DATABASE` scope — the case that reordering the statements cannot fix.
4. `pg_vault_tde_wallet_status()` on a fresh connection reports the state of
   the *wallet*, not of the session: because `init()` is lazy, status has to go
   through the provider accessor or it would answer "nothing opened it yet".
5. Changing a passphrase source mid-session drops the KEK cached by
   `pg_vault_tde_wallet_unlock()`.

Plus a regression guard on the postmaster: a cluster-level `local` provider
must not attempt to open a wallet at startup, where there is no database and
therefore no wallet path.

### Isolation Tests (`test/isolation/specs/`)

- `per_table_dek_rotation.spec`: DEK rotation racing readers, writers and VACUUM.
- `encrypted_rewrite_concurrency.spec`: `VACUUM FULL` / `CLUSTER` with a reader
  holding a snapshot across the relfilenode change, and with a writer to wait for.
- `toast_update_concurrency.spec`: an `UPDATE` replacing an out-of-line value
  while another transaction updates or deletes the row, and a `DELETE` waiting on an
  `UPDATE`; every permutation ends
  with `VACUUM` and counts the values left in the TOAST relation (PSQLE-193).

### Anti-Patterns (DO NOT)

- **DO NOT** mix rows encrypted with different DEKs in the same table during
  sequential scan tests — the scan encounters wrong-DEK rows first and raises
  GCM authentication errors before reaching target rows. Use **separate tables
  per DEK epoch** in rotation tests.
- **DO NOT** assume `slot->tts_tid` is valid after
  `ExecFetchSlotHeapTuple(slot, false, ...)` — it returns the `tupdata`
  workspace with uninitialised `t_self`. Read TID from
  `bslot->base.tuple->t_self` while the buffer pin is held.
- **DO NOT** call `pg_vault_tde_decode_slot` on an already-decoded slot
  without the double-decode guard (`bslot->buffer == InvalidBuffer`).

---

## Packaging

### Pre-installation requirement: wallet base directory

Before the extension can initialise a local wallet, the base directory
`/var/lib/pg_vault_tde/` must exist and be owned by the OS user that runs
PostgreSQL (typically `postgres`).  The directory must **not** be inside
`PGDATA` — see [Security Considerations](#security-considerations).

The package installers (postinst / %pre) create it automatically.  For
bare source builds or container images, run once as root:

```bash
mkdir -p /var/lib/pg_vault_tde
chown postgres:postgres /var/lib/pg_vault_tde
chmod 0700 /var/lib/pg_vault_tde
```

`pg_vault_tde_wallet_init()` creates the per-database subdirectory
(`/var/lib/pg_vault_tde/<db_oid>/`) at runtime; it does **not** create
the base directory.  If the base directory is missing the function fails
with an actionable error and hint.

### DEB (Debian / Ubuntu)

```bash
# Build
bash packaging/build_deb.sh --no-sign

# Install (creates /var/lib/pg_vault_tde via postinst)
dpkg -i ../postgresql-18-pg-vault-tde_1.0-1_amd64.deb

# Verify
dpkg -l | grep pg-vault-tde
```

### RPM (RHEL / Rocky / Fedora)

```bash
# Build
bash packaging/build_rpm.sh

# Install (creates /var/lib/pg_vault_tde via %pre scriptlet)
dnf install ~/rpmbuild/RPMS/x86_64/postgresql18-pg_vault_tde-1.0-1.*.rpm

# Verify
rpm -qi postgresql18-pg_vault_tde
```

### Hardware Acceleration

There is a single build/package — no `TDE_TARGET_ARCH` variant. Hardware-
accelerated AES dispatch (AES-NI, VAES, ARM CE, SVE2) is provided by
OpenSSL's EVP layer automatically at runtime on this one build; see
"Hardware acceleration" above and `src/crypto/pg_vault_tde_hw_accel.c`.
A previous version of this codebase shipped `TDE_TARGET_ARCH`-selected
compiler flags and matching `-aesni`/`-vaes`/`-armce` package variants, but
none of the corresponding preprocessor defines (`TDE_HW_AES_NI`,
`TDE_HW_VAES`, `TDE_HW_ARM_CE`, `TDE_HW_ARM_SVE2`) were ever referenced by
any `#ifdef`/`#if defined` in `src/`, so those variants never produced a
measurably faster `.so` — they were removed.

### Support Matrix

Which (OS, PostgreSQL major) combinations a package is built for, and how much
verification stands behind each one.

**Generated from `packaging/build-matrix.json` by
`packaging/gen-support-matrix.sh`.** Edit the JSON and re-run the script; do not
edit the table below by hand, or the two disagree the next time anyone does run
it. No CI job enforces this — keeping them in step is part of changing the build
matrix.

- **Functional suite** — the regression, TAP, isolation, KMS and backup suites
  run on this OS and PostgreSQL major (`make ci-all`), building from source.
- **Package install** — the built `.deb`/`.rpm` is installed on a clean system
  of this OS and `CREATE EXTENSION` is verified
  (`ci/scripts/run-install-test.sh`).
- **Neither** — the package is compiled for that combination and nothing more.

The two columns are independent checks, not levels of one scale: a row may have
its sources fully exercised while its package is never installed, and the other
way round. Every combination requires OpenSSL 3.x, which is why distributions
shipping only 1.1.1 (Debian 11, EL8) are absent from the matrix rather than
listed as unsupported.

<!-- BEGIN GENERATED: support matrix (packaging/gen-support-matrix.sh) -->

| Format | OS | PG | Functional suite | Package install |
|---|---|---|---|---|
| deb | `ubuntu:22.04` | 17 | no | yes |
| deb | `ubuntu:22.04` | 18 | no | yes |
| deb | `ubuntu:24.04` | 17 | no | yes |
| deb | `ubuntu:24.04` | 18 | no | yes |
| deb | `ubuntu:26.04` | 17 | no | yes |
| deb | `ubuntu:26.04` | 18 | no | yes |
| deb | `debian:12` | 17 | no | yes |
| deb | `debian:12` | 18 | no | yes |
| deb | `debian:13` | 17 | yes | yes |
| deb | `debian:13` | 18 | yes | yes |
| rpm | `rockylinux:9` | 17 | no | yes |
| rpm | `rockylinux:9` | 18 | no | yes |
| rpm | `rockylinux:10` | 17 | no | yes |
| rpm | `rockylinux:10` | 18 | no | yes |
| rpm | `almalinux:9` | 17 | no | yes |
| rpm | `almalinux:9` | 18 | no | yes |
| rpm | `almalinux:10` | 17 | no | yes |
| rpm | `almalinux:10` | 18 | no | yes |

<!-- END GENERATED: support matrix -->

### KMS Provider Coverage

Which key-management backend each `pg_vault_tde.kms_provider` value is actually
exercised against in CI, and with what. Maintained by hand — unlike the table
above, nothing generates it.

| Provider | Backend under test | Version under test | Suites |
|---|---|---|---|
| `local` | PKCS#12 wallet on local disk; no external service | — | `regress`, `wallet`, `checksums`, `isolation`, `schema`, `bench`, `tap/02_backup_local.t` |
| `vault` | HashiCorp Vault, Transit secrets engine (`ci/dump-compose.yml`) | 2.1.0, pinned by digest in `ci/containers/real-vault.Containerfile` | `vault`, `tap/03_backup_vault.t` |
| `openbao` | OpenBao, Transit secrets engine, 3-node Raft cluster (`bao-1`…`bao-3` plus `bao-init`) | 2.6.2, pinned by digest in `ci/compose-openbao*.yml` | `openbao` |
| `pkcs11` | SoftHSM2 software token, created fresh per run in a tempdir | 2.6.1-3, from the base image's Debian | `pkcs11`, `tap/16_pkcs11.t` |

Two things this table is saying, and one it is not:

- **No physical HSM is exercised.** The `pkcs11` provider is verified against a
  software token only. A vendor module is loaded through the same `dlopen`
  path, but no real device, PIN policy or slot behaviour is covered here — see
  [PKCS#11 / HSM Provider](#pkcs11--hsm-provider) for what the provider expects
  of one.
- **Both service images are pinned** by version and digest (PSQLE-180), so a new
  upstream release enters CI only when someone moves the pin — see PGXN.md,
  *Keeping pins current*; `make ci-pins` fails on an unpinned one. OpenBao's is
  overridable with `$OPENBAO_IMAGE` to try another version; Vault's lives in
  `ci/containers/real-vault.Containerfile`, which builds the local `real-vault`
  image.


---

## Roadmap

See [ROADMAP.md](ROADMAP.md) for the full release roadmap.

| Version | Theme | Status | Tests |
|---------|-------|--------|-------|
| **v1.1** | KMS/Vault + Key Rotation + HW Accel | ✅ Completed | 41 |
| **v1.2** | Logical Decoding | ✅ Completed | — |
| **v1.3** | Vault KEK + multi_insert + BGW | ✅ Completed | 48 |
| **v1.4** | CI/CD + tde_btree + Wire Format v2 | ✅ Completed | 52 |
| **v1.5** | Per-Table DEK + Online Rotation + AAD | ✅ Completed | 72 |
| **v1.6** | Local Wallet KMS (production-ready) | ✅ Completed | 109 |
| **v1.7** | Per-database KMS + pg_restore_tde + PGC_SUSET + enc_ops indexes | 🔄 Current | 145 |
| **v1.8** | KMIP + Column-Level + GIN/Hash/GiST/BRIN + HA + Dual-Control | 📋 Q2 2027 | ~160 |

### Permanent Deferrals

| Gap | Reason |
|-----|--------|
| WAL encryption | Requires PG core hook (`XLogInsert()`) — not extension API |
| BRIN on encrypted columns | min/max of ciphertexts is meaningless |
| General GiST (range, geometric) | Penalty/picksplit requires ordering |
| Full-text phrase search | Positional ordering destroyed by AES-SIV |

---

## Contributing

This extension follows PostgreSQL's BSD-derived coding style and
`pgindent` formatting conventions. All contributions must:

- Pass `make` with `-Wall -Wextra` and zero warnings
- Pass the full 109-test regression suite (`make ci-regress`)
- Pass the page checksum compatibility test (`make ci-checksums`)
- Use `palloc` / `pfree` exclusively (never `malloc` / `free`)
- Use `ereport` / `elog` exclusively (never `printf` / `exit`)
- Be C99 conformant with `snake_case` naming
- Clean IV + DEK memory with `OPENSSL_cleanse` before `pfree`
- Not introduce circular module dependencies (TAM → Crypto → KMS; never reverse)
- Not use GPL/AGPL libraries (breaks PostgreSQL License compatibility)