# pg_vault_tde Roadmap

> Last updated: 2026-06-29 — **v1.7 current**. 109 regression tests (52 v1.4 + 20 v1.5 + 37 v1.6) carried forward, plus the `tap/12_logical_repl_toast.t` end-to-end logical replication test. Key v1.7 changes: all KMS GUCs promoted to PGC_SUSET (per-database KMS via `ALTER DATABASE SET`); `pg_restore_tde` decrypt-and-pipe loop completed; logical replication of `encrypted_heap` TOAST columns via a custom WAL resource manager (`pg_vault_tde.toast_custom_rmgr`); documentation updated throughout.

---

## Completed Releases (Summary)

### v1.1 — KMS / Vault + Key Rotation + TOAST + HW Accel — COMPLETED ✅

> **41 regression tests** — PG 17 + PG 18, zero compiler warnings.

Vault HTTP connector (libcurl async, Transit API), AppRole + K8s JWT auth, graceful key rotation
with `prev_dek` fallback, `pg_vault_tde_reencrypt_table()`, TOAST pre-TOAST fix, PG 17/18/19
build infrastructure, hardware acceleration (OpenSSL 3.x QAT/FIPS/default provider), `tde_btree`
IAM stubs, tests 1–43.

### v1.2 — Logical Decoding Compatibility — COMPLETED ✅

Custom output plugin `pg_vault_tde_pgoutput` — intercepts `change_cb`, decrypts `encrypted_heap`
tuples in-place. Test 48. Known limitation: externally-TOAST'd columns not supported.

### v1.3 — Vault Transit KEK + multi_insert + BGW + health_check — COMPLETED ✅

> **48 regression tests** — PG 17 + PG 18, zero compiler warnings.

Vault Transit KEK wrapping (wrapped DEK persisted to `$PGDATA`), `multi_insert` batching
(3-phase pre-TOAST+encrypt), background worker for token renewal, `pg_vault_tde_health_check()`
14-column composite. Tests 44–48.

### v1.4 — CI/CD + tde_btree + Wire Format v2 — COMPLETED ✅

> **52 regression tests** — PG 17 + PG 18, zero compiler warnings.

CI benchmark pipeline (`run-bench.sh`), OpenBao 3-node Raft integration (12 tests),
wire format v2 with generation tag, `tde_btree` full wiring (ambuild/aminsert/amrescan),
security hardening (file permissions, secret_id rotation, token TTL logging). Tests 49–52.

### v1.5 — Per-Table DEK + Online Rotation + Wire Format v3 AAD — COMPLETED ✅

> **72 regression tests** (52 v1.4 + 20 new) — PG 17 + PG 18, zero compiler warnings.

Per-table DEK catalog (`pg_vault_tde_catalog`), KMS provider abstraction layer
(`pg_vault_tde_kms_provider.h`), TOAST heap-level round-trips, `tde_btree` native type
operator classes (text/int4/uuid/numeric/date/timestamptz), wire format v3 AEAD AAD binding
(cross-table paste attack prevention), online key rotation BGW (`pg_vault_tde_rotate_online`),
AppRole response-wrapping. Wallet SQL stubs registered (not functional). Tests 53–72.

---

## v1.6 — Local Wallet KMS — Production-Ready Offline Encryption — COMPLETED ✅

> Completed: 2026-06-03, patched 2026-05-08 — **109 regression tests** (52 v1.4 + 20 v1.5 + 37 v1.6) — PG 17 + PG 18, zero compiler warnings.

Local Wallet KMS provider (`src/kms/pg_vault_tde_kms_local.c`) — full PKCS#12 / AES-256-WRAP
implementation, PBKDF2-SHA256 (600,000 iterations, NIST SP 800-132), `0600` wallet file
permissions. Flexible passphrase ingestion via GUCs (env var, file, shell command, dev-mode
convenience; priority `command > env > file > dev_mode`) plus SQL `wallet_unlock`/`wallet_lock`
for interactive control without a restart. `pg_vault_tde_wallet_status()` 6-column SRF.
KEK rotation and passphrase change re-wrap all DEKs atomically via SPI. Zero-downtime
`pg_vault_tde_migrate_vault_to_wallet()`. TOAST chunk-level storage encryption shipped here
as the foundation for v1.7's logical-replication work. Patch fixed a write-path error-handling
gap (`PG_TRY` widened to cover the full write pipeline in all four write callbacks) and added
the `RELKIND_TOASTVALUE` read-path bypass so real TOAST chunks round-trip correctly. Tests 73–109.

> The original wallet export/import bundle functions (`pg_vault_tde_wallet_export_bundle`/
> `_import_bundle`) shipped in v1.6 were removed in v1.7, superseded by
> `pg_vault_tde_seal_keys()`/`pg_vault_tde_unseal_keys()`.

---

## v1.7 — TOAST Chunks + KEK Hierarchy + HSM + Audit

> Status: ✅ Completed
> **Target**: ~100 regression tests — PG 17 + PG 18 + PG 19.

**Theme**: Close the TOAST data-leak gap, formalize the KEK/DEK wrap hierarchy across
all providers, add PKCS#11/HSM support, audit trail for compliance (PCI-DSS, HIPAA).

### 1. TOAST Chunk-Level Storage Encryption (foundation — shipped in v1.6)

Per-chunk AES-256-GCM at the `pg_toast_NNNNN` storage layer using the parent
relation DEK (delivered in v1.6; listed here as the foundation the v1.7 logical
replication work in §4 builds on). Raw TOAST pages no longer contain plaintext.

### 2. Proper KEK/DEK Wrapping Hierarchy (Critical)

Provider-agnostic `pg_vault_tde_kms_wrap_dek()`/`unwrap_dek()` API. Vault Transit acts
as key protector (not key store) — raw DEK never sent to Vault, only wrapped ciphertext.
`pg_vault_tde_catalog.wrapped_dek` authoritative for all providers.

### 3. tde_btree Fixed-Size Type Encryption — ✅ Completed in v1.7

Custom btree key serialisation layer for `int4`/`int8`/`uuid`/`date`/`timestamptz`.
All operator classes now store encrypted index keys. Index-only scans are disabled
by design to prevent returning raw AES-256-SIV ciphertext to clients.

### 4. Logical Replication of TOAST Columns (Medium) — COMPLETED ✅

Custom WAL resource manager (`pg_vault_tde.toast_custom_rmgr`, PGC_POSTMASTER,
default off): `tde_toast_wal_insert()` logs encrypted TOAST chunks under
`TDE_RMGR_ID` so the logical decoder routes them away from the reorder buffer's
`toast_hash`; `rm_decode` captures them per transaction and `tde_toast_stitch()`
reconstructs the plaintext value into the decrypted main tuple before `pgoutput`
serializes it. UPDATE/DELETE require `REPLICA IDENTITY FULL` + a primary key
(`DEFAULT` / PK-less unsupported — the replica identity would be read from
ciphertext). Covered end-to-end by `tap/12_logical_repl_toast.t`.
See doc/pg_vault_tde.md → "Logical Decoding and Replication".

### 5. PKCS#11 / HSM Integration (Critical) — COMPLETED ✅

`src/kms/pg_vault_tde_kms_pkcs11.c` — direct Cryptoki: the vendor module is
dlopen()ed and DEKs are wrapped with `C_WrapKey`/`C_UnwrapKey`
(`CKM_AES_KEY_WRAP`, AES-256 KEK with `CKA_EXTRACTABLE=FALSE`). The
originally-planned OpenSSL 3.x pkcs11-provider route was evaluated and
discarded: symmetric key wrap with an opaque token key is not expressible
through EVP (would force an RSA KEK), and the `pkcs11-provider` package is
missing/outdated on the DEB targets. OASIS v3.2 headers vendored under
`src/include/pkcs11/`. KEK provisioning via `pg_vault_tde_pkcs11_keygen()`;
rotation via the standard `pg_vault_tde_rotate_kek()`. Every KEK generation
is an immutable token object labelled `<pkcs11_key_label>.v<N>` (N never
reused, never renamed or destroyed); "current" is simply the highest N on
the token, and every `wrapped_dek` blob is prefixed with the version tag
of the KEK that produced it, so unwrap always finds the right key
regardless of what is "current" — including across a crash mid-rotation.
GUCs: `pkcs11_library`, `pkcs11_token_label`, `pkcs11_slot_id`,
`pkcs11_pin_env`, `pkcs11_key_label`. CI with SoftHSM2
(`tap/16_pkcs11.t`, 19 assertions, `make ci-pkcs11`). Follow-up: `pg_dump_tde`/
`pg_restore_tde` FRONTEND shim (they currently error out cleanly).

**Cross-backend KEK-rotation propagation**: a shared-memory beacon
(`Pkcs11SharedState`: one `LWLock` + a `uint32 current_kek_version`,
mapped via `pg_vault_tde_kms_pkcs11_shmem_request`/`_shmem_init`, same
dynamic-tranche pattern as the Vault token cache) lets an already-connected
backend pick up a KEK rotation committed by a *different* connection
without reconnecting. The raw `CK_OBJECT_HANDLE` is never shared across
processes (PKCS#11 handles are only meaningful within the session that
resolved them) — only the version number is; each backend re-resolves its
own handle locally via `pkcs11_find_key_by_label()`. Written only from
`pkcs11_commit_kek_rotation()` and the initial keygen (never from
`prepare_kek_rotation`, to avoid leaking an armed-but-uncommitted rotation
cluster-wide); read opportunistically on every wrap/unwrap/rewrap call via
`pkcs11_refresh_kek_if_stale()`, so staleness is bounded by "this backend's
next operation", not wall-clock time.

### 6. Audit Trail / Event Log (Critical) — COMPLETED ✅

`src/audit/pg_vault_tde_audit.c` — 10 event types (`KEY_ROTATION`, `DEK_ACCESS`,
`INTEGRITY_VIOLATION`, `WALLET_OPEN`, etc.). `pg_vault_tde_audit_log` encrypted table.
PCI-DSS Requirement 10 / HIPAA §164.312(b).

### 7. pg_dump Plaintext Leak Protection (Medium) — NOT completed, moved to v1.8

Designed (see `src/backup/pg_vault_tde_backup.c` header comment, "Layer 2 — SQL-LEVEL
GUARD"): `ProcessUtility_hook` would intercept `COPY TO` on encrypted tables and emit a
WARNING, gated by GUC `pg_vault_tde.dump_plaintext_warning`. Neither the hook nor the GUC
exist in code yet — tracked as v1.8 §10 below.

### 8. Physical Backup Key Sealing / `pg_restore_tde` (Medium) — COMPLETED ✅

`pg_restore_tde` standalone binary (`src/backup/pg_restore_tde.c`): reads the
`tde_backup_header`, unwraps the DEK via the active KMS provider
(`tde_backup_header_validate()`), decrypts the AES-256-GCM block stream
(`tde_backup_decrypt_block()` with block_seq as AAD), and pipes plaintext to
`pg_restore -Fc`.

`pg_vault_tde_seal_keys()`/`pg_vault_tde_seal_keys_bytea()`/`pg_vault_tde_unseal_keys()` (`src/kms/pg_vault_tde_seal.c`) — signed bundle of all `wrapped_dek` entries (KEK excluded), for `pg_basebackup`; TAP `tap/14_seal_keys.t`.
`pg_basebackup_tde` (`src/backup/pg_basebackup_tde.c`) — pg_basebackup wrapper: seals every database's keys via `seal_keys_bytea` before the backup and writes one `pg_vault_tde_keys.<datname>.sealed` bundle per database after it succeeds; TAP `tap/15_basebackup_tde.t`.
A core-side `BackupState`/`bbsink` hook was evaluated and discarded: PostgreSQL exposes no extension hook to inject files into the `pg_basebackup` stream, and a custom `bbsink` runs in the walsender without SPI.

---

## v1.8 — KMIP + Column-Level + GIN/Hash/GiST/BRIN + HA + Dual-Control (Q2 2027)

> Status: 📋 Defined
> **Target**: ~130 regression tests.

**Theme**: Enterprise HA, KMIP standards compliance, column-level encryption,
regulated-industry features.

### 1. Column-Level Encryption (High)

`ALTER TABLE ... ENABLE/DISABLE COLUMN ENCRYPTION` DDL. Per-column DEK support.
`pg_vault_tde_columns` catalog. `src/tam/pg_vault_tde_column.c`.

**Feasibility (verified against the current TAM architecture, see
`tam.instructions.md`)**: `encrypted_heap` today encrypts the whole tuple as
one opaque AES-256-GCM blob (`tde_encrypt_heap_tuple`, wire format v4) —
there is no per-Datum boundary. Column-level encryption needs the write
path to operate around `heap_deform_tuple`/`heap_form_tuple` for specific
attributes instead of the raw tuple bytes:
- **Varlena columns** (`text`, `bytea`, `jsonb`, `numeric`, arrays):
  straightforward — store `[IV|ciphertext|GCM-tag]` as the Datum's own
  varlena payload, the same shape already used at the tuple level, just
  scoped to one attribute. No storage-layout change needed.
- **Fixed-size columns** (`int4`, `int8`, `date`, `timestamptz`, ...):
  AES-256-GCM's IV+tag overhead does not fit the type's fixed storage
  width. Either (a) reuse `tde_btree`'s AES-256-SIV scheme — deterministic,
  same output length as input, same security trade-off already accepted
  for index keys (no protection against frequency analysis) — or (b)
  widen physical storage (bigger lift: a pseudo-type or forced
  `bytea`-backed column; likely out of scope for a first cut).
- **Query pushdown**: `WHERE col = ...` on an encrypted column needs the
  same encrypt-then-compare trick `tde_btree` already implements for an
  index to be usable; without a matching index it falls back to sequential
  scan + per-Datum decrypt (same cost model as today's whole-row decrypt,
  just narrower).
- **Two distinct feature shapes to choose between**: (a) column encryption
  as an *additional* layer inside `encrypted_heap` — a specific sensitive
  column (SSN, card number) gets its own DEK/rotation/audit trail
  independent of the table DEK, for defense-in-depth or per-column access
  control; (b) column encryption on an *ordinary* `heap` table, without
  switching the whole table to `encrypted_heap` — a lighter-weight opt-in
  for one or two sensitive columns. (a) reuses most of the existing TAM
  plumbing; (b) needs a new, narrower write/read hook that does not exist
  anywhere in the codebase today.

### 2. GIN Index Encryption (Medium)

`src/iam/pg_vault_tde_gin.c` — per-entry AES-256-SIV. Equality operators only
(`@>`, `?`, `&&`). Phrase search permanently rejected by `amvalidate`.

**Feasibility**: same delegation pattern already proven by `tde_btree` (see
`iam.instructions.md` — `amgettuple`/`amendscan`/`ambulkdelete`/
`amvacuumcleanup` delegate unchanged to the real AM; only the key
boundary is intercepted). GIN's entry tree needs a *consistent* comparator
for its internal structure, not a semantically meaningful order — encrypting
each key extracted by `extractValue`/`extractQuery` with AES-256-SIV before
handing it to GIN's own entry-tree code preserves exactly that: equal
plaintexts still compare equal, and a stable (if arbitrary) ciphertext
byte-order is all GIN's internals require. Lower risk than GiST (below)
precisely because GIN, like btree, has no semantic-distance requirement.

### 3. Hash Index Encryption (Low Effort)

`src/iam/pg_vault_tde_hash.c` — same AES-256-SIV pattern as `tde_btree`;
hash index buckets only need bucket-hash + exact equality, both of which
survive deterministic encryption unchanged. Same low-risk delegation
pattern as GIN above.

### 4. pg_statistic Plaintext Mitigation (Low)

Post-`ANALYZE` hook: NULL out `stavalues` for encrypted columns.
GUC `pg_vault_tde.encrypt_statistics`.

### 5. KMIP 1.2 Client (Enterprise)

`src/kms/pg_vault_tde_kms_kmip.c` — KMIP 1.2 over mutual TLS. CI with PyKMIP.

### 6. GiST Equality-Only Encryption (Medium)

`src/iam/pg_vault_tde_gist.c` — equality-only operator classes. `amvalidate`
rejects range/geometric strategies.

**Feasibility, and why this is harder than GIN/Hash above**: unlike btree/
GIN/Hash, GiST cannot delegate its tree-shaping support functions
(`penalty`, `picksplit`, `union`, `distance`) to the real opclass on
ciphertext — those functions encode actual geometric/semantic distance in
the plaintext domain, which AES-SIV ciphertext has none of by design (that
*is* the point of encryption). A working equality-only GiST needs genuinely
custom, non-delegated support functions that make no attempt at
selectivity (e.g. constant penalty, arbitrary picksplit) and rely entirely
on `consistent` for an exact ciphertext match — functionally correct, but
with materially worse pruning than a real GiST tree, closer in practice to
a linear scan over each visited page. Worth it specifically for types that
have **no other native access method** in PostgreSQL (`point`, `circle`,
`box`, `inet` with non-equality operators unused) — for anything with a
usable `tde_btree` or the GIN path above, prefer those instead.

### 7. Streaming Replication Standby DEK Distribution (Medium)

`pg_vault_tde_replica_setup()` — read-only KMS credentials for standby. HA
documentation for all KMS providers.

### 8. Dual-Control / M-of-N Key Ceremony (High)

`pg_vault_tde_key_custody_info()`. Vault Shamir + PKCS#11 PIN-split documentation.

### 9. BRIN Bloom Equality Encryption (Medium — new candidate, needs a spike)

`src/iam/pg_vault_tde_brin.c`. The Permanent Deferrals table below correctly
rules out `minmax` BRIN opclasses (ciphertext has no meaningful min/max) —
but PostgreSQL's `bloom` BRIN opclasses (core since PG 14,
`src/backend/access/brin/brin_bloom.c`) only need a per-block-range Bloom
filter of value hashes, never an ordering. Since AES-256-SIV is
deterministic (equal plaintext → equal ciphertext, the same property
`tde_btree` already relies on), hashing the raw ciphertext bytes directly
(`hash_any()`) produces exactly the membership test a bloom filter needs —
no type-specific logic required at all, unlike `tde_btree`/GIN/GiST which
need per-type SIV encode/decode. A single generic "encrypted equality"
bloom opclass could work uniformly across every type this project already
supports, giving cheap block-range pruning for equality predicates on
large encrypted tables at a fraction of `tde_btree`'s storage cost.
Needs a short technical spike before committing engineering time: confirm
the BRIN opclass support-function contract (`opcinfo`/`add_value`/
`consistent`/`union`) can be satisfied purely on ciphertext bytes without
ever needing the plaintext inside the index AM.

### 10. pg_dump Plaintext Leak Protection (Medium — carried over from v1.7, never implemented)

`ProcessUtility_hook` intercepts `COPY TO` on encrypted tables — emits WARNING.
GUC `pg_vault_tde.dump_plaintext_warning = on`. Designed in v1.7 (see
`src/backup/pg_vault_tde_backup.c` header comment) but the hook and GUC were
never written; carried forward here as the actual target release.

---

## Permanent Deferrals

These gaps **cannot be closed without modifying PostgreSQL core**.

| Gap | Reason |
|-----|--------|
| **WAL / redo encryption** | Requires hook in `XLogInsert()` / `XLogWrite()` — no extension API |
| **BRIN minmax on encrypted columns** | min/max of AES-SIV ciphertexts is meaningless — no ordering preserved. (Bloom-based BRIN equality pruning is *not* in this category — tracked as a real candidate, see v1.8 §9.) |
| **General GiST** (range, geometric) | Penalty/picksplit requires ordering; AES-SIV destroys it. (Equality-only GiST is *not* in this category — tracked separately, see v1.8 §6.) |
| **pg_upgrade transparent migration** | `pg_upgrade` copies files without TAM; manual `reencrypt_table()` required |
| **Full-text phrase search on encrypted tsvector** | `<->` proximity requires positional ordering |
| **`WITH HOLD` cursor temp file encryption** | The held-cursor tuplestore is written by the executor's storage layer directly, bypassing the TAM — no hook exists anywhere in the `WITH HOLD` cursor lifecycle to intercept it. See README.md § Limitations item 6. |

---

## Version Summary

| Version | Theme | Completed | Tests | Key Features |
|---------|-------|-----------|-------|-----------------------|
| **v1.1** | KMS/Vault + Key Rotation + HW Accel | ✅ 2026 | 41 | Vault Transit, AppRole, prev_dek fallback, OpenSSL 3.x HW dispatch |
| **v1.2** | Logical Decoding | ✅ 2026 | — | `pg_vault_tde_pgoutput` output plugin |
| **v1.3** | Vault KEK + multi_insert + BGW | ✅ 2026 | 48 | Transit KEK wrapping, batch COPY, token renewal BGW, health_check |
| **v1.4** | CI/CD + tde_btree + Wire Format v2 | ✅ 2026-07-05 | 52 | OpenBao 3-node Raft, ambuild/aminsert/amrescan, generation tag |
| **v1.5** | Per-Table DEK + Online Rotation + AAD | ✅ 2026 | 72 | Per-table catalog, native type ops, wire format v3, rotate_online BGW |
| **v1.6** | Local Wallet KMS (production-ready) + write-path / catalog bugfix patch | ✅ 2026-07-20 (patched 2026-05-08) | 109 | Wallet unlock/lock, passphrase flexibility, KEK rotation, export/import, Vault→wallet migration; PG_TRY widening; TOAST relid auto-registration; STORAGE EXTERNAL TAM read bypass; all-read-paths TOAST coverage; forensic helpers; tests 73–109 |
| **v1.7** | Per-database KMS + pg_restore_tde + PGC_SUSET + PKCS#11 + HSM + v1.4 removal | ✅ Completed | 109 | All KMS GUCs PGC_SUSET → per-database KMS via `ALTER DATABASE SET`; `pg_restore_tde` full decrypt-and-pipe restore loop; removed v1.4 global-DEK backward compat (`TdeShmemData`, `rotate_key`, `key_generation`, `clear_prev_dek`, `encrypt_test`, `decrypt_test`); PKCS#11/HSM provider with cross-backend KEK-rotation propagation; documentation overhaul |
| **v1.8** | KMIP + Column-Level + GIN/Hash/GiST/BRIN + HA + Dual-Control | Q2 2027 | ~130 | KMIP 1.2 client, per-column encryption, GIN/Hash/GiST(equality)/BRIN(bloom) index AMs, streaming replication standby DEK distribution, M-of-N key ceremony, pg_dump/COPY TO plaintext-leak WARNING (carried over from v1.7) |