# v0.87.17 - Versioned Row Identity V2 Hardening and Recreation

> **Status:** Planned
> **Scope:** 9 person-weeks
> **User promise:** *"I can adopt exact row identities with a clear, repeatable rebuild and no silent compatibility trap."*
> **Blocked by:** [v0.87.16](v0.87.16.md)
> **Proposal:** [Exact, Versioned Row Identity V2](../proposals/PROPOSAL_VERSIONED_ROW_IDENTIFY_ENCODING.md)

## Theme

Turn the V2 implementation into a releasable pre-1.0 contract. Exercise all
producer and consumer paths together, finish the index and encoder performance
matrix, and make the intentional breaking upgrade operationally boring through
preflight checks and a documented stream-table recreation workflow. External
replication, outbox, DuckLake, and dbt consumers receive explicit resnapshot
requirements, while diagnostics and permissions acknowledge that canonical
identities can contain reversible source values.

There is no in-place V1/V2 migration protocol in this release. The supported
upgrade drops V1 stream-table state in reverse dependency order, preserves source
tables and data, installs the V2 extension, recreates the graph in dependency
order, and establishes a fresh V2 initial-refresh baseline.

## Scope allocation

| Work package | Person-weeks |
|---|---:|
| Cross-path correctness and failure hardening | 2.00 |
| Performance, capacity, and regression gates | 1.50 |
| Recreation preflight and operator workflow | 2.00 |
| External consumer and replication compatibility | 1.25 |
| Privacy, privileges, documentation, and examples | 1.00 |
| Release review and full CI closure | 1.25 |
| **Total** | **9.00** |

## Items

### ROWID-12: Prove cross-path V2 equivalence

Run the exact same source histories through trigger CDC, WAL CDC, FULL,
differential, and IMMEDIATE paths and compare complete table contents and
identity bytes with the v0.87.14 oracle. Cover joins, aggregates, set
operations, windows, keyless multiplicity, enum DDL invalidation, source
updates that change keys, and common-prefix identities.

**Dependencies:** v0.87.16; v0.87.14 correctness and replay gates.
**Schema change:** No.

### ROWID-13: Close performance and capacity gates

Complete the encoder, probe, index, refresh, and replication benchmark matrix.
Measure throughput, allocations, index size, buffer hits, WAL, encoded-ID
storage, change-buffer growth, cached and cold MERGE latency, and full old-row
cost for `REPLICA IDENTITY FULL`. Freeze only the simplest strategy that meets
correctness and capacity requirements; retain direct full-ID indexes for proven
bounded identities and do not ship speculative generated-column machinery.

**Dependencies:** v0.87.16 benchmark scaffolding and selected probe version.
**Schema change:** No.

### ROWID-14: Implement recreation preflight and workflow

Add a small fail-safe preflight that validates identity types, typmods,
collations, source-key eligibility, operator classes, PostgreSQL-major registry
contracts, index-size readiness, and remaining V1 stream tables or buffers.
The preflight is read-only: it may inspect catalogs and report blockers, but it
must not alter source data, stream-table state, indexes, buffers, grants, or
external objects. It must also require a recorded inventory and acknowledgment
of external consumers before teardown begins.
Provide an ordered procedure to pause scheduling, export definitions and
metadata, drop V1 stream tables in reverse dependency order, clean unused V1
buffers, install/restart the V2 binary, recreate stream tables in dependency
order, and perform fresh initial refreshes. Never change source data or
partially convert a stream table.

**Dependencies:** v0.87.16 version guards; existing lifecycle and catalog APIs.
**Schema change:** Yes.

### ROWID-15: Publish external consumer resnapshot contracts

Inventory and document the breaking `BYTEA` contract for logical replication,
publications, outbox consumers, DuckLake sinks, dbt models, and user SQL that
assumes numeric row IDs. Require consumer schema updates and resnapshot after
recreation; provide no numeric cast because V1 values do not contain the
original logical identity. The inventory must record the consumer owner,
affected stream tables, whether `__pgt_row_id` or storage layout is consumed,
required schema change, resnapshot status, and acknowledgment before V1 state
is dropped. Include pause/write-window guidance and state that writes during
recreation are not replayed by V2.

**Dependencies:** ROWID-14; existing publication, outbox, DuckLake, and dbt
integration contracts.
**Schema change:** No.

### ROWID-16: Harden identity privacy and diagnostics

Keep `__pgt_row_id` out of default user-facing views and grants unless explicit
ownership or privilege permits access. Logs, errors, traces, monitoring,
support bundles, and dump guidance expose length and a short diagnostic
fingerprint rather than complete bytes or reversible prefixes. Document that
pass-through and keyless identities may retain source or projected values and
must receive the same controls as those source columns.

**Dependencies:** v0.87.15 opaque-identity contract; v0.87.16 storage exposure.
**Schema change:** No.

### ROWID-17: Make V2 a release-gated contract

Add release checks for exact vectors, cross-path table equality, version
mismatch rejection, supported-major validation, direct/probe classification,
replication behavior, recreation preflight, external-consumer documentation,
privacy review, and benchmark regressions. Check that older binaries reject V2
catalog state and V2 binaries reject leftover V1 state. Keep the supported
matrix and the recreation runbook synchronized with generated SQL and package
artifacts. The gate must verify the checked-in wire-format artifact against the
encoder, the named initial structural support/rejection matrix, interval's
complete 128-bit comparison encoding, all source-key eligibility cases, the
authoritative uniqueness rule, and planning-time rejection of unique unbounded
`IMMEDIATE` identities.

**Dependencies:** ROWID-12 through ROWID-16; all prior correctness and lifecycle
security release gates.
**Schema change:** No.

## Correctness

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| CORR-8 | Compare every maintenance path | L | P0 |
| CORR-9 | Verify enum DDL transitions | M | P0 |
| CORR-10 | Protect recreation baselines | M | P0 |

**CORR-8** compares exact rows, multiplicities, identity bytes, and refresh
outcomes across all CDC and refresh modes. The oracle treats effective mode,
fallback, skipped work, and errors as part of the result rather than accepting a
row-count match alone.

**CORR-9** warms enum encoder caches, renames enum labels in another session,
checks DDL invalidation and syscache visibility, and verifies that dependent
stream tables are reinitialized before consuming new label bytes.

**CORR-10** proves that a failed or interrupted recreation leaves source data
unchanged and produces a repairable incomplete V2 graph. Recovery drops the
incomplete graph and reruns the documented procedure; it never attempts a
partial V1-to-V2 replay.

## Stability

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| STAB-7 | Make preflight non-destructive | M | P0 |
| STAB-8 | Handle restart and stale binaries | M | P0 |
| STAB-9 | Bound repair diagnostics | S | P1 |

**STAB-7** runs every unsupported-type, bad-collation, stale-buffer,
missing-relation, wrong-major, and insufficient-index-size check in a read-only
preflight before state mutation. It verifies that no catalog, relation, index,
buffer, grant, or external object changes, and that the report is complete
enough for an operator to repair all blockers in one pass.

**STAB-8** tests PostgreSQL restart after shared-library replacement, backend
reuse, scheduler pause/resume, old binary readers, and extension-update order.
Any V1/V2 ambiguity fails closed with no partial refresh or source mutation.

**STAB-9** keeps fingerprints, candidate counts, and identity lengths bounded in
logs and support artifacts. Error paths never dump complete identities by
accident, and cleanup remains idempotent after a failed recreation.

## Performance

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| PERF-7 | Gate identity regressions | M | P0 |
| PERF-8 | Measure replication cost | M | P1 |
| PERF-9 | Validate cold-index behavior | M | P1 |

**PERF-7** adds Criterion and PostgreSQL benchmark thresholds for encoder
throughput, allocations, CDC overhead, direct/probe lookup, and refresh latency.
Baselines are checked in CI and changes that exceed the agreed budget require an
explicit review.

**PERF-8** measures publication, WAL, network, and subscriber apply cost for
bounded unique identity indexes versus `REPLICA IDENTITY FULL`. Results become
part of operator guidance for unbounded and keyless schemas.

**PERF-9** runs MERGE and exact-recheck workloads with indexes larger than
`shared_buffers`, ordered wide values, random wide values, and long common
prefixes. It confirms the selected probe improves candidate lookup without
making correctness depend on digest behavior.

## Scalability

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| SCAL-7 | Rebuild large dependency graphs | M | P0 |
| SCAL-8 | Stress multi-consumer buffers | M | P1 |
| SCAL-9 | Test wide keyless workloads | M | P1 |

**SCAL-7** recreates deep and branching stream-table DAGs in dependency order,
checking bounded catalog work, scheduler pause behavior, cleanup, and fresh
initial-refresh capacity.

**SCAL-8** exercises one source buffer with many bounded and unbounded
consumers, concurrent refreshes, cleanup, and retries. Consumer index strategy
must not alter shared source identity bytes or cause cross-consumer state leaks.

**SCAL-9** tests large keyless rows, duplicate logical outputs, nested values,
and high change volume under the configured memory and disk budgets. The system
must report resource failures before data correctness is compromised.

## Ease of Use

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| UX-7 | Ship the recreation runbook | M | P0 |
| UX-8 | Provide consumer migration guidance | S | P0 |
| UX-9 | Document identity data handling | S | P0 |

**UX-7** provides a copyable operator sequence for pause, export, reverse-order
drop, V1 cleanup, binary installation and restart, dependency-order recreation,
initial refresh, consumer resnapshot, and resume. It states precisely which
source data is preserved and which writes must be paused.

**UX-8** gives publication, outbox, DuckLake, dbt, logical-replication, and
user-SQL examples for changing numeric storage to `BYTEA`, updating schemas, and
resnapshotting. It calls out that old numeric IDs cannot be cast into V2
identities.

**UX-9** warns that opaque does not mean secret. Documentation covers explicit
privileges, logs and support bundles, pass-through identities, keyless rows,
short fingerprints, dump handling, and source-column data classification.

## Test Coverage

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| TEST-8 | Build the cross-mode matrix | L | P0 |
| TEST-9 | Test destructive-window recovery | M | P0 |
| TEST-10 | Verify consumer resnapshot paths | M | P0 |
| TEST-11 | Enforce release artifacts | M | P0 |

**TEST-8** runs the full source-history matrix through trigger and WAL capture,
FULL, differential, and IMMEDIATE refresh, including all supported DVM operator
families, source-key changes, keyless duplicates, enum DDL, non-deterministic
collation rejection, every accepted/rejected source-index form, interval's
128-bit value, unique-unbounded `IMMEDIATE` rejection, and probe-shaped
collisions.

**TEST-9** injects failures before and after pause, export, drop, cleanup,
install, restart, recreation, initial refresh, and resume. It verifies source
preservation, idempotent cleanup, no mixed graph, and actionable repair state.

**TEST-10** uses representative publications, outbox consumers, DuckLake sinks,
and dbt models to check schema migration, old-ID rejection, resnapshot ordering,
and the documented write pause. No external consumer is allowed to infer a
numeric identity from V2 bytes. It verifies that the inventory and resnapshot
acknowledgment are recorded before V1 teardown.

**TEST-11** makes vectors, support matrices, generated SQL, docs links,
benchmark baselines, package contents, and upgrade guards part of CI. The
release cannot pass with stale V1 references or missing recreation instructions;
the wire artifact, spec-to-code check, structural support matrix, exact DML
predicate checks, and identity eligibility rules are release inputs rather than
untracked design notes.

## Conflicts & Risks

- The accepted upgrade is breaking and destructive only to stream-table state,
  never to source tables. Operators must acknowledge external resnapshot work
  and the write window before dropping V1 state.
- V2 does not replay writes made during recreation. This is deliberate and must
  remain prominent in the runbook rather than being hidden behind an upgrade
  helper.
- Unique unbounded `IMMEDIATE` mode remains rejected until a separate locking
  RFC proves concurrency and capacity behavior; no release gate may weaken that
  rule.
- Identity bytes can contain reversible source values. Privacy controls and
  default grants are part of correctness of the shipped feature, not optional
  documentation polish.
- The release changes no unrelated DVM semantics. Failures in the row-identity
  gate block v0.88.0 because vectorized aggregates and delta planning depend on
  exact matching.

## Exit criteria

- [ ] All CDC and refresh paths agree with the exact oracle across the complete
      supported operator and identity matrix.
- [ ] The checked-in wire-format artifact, encoder, identity/probe version
      values, structural support matrix, interval 128-bit rule, and source-key
      eligibility cases agree and are independently tested.
- [ ] Encoder, probe, index, replication, and refresh benchmarks meet reviewed
      regression budgets.
- [ ] Direct-index uniqueness is derived from validated identity semantics, and
      unique unbounded `IMMEDIATE` identities are rejected before mutation.
- [ ] Non-destructive preflight and the pause/drop/recreate/resnapshot workflow
      are tested with injected failures and documented for operators; preflight
      is read-only and external-consumer acknowledgment precedes teardown.
- [ ] External consumers, permissions, diagnostics, and privacy warnings match
      the `BYTEA` contract.
- [ ] Older binaries reject V2 state, V2 binaries reject leftover V1 state, and
      no mixed graph can refresh.
- [ ] Full unit, integration, light/full E2E, benchmark, documentation, package,
      and release CI gates pass.