# v0.87.16 - Versioned Row Identity V2 Engine Integration

> **Status:** Planned
> **Scope:** 12 person-weeks
> **User promise:** *"The same exact row identity drives storage, capture, refresh, and matching."*
> **Blocked by:** [v0.87.15](v0.87.15.md)
> **Proposal:** [Exact, Versioned Row Identity V2](../proposals/PROPOSAL_VERSIONED_ROW_IDENTIFY_ENCODING.md)

## Theme

Move the V2 identity contract through every internal relation and maintenance
path. Stream-table storage, change buffers, temporary relations, trigger CDC,
WAL CDC, full refresh, differential refresh, joins, aggregates, and propagation
must carry the complete canonical `BYTEA` identity, while indexes use either a
bounded direct key or a bounded probe plus exact recheck. Version columns and
runtime guards make a mixed V1/V2 graph fail closed before it can apply a wrong
change.

This release changes the physical row-identity type and is intentionally a
breaking pre-1.0 release. It does not attempt to convert existing V1 stream
tables in place; the recreation workflow and external-consumer runbook land in
v0.87.17.

## Scope allocation

| Work package | Person-weeks |
|---|---:|
| Storage, catalog version state, and generated DDL | 2.00 |
| Direct indexes, boundedness, and expression probes | 2.00 |
| Trigger and WAL CDC producer integration | 1.75 |
| DVM producers, transport, and exact DML matching | 3.00 |
| Runtime compatibility and replication contracts | 1.25 |
| PostgreSQL, upgrade, benchmark, and review work | 2.00 |
| **Total** | **12.00** |

## Items

### ROWID-6: Replace identity storage with versioned `BYTEA`

Change every storage, delta, temporary, CDC, and cache relation carrying a row
identity to `__pgt_row_id BYTEA NOT NULL`. Add extension-wide
`row_identity_version = 2` and `row_probe_version = 1` state to stream-table and
change-buffer metadata, and update generated SQL, catalog inspection, cleanup,
and repair paths. Runtime readers compare these values before consuming rows,
including when a new shared library is installed before `ALTER EXTENSION ...
UPDATE`. Transport relations carry only the complete identity; no generated
probe column or parallel V1/V2 storage strategy is introduced.

**Dependencies:** v0.87.15; proposal sections 3, 9, and 10.
**Schema change:** Yes.

### ROWID-7: Add bounded direct and unbounded probe indexes

Compute the maximum encoded identity size conservatively from the resolved type
schema, typmods, collation, nesting, and running cluster `BLCKSZ`. Check
`BTMaxItemSize` together with varlena and index-tuple overhead at every create,
alter, and preflight boundary, and test every supported page-size build. Use a
direct B-tree on the complete `BYTEA` whenever the complete datum is proven to
fit; make it unique only when the validated identity schema proves one row per
identity, never merely because an index or probe exists. For unbounded schemas,
select and freeze the benchmarked probe prefix, implement immutable
parallel-safe `row_probe_v1(bytea)`, and create a non-unique expression index
with no heap probe column.

**Dependencies:** ROWID-2 through ROWID-5; running PostgreSQL page-size limits.
**Schema change:** Yes.

### ROWID-8: Route all CDC producers through V2 encoding

Update row-level trigger CDC and WAL CDC to emit the same full V2 identity bytes
from typed datums. A shared source buffer may feed multiple consumers, so no
consumer-specific encoding option or downstream preference lookup is allowed.
Capture version metadata atomically with the buffer row and reject a producer
configuration that cannot validate the source identity schema.

**Dependencies:** ROWID-3 and ROWID-6.
**Schema change:** Yes.

### ROWID-9: Update all DVM identity producers

Replace hard-coded `BIGINT` identity creation in full refresh, differential
refresh, IMMEDIATE mode, joins, aggregates, set operations, windows, keyless
rows, and synthetic singleton/dummy rows. Pass typed records to the single V2
encoder, preserve the existing `RowIdStrategy` and `RowIdSchema` field choices,
and frame derived child identities under their registered domains. Planning must
reject a unique unbounded identity for `IMMEDIATE` mode before any DML or
capture state is created; the non-unique expression index is not an authority for
uniqueness and this release does not add per-identity locking.

**Dependencies:** ROWID-3, ROWID-6, and the v0.87.14 DVM contracts.
**Schema change:** No.

### ROWID-10: Make matching exact for every index strategy

For bounded identities, use full `BYTEA` equality and the direct B-tree; use
`ON CONFLICT (__pgt_row_id)` only where the identity is genuinely unique. For
unbounded or keyless identities, use non-unique direct or expression indexes as
an accelerator and include complete row-ID equality in every UPDATE, DELETE,
MERGE, and counted-delete predicate. For an unbounded expression index, every
candidate predicate must contain both
`row_probe_v1(target.__pgt_row_id) = row_probe_v1(delta.__pgt_row_id)` and
`target.__pgt_row_id = delta.__pgt_row_id`; the counted-delete path must retain
the existing multiplicity accounting rather than deleting by probe alone. Never
use a probe or digest as proof of identity or as a uniqueness constraint.

**Dependencies:** ROWID-7 and ROWID-9.
**Schema change:** No.

### ROWID-11: Enforce V1/V2 and replication compatibility

Make runtime readers reject mismatched row-identity or probe versions before
consuming a row, including when a new shared library is installed before
`ALTER EXTENSION ... UPDATE` runs. Apply the V2 replica-identity rules:
bounded unique tables may use the direct unique identity index, while unbounded
and keyless tables use `REPLICA IDENTITY FULL`; publication column lists include
what PostgreSQL requires and never expose the expression probe as a schema
column. The version constants and expected relation/catalog shape are defined in
one compatibility contract so old binaries reject V2 state and V2 binaries
reject leftover V1 state without guessing.

**Dependencies:** ROWID-6, ROWID-7, and existing catalog/replication hooks.
**Schema change:** Yes.

## Correctness

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| CORR-4 | Unify all identity producers | L | P0 |
| CORR-5 | Recheck complete identities | M | P0 |
| CORR-6 | Preserve keyless multiplicity | M | P0 |
| CORR-7 | Reject mixed identity versions | M | P0 |

**CORR-4** runs trigger CDC, WAL CDC, FULL refresh, differential refresh,
IMMEDIATE mode, and every DVM operator through shared typed encoding. The same
logical fields must produce the same bytes regardless of the producer or
refresh mode.

**CORR-5** proves that direct-index and probe-index paths produce identical
results under insert, update, delete, collision-shaped values, and common
prefixes. A probe collision may increase candidates but must never change the
matched complete row identity.

**CORR-6** verifies that identical logical keyless rows still receive identical
identities and remain independently counted. Non-unique index paths and
counted-delete behavior must not collapse multiplicity.

**CORR-7** injects stale buffer rows, stale temporary relations, old catalog
versions, and mixed dependency graphs. Readers fail before mutation with a
version-specific diagnostic rather than guessing or replaying incompatible
state.

## Stability

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| STAB-4 | Make catalog changes atomic | M | P0 |
| STAB-5 | Fail closed before DML | M | P0 |
| STAB-6 | Preserve replica identity safety | M | P0 |

**STAB-4** applies identity and probe version metadata in the same transaction
as storage and buffer definition changes. Partial index creation, cleanup, or
catalog updates leave no relation that a later refresh could mistake for V2.

**STAB-5** validates type contracts, index readiness, version compatibility,
relation shape, and the unique/unbounded `IMMEDIATE` rule before reading or
mutating a delta. Errors include the stream table, buffer, expected versions,
actual versions, rejected identity property, and remediation.

**STAB-6** tests publication, UPDATE, DELETE, replica identity, and subscriber
apply behavior for bounded unique, bounded keyless, and unbounded tables. The
expression probe never becomes a replica identity or a published generated
column.

## Performance

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| PERF-4 | Freeze a measured probe size | M | P0 |
| PERF-5 | Compare physical lookup strategies | L | P0 |
| PERF-6 | Keep CDC identity overhead bounded | M | P1 |

**PERF-4** benchmarks 32, 64, 128, and 256-byte prefixes across short keys,
ordered wide keys, random wide keys, and long common-prefix identities. Select a
probe size only after checking the actual `BTMaxItemSize`, varlena overhead, and
index-tuple overhead for each supported `BLCKSZ`; the selected prefix and
XXH3-128 algorithm/seed become the immutable probe-version-1 contract.

**PERF-5** compares direct full-ID B-trees, expression probe indexes, generated
probe columns, and PostgreSQL hash indexes for applicable workloads. Measure
index size, buffer hits, WAL volume, cached and cold MERGE latency, and behavior
when the index exceeds `shared_buffers`; generated columns and hash indexes are
benchmarks, not required shipped strategies.

**PERF-6** measures typed encoding, trigger and WAL capture, change-buffer
storage, and refresh overhead against the V1 baseline. The common fixed-width
path must retain the encoder cache and scratch-buffer properties established in
v0.87.15.

## Scalability

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| SCAL-4 | Share one source identity safely | M | P0 |
| SCAL-5 | Scale unbounded identity matching | M | P0 |
| SCAL-6 | Bound wide-row maintenance memory | M | P1 |

**SCAL-4** feeds several downstream stream tables from one source buffer and
checks that each consumer applies its own boundedness and probe classification
without changing the source identity bytes.

**SCAL-5** exercises long and common-prefix identities with non-unique
expression indexes, exact rechecks, concurrent refreshes, and indexes larger
than memory. It records candidate amplification and confirms that performance
tradeoffs do not alter results.

**SCAL-6** runs keyless wide rows, nested supported values, large deltas, and
multi-consumer buffers under the existing memory budgets. Buffer cleanup and
DVM transport must not create a second full-ID representation or unbounded probe
state.

## Ease of Use

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| UX-4 | Explain direct versus probe indexing | S | P1 |
| UX-5 | Document replication consequences | S | P0 |
| UX-6 | Remove obsolete V1 assumptions | M | P0 |

**UX-4** reports whether a stream table uses a direct full-ID index or an
expression probe, the selected probe version, and the reason an identity was
classified as unbounded. It does not display complete identity bytes by default.

**UX-5** documents the replica-identity and publication behavior for each table
class, including the extra old-row cost of `REPLICA IDENTITY FULL` on unbounded
and keyless UPDATE/DELETE workloads.

**UX-6** updates generated SQL, SQL reference, diagnostics, examples, and
operator-facing names so users do not infer that `BIGINT`, a generated probe
column, or a probe uniqueness guarantee still exists.

## Test Coverage

| ID | Title | Effort | Priority |
|---|---|---:|:---:|
| TEST-4 | Run the producer matrix | L | P0 |
| TEST-5 | Test index equivalence | M | P0 |
| TEST-6 | Verify replication matrix | M | P0 |
| TEST-7 | Exercise version guards | M | P0 |

**TEST-4** covers trigger CDC, WAL CDC, full refresh, differential refresh,
IMMEDIATE mode, joins, aggregates, set operations, windows, and synthetic
identities with the shared vectors from v0.87.15.

**TEST-5** compares direct and probe-index plans against the exact schema oracle
for inserts, updates, deletes, common-prefix identities, forced probe
collisions, keyless duplicates, and zero-change refreshes. It asserts both
byte-identical identities across producer paths and the complete probe-plus-ID
predicate in generated UPDATE, DELETE, MERGE, and counted-delete SQL.

**TEST-6** uses PostgreSQL publication and subscriber fixtures for bounded
unique, bounded non-unique, unbounded, and keyless tables. It checks old-row
availability, column lists, subscriber `BYTEA` storage, and absence of a
published generated probe.

**TEST-7** tests old binaries, new binaries before extension update, mixed
buffers, stale temporary relations, and partial catalog metadata. All mismatch
cases fail before consuming or mutating state.

## Conflicts & Risks

- This release changes the physical type of `__pgt_row_id` and all dependent
  relations. Existing stream-table state is not converted; recreation belongs
  to v0.87.17.
- Unique unbounded identities remain unsupported for `IMMEDIATE` mode. The
  non-unique expression index cannot enforce uniqueness, and this release must
  not introduce per-identity locking as a workaround.
- Direct-index size classification is cluster-specific. A bound safe for one
  `BLCKSZ` must not be reused blindly on another build.
- Expression probes are not identity values, replica identities, or publication
  columns. Every correctness predicate must still compare complete `BYTEA` IDs.
- DVM engine changes require property tests and the v0.87.14 correctness gate;
  performance work cannot relax fail-closed behavior or durability.

## Exit criteria

- [ ] All identity-bearing relations use `BYTEA NOT NULL` and carry compatible
      version metadata where required.
- [ ] Identity version `2` and probe version `1` are enforced at runtime, with
  old/new binary mismatch tests before any row is consumed or mutated.
- [ ] Trigger CDC, WAL CDC, refresh, DVM operators, and propagation use one V2
      typed encoder.
- [ ] Direct and expression index paths are selected from conservative,
  cluster-specific `BLCKSZ`/`BTMaxItemSize` bounds and use exact full-ID
  matching; the unique-index rule is tied to validated identity semantics.
- [ ] Every unbounded UPDATE, DELETE, MERGE, and counted-delete predicate
  includes probe equality plus complete-ID equality, and unique unbounded
  `IMMEDIATE` identities are rejected during planning.
- [ ] Version mismatches fail before DML, and no V1/V2 graph can be consumed.
- [ ] Replication behavior is tested for every bounded, unbounded, and keyless
      table class.
- [ ] Unit, PostgreSQL, integration, benchmark, upgrade-guard, and full
      repository CI gates pass.