> **Plain-language companion:** [v0.36.0.md](v0.36.0.md)

## v0.36.0 — Structural Hardening, Performance & Temporal IVM

**Status: Planned.** Derived from [plans/PLAN_OVERALL_ASSESSMENT_7.md](plans/PLAN_OVERALL_ASSESSMENT_7.md) §4, §5, §7, §8, §9, §10, §11, §12, §13; A09, A12, A14, A17–A18, A20, A25–A26, A29, A31, A35–A36, A40; F2, F5, F11–F13.

> **Release Theme**
> v0.36.0 closes the structural and performance gaps that have accumulated
> since the Citus arc. The L0 shared-memory dshash template cache is finally
> constructed (wired-but-empty since v0.31.0), WAL slot backpressure is
> enforced, `src/api/mod.rs` is split into focused modules, structured JSON
> logging arrives for OpenTelemetry/Loki integration, and the `RowIdSchema`
> type formalises cross-operator row-id compatibility — addressing the root
> cause of the EC-01 residual issue at the architectural level.
> Feature-wise: Temporal IVM (SCD Type 2, `AS OF TIMESTAMP`), columnar storage
> backend, online schema evolution, `CREATE STREAM TABLE` SQL syntax,
> pluggable relay sinks, and column lineage.

---

### Performance

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| A09 | Construct L0 dshash shared-memory template cache | L | P0 |
| A12 | WAL decoder: `ENFORCE_BACKPRESSURE` mode + slot-lag cap | M | P0 |

**A09 — L0 dshash template cache.**
`src/shmem.rs:680-710` wires `L0_POPULATED_VERSION: PgAtomic<AtomicU64>` and
the cross-backend invalidation signal, but the actual dshash table holding
compiled delta SQL templates is never constructed. For connection-pooler
workloads where every backend is short-lived the L0 win is ~45 ms cold-start
savings per backend. Construct the dshash table in `pg_shmem_init!()`,
populate on first access, and honour the generation counter for invalidation.
Document the final `cold_start_ms` improvement in `docs/SCALING.md`.

**A12 — WAL backpressure enforcement.**
`src/wal_decoder.rs` does not enforce `pg_replication_slot.confirmed_flush_lsn`
advancement bounds. `slot_lag_warning_threshold_mb` and
`slot_lag_critical_threshold_mb` GUCs emit alerts but do not throttle.
At 100 MB/s WAL with a stuck refresh, disk fills in minutes.
Add an `ENFORCE_BACKPRESSURE` mode (GUC, default `off`) that pauses CDC
trigger writes when the slot lag exceeds `slot_lag_critical_threshold_mb`,
then resumes when it falls below 50% of the threshold. Document the trade-off
(change-buffer growth vs. slot retention) in `docs/SCALING.md`.

---

### Architecture

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| A14 | Split `src/api/mod.rs` (6,322 LOC) into focused sub-modules | M | P1 |
| A17 | Typed DDL event payload (replace string tag matching) | S | P1 |
| A18 | `RowIdSchema` type: every operator declares its row-id scheme | L | P1 |

**A14 — Split `src/api/mod.rs`.**
`src/api/mod.rs` at 6,322 LOC carries lifecycle, refresh, fuse control,
status reporting, and all ~80 `#[pg_extern]`-decorated SQL functions.
Split into:
- `src/api/lifecycle.rs` — `create_stream_table`, `alter_stream_table`, `drop_stream_table`
- `src/api/refresh.rs` — `refresh_stream_table` and refresh mode helpers
- `src/api/fuse.rs` — circuit-breaker and fuse control functions
- `src/api/status.rs` — `pgt_status`, `pgt_stream_table_status`, diagnostics

No behaviour change. Follow the pattern established by the `src/refresh/`
sub-module split in v0.26.0.

**A17 — Typed DDL event payload.**
`src/hooks.rs:130-176` matches on string tags (`"ALTER TABLE"`, etc.) from the
PostgreSQL event trigger payload. A wording change in a PostgreSQL minor release
would break classification silently. Add a `DdlCommandKind` enum populated via
`pg_event_trigger_ddl_commands()` introspection; retire the string-match path.

**A18 — `RowIdSchema` across operators.**
`src/hash.rs` is an excellent xxh3 streaming hasher, but each operator
constructs its own row-id hash from a different tuple — this is the root
architectural cause of EC-01. Define a `RowIdSchema` type that every operator
declares as part of its signature. Add a plan-time verifier that asserts
cross-operator compatibility before execution begins. This makes future EC-01-class
bugs compile-time detectable.

---

### Observability

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| A20 | Structured JSON logging: `pg_trickle.log_format = text\|json` GUC | M | P1 |

**A20 — Structured logging.**
`pgrx::log!()` / `info!()` / `warning!()` emit unstructured text. Add a
`pg_trickle.log_format = text|json` GUC and a wrapper macro that emits
structured fields: `event`, `pgt_id`, `cycle_id`, `duration_ms`,
`refresh_reason`, `error_code`. JSON format targets OpenTelemetry / Loki
integration without lossy regex parsing.

---

### Ease of Use / Operational

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| A25 | Bulk `alter_stream_tables()` / `drop_stream_tables()` APIs | S | P2 |
| A35 | Drain mode: graceful scheduler quiesce before maintenance | M | P1 |
| A36 | Capacity sizing calculator in `docs/SCALING.md` | M | P2 |
| F5  | Online schema evolution (`ALTER STREAM TABLE EVOLVE`) | L | P1 |
| F11 | `CREATE STREAM TABLE x AS SELECT …` SQL syntax | L | P1 |
| F12 | Column lineage graph + `pgtrickle.stream_table_lineage()` | M | P2 |
| F13 | Pluggable sink architecture in the relay | M | P1 |
| CORR-1 | Temporal IVM: two-dimensional frontier (LSN, timestamp) | L | P0 |
| CORR-2 | Columnar: verify MERGE compatibility with columnar storage | M | P0 |
| UX-1 | `create_stream_table(…, temporal := true)` | M | P0 |
| UX-2 | `AS OF TIMESTAMP` query rewrite in DVM parser | L | P1 |
| UX-3 | `create_stream_table(…, storage_backend := 'columnar')` | M | P1 |
| UX-4 | Automatic `delete_insert` strategy for columnar backends | S | P1 |
| UX-5 | Temporal IVM tutorial + SCD-Type-2 worked example | M | P1 |
| UX-6 | Columnar backend setup guide (Citus + pg_mooncake) | S | P1 |

**A25 — Bulk alter/drop APIs.**
dbt deployments commonly need to flip 100+ STs to a new schedule at once;
today this requires 100 individual round-trips. Add
`pgtrickle.bulk_alter_stream_tables(names TEXT[], params JSONB)` and
`pgtrickle.bulk_drop_stream_tables(names TEXT[])`.

**A35 — Drain mode.**
Add `pgtrickle.drain(timeout_s INT DEFAULT 60)` that signals the scheduler
to stop accepting new cycles, waits for all in-flight refreshes to complete,
and confirms via `pgtrickle.is_drained() BOOLEAN`. Useful before `pg_upgrade`,
rolling restarts, or backup windows.

**A36 — Capacity sizing calculator.**
`docs/SCALING.md` has limits but no formula. Add a "sizing calculator" section
with a worked example: X stream tables × Y refreshes/min × Z CDC volume =
N MB/s WAL + M cores + K GB shared buffers. Derive from the benchmark data in
`tests/e2e_bench_tests.rs`.

**F5 — Online schema evolution.**
`ALTER QUERY` requires a full reinit today. Detect type-compatible column
additions in the `ALTER QUERY` diff and emit `ALTER TABLE` on the storage table,
then re-prepare templates without flushing data. Gate behind
`pg_trickle.online_schema_evolution = on` GUC (default `off`, opt-in).

**F11 — `CREATE STREAM TABLE` SQL syntax.**
Use PostgreSQL's `ProcessUtility_hook` to intercept `CREATE STREAM TABLE name
AS SELECT …` lexically and rewrite to
`pgtrickle.create_stream_table('name', $$SELECT …$$)`. Add `DROP STREAM TABLE`
and `ALTER STREAM TABLE` syntax similarly.

**F12 — Column lineage.**
During parsing, record a `column_lineage` JSON graph in `pgt_stream_tables`
mapping each ST output column to its source table + column. Expose via
`pgtrickle.stream_table_lineage(name TEXT)` returning `TABLE(output_col,
source_table, source_col)`.

**F13 — Pluggable relay sinks.**
Define a stable `Sink` trait + dynamic loading pattern in the relay; ship the
existing Kafka/NATS/SQS sinks as separate `pgtrickle-sink-*` crates. Third
parties can then add Snowflake, Iceberg, ClickHouse sinks without a relay
rebuild.

**CORR-1 — Temporal IVM two-dimensional frontier.**
Temporal IVM extends the frontier model from a one-dimensional LSN cursor to
a `(frontier_lsn, valid_from_ts)` pair. Each row carries
`__pgt_valid_from TIMESTAMPTZ` and optional `__pgt_valid_to TIMESTAMPTZ`. Rows
are never physically deleted; a "close" delta sets `valid_to`. Queries with
`AS OF TIMESTAMP $1` resolve to the materialised row version valid at that
timestamp. **Schema change:** new `temporal_mode BOOLEAN` column on
`pgtrickle.pgt_stream_tables`; new `__pgt_valid_from`/`__pgt_valid_to` columns
auto-added to storage when `temporal_mode = true`.

**CORR-2 — Columnar merge compatibility.**
Citus columnar and pg_mooncake use append-only storage. Verify that the
differential MERGE is compatible with append-only semantics via `delete_insert`
strategy; add `storage_backend` column to `pgtrickle.pgt_stream_tables` and
route merge codegen accordingly.

---

### Test Coverage

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| A26 | OOM injection + disk-full chaos tests | M | P1 |
| A29 | Aggregation/window/recursive-CTE fuzz targets | M | P2 |
| TEST-1 | Integration: temporal stream table `AS OF TIMESTAMP` round-trip | L | P0 |
| TEST-2 | Integration: SCD-Type-2 dimension pattern end-to-end | M | P1 |
| TEST-3 | Integration: columnar stream table MERGE parity with heap | M | P0 |
| TEST-4 | Integration: temporal + columnar combined (stretch, P2) | M | P2 |

**A26 — Chaos tests.**
`tests/e2e_concurrent_tests.rs` covers DDL races and refresh-vs-DROP.
Missing: OOM injection during refresh, disk-full simulation, SIGKILL of bgworker
mid-refresh. Add three new tests behind a `chaos` feature flag requiring
`LD_PRELOAD` of a small `write()` interceptor library.

**A29 — New fuzz targets.**
`fuzz/fuzz_targets/` has `parser_fuzz`, `cdc_fuzz`, `cron_fuzz`, `guc_fuzz`.
Add: `aggregate_fuzz`, `window_fuzz`, `recursive_cte_fuzz`,
`join_predicate_fuzz`, `envelope_fuzz` (inbox/outbox).

---

### Security

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| A31 | Sigstore + SBOM provenance attestation at release | M | P1 |

**A31 — Release provenance.**
Add `cosign` signing + CycloneDX/SPDX SBOM generation to the release workflow.
Publish attestations alongside Docker images and release archives. Required for
SLSA Level 2 compliance at v1.0.

---

### Conflicts & Risks

- **CORR-1** (temporal IVM) requires a non-trivial DVM engine extension
  (two-dimensional frontier). Spike before committing; do not commit until
  the spike proves the approach is sound.
- **CORR-2** depends on Citus columnar or pg_mooncake being present in CI.
  Gate columnar support behind `pg_trickle.columnar_backend` GUC (default
  `none`); the CI matrix should test at least one columnar provider.
- **A18** (`RowIdSchema`) touches every DVM operator. Land as a pure type
  annotation pass first with no behaviour change; then add the plan-time
  verifier separately.
- The combination of temporal mode and columnar storage is a P2 stretch goal;
  do not block the release on it.

### Exit Criteria

- [ ] A09: L0 dshash table constructed; cold-start penalty measured < 10 ms in `e2e_bench_tests.rs`
- [ ] A12: `ENFORCE_BACKPRESSURE` mode tested; slot lag stops growing when cap is reached
- [ ] A14: `src/api/mod.rs` replaced by four sub-modules; all existing tests pass
- [ ] A18: `RowIdSchema` declared on all operators; plan-time verifier rejects incompatible schemas
- [ ] A20: `pg_trickle.log_format = json` emits valid JSON with all required fields; tested in `e2e_monitoring_tests.rs`
- [ ] A31: `cosign` sign + SBOM generated in release workflow; attestation visible on GHCR
- [ ] CORR-1: Temporal round-trip test passes at all three history points
- [ ] CORR-2: Columnar MERGE E2E parity test passes against at least one columnar provider
- [ ] F11: `CREATE/DROP/ALTER STREAM TABLE` SQL syntax works end-to-end
- [ ] Extension upgrade path tested (`0.35.0 → 0.36.0`)
- [ ] `just check-version-sync` passes

---

