# v0.43.0 — Performance Tunability, Diagnostics, Refresh Intelligence & Delta Engine Refactor

> **Full technical details:** [v0.43.0.md-full.md](v0.43.0.md-full.md)

**Status: Planned** | **Scope: Large**

> Promote hardcoded deep-join and WAL thresholds to GUCs, improve GROUP_RESCAN
> performance for mutable aggregate arguments, add configurable cost-cache
> capacity, implement `explain_stream_table()` diagnostics, expose WAL CDC
> per-source status, add mandatory microbenchmarks in CI, and replace the
> wide-model change buffer with a native D+I decomposition schema.

---

## What is this?

The assessment found performance cliffs behind hardcoded thresholds that users
cannot tune, GROUP_RESCAN paths that lose O(delta) advantage for large groups,
fixed WAL batch sizes that cannot adapt to workload, and missing diagnostic
output explaining why a stream table chose a fallback strategy. v0.43.0 turns
these cliffs into tunable, observable, and benchmarked behavior.

---

## Deep-join and Part 3 thresholds as GUCs

Deep join behavior is governed by `PART3_MAX_SCAN_COUNT` and
`DEEP_JOIN_L0_SCAN_THRESHOLD` — hardcoded constants that can flip workloads
from fast differential execution to massive rescans. v0.43.0 promotes these to
documented GUCs with sane defaults and emits diagnostics when a query crosses
thresholds.

---

## GROUP_RESCAN improvement

Fallbacks for `SUM(CASE)`, MIN/MAX-like aggregates, and FULL JOIN aggregate
combinations can require rescanning affected groups. v0.43.0 implements the
DI-2-style old/new UPDATE split for mutable aggregate arguments, reducing
full-group rescans to targeted delta processing.

---

## WAL CDC tunability and diagnostics

WAL poll batch size and transition lag tolerance are constants. v0.43.0 exposes
them as GUCs and adds metrics for poll duration, rows decoded, lag bytes, and
fallback reason. Per-source WAL status includes `blocked_reason`, slot lag,
publication state, and last decoder error.

---

## `explain_stream_table()` diagnostics

Users cannot tell whether a slow refresh is expected, a regression, or a query
design issue. v0.43.0 adds `pgtrickle.explain_stream_table(name)` that reports
the chosen DVM plan, fallback reasons, thresholds crossed, estimated complexity,
and suggested tuning actions.

---

## D+I change buffer schema refactor (breaking)

The change buffer (`pgtrickle_changes.changes_<oid>`) currently mirrors every
source column twice (`new_col` / `old_col`) and stores UPDATE as a single wide
row. The DVM scan operator then decomposes UPDATE into DELETE+INSERT at read
time via a 600-line, 5-CTE UNION ALL pipeline. v0.43.0 replaces this with the
native D+I storage model: the CDC trigger emits `action='D'` (OLD values) and
`action='I'` (NEW values) as separate rows, and the delta scan collapses to a
single query with no UNION ALL or column-prefix aliasing. CB table width halves
for all source tables. The `changed_cols` WB-1 bitmask is retained at read
time: both the D-row and I-row of an UPDATE pair carry the same VARBIT, with
`NULL` reserved for genuine INSERT or DELETE rows. This keeps the CB table
fully decoupled from downstream view column selections. This is a breaking
schema migration — all existing change buffers are rebuilt and all CDC
triggers are regenerated during upgrade.

See issue [#728](https://github.com/grove/pg-trickle/issues/728) and
discussion [#726](https://github.com/grove/pg-trickle/discussions/726).

---

## Also in v0.43.0

- Cost-cache capacity configurable at preload time with hit/miss/collision metrics
- Pool-vs-spawn benchmark comparison in CI and docs
- Write-amplification benchmarks for UPDATE-heavy and keyless workloads
- Mandatory microbenchmarks for join codegen, placeholder resolution, and DAG rebuild

---

## Scope

v0.43.0 is a large release because it touches GUC infrastructure, DVM operator
logic, WAL decoder, diagnostics API, shared memory, CI benchmarks, and
documentation. The unifying theme is turning invisible performance behavior into
visible, tunable, and measured operations.

> **Source:** [plans/PLAN_OVERALL_ASSESSMENT_9.md](../plans/PLAN_OVERALL_ASSESSMENT_9.md)
> findings PERF-01 through PERF-07, FEAT-04, FEAT-05, CI-04.

---

*Previous: [v0.42.0 — Test Quality, Aggregate Correctness & Fuzz Automation](v0.42.0.md)*
*Next: [v0.44.0 — Security Hardening & Code Quality](v0.44.0.md)*

> **Breaking change note:** The D+I change buffer refactor (A44-10) requires an
> explicit upgrade step — change buffers are rebuilt and CDC triggers are
> regenerated. Treat this as a major version boundary for any deployment that
> uses direct SQL queries against `pgtrickle_changes.*`.