# v0.88.0 — Vectorized Aggregates & Delta Planning

> **Status:** Planned
> **Scope:** Large
> **User promise:** *"One PostgreSQL instance can handle a lot."*
> **Blocked by:** [v0.87.6](v0.87.6.md)
> **Renumbered and split:** previously the first half of v0.84.0 "Fast
> Incremental Engine". Incremental window functions moved to
> [v0.89.0](v0.89.0.md); parallel delta fan-out moved past 1.0.

## Theme

Make the delta path faster without changing what users see: a vectorized
execution path for pure-aggregate stream tables, and cost-based operator
ordering inside delta queries so intermediate results stay small.

The v0.87.1 through v0.87.6 correctness program lands first. Its exact oracle,
replay corpus, semantic coverage, schema contracts, snapshot plans, and release
gate protect each physical rewrite in this release.

This is one of two places where deep complexity is justified, because all of it
is **inside the engine**. Users get the same extension, the same PostgreSQL, the
same SQL — just faster. Nothing here changes deployment, topology or the API.

## Items

### ENG-1: `DiffContext` Decomposition (first)

Split the 15-field `DiffContext` struct into focused sub-contexts so the engine
work below lands in code that can be reasoned about:

- `CdcContext` — frontier LSNs, source OIDs, CDC columns, key columns
- `CacheContext` — CTE delta cache, scan-pushed predicates, bypass tables
- `OptimizationContext` — depth tracking, CTE counters, max limits

The top-level `DiffContext` composes these via delegation, so no call sites
change semantics. This lands **before** the items below, which each need only a
subset of the context.

### MT-8: Vectorized Aggregate Path

For pure-aggregate STs (GROUP BY with no joins), process deltas using columnar
batches:

1. Read change buffer rows into columnar batches (1024-row pages)
2. Vectorized hash aggregation: group-by keys → aggregate accumulators
3. SIMD-friendly SUM/COUNT/MIN/MAX operations on batch columns
4. Emit only changed groups as delta output
5. Apply via batched MERGE (INSERT/UPDATE per group)

Benefits:
- Large throughput gain for SUM/COUNT/AVG aggregates (auto-vectorization)
- Cache-friendly memory access patterns (column-at-a-time processing)
- Reduced SPI overhead (bulk operations instead of per-row)

Implementation:
- New `VectorizedAggregateOperator` in `src/dvm/operators/vectorized_agg.rs`
- Activated when OpTree is a single Aggregate node over a Scan (no joins)
- Falls back to standard SQL-based path for complex trees

**Dependency decision required first.** The original plan reached for
`arrow-array` and `arrow-compute`. v0.76.0 deliberately removed
`arrow-array`, `arrow-schema`, `parquet`, `object_store` and `bytes` to cut
compile time and attack surface. Re-adding two of them needs an ADR that either
reverses that decision explicitly, with the compile-time and audit-surface cost
measured, or selects a narrower implementation — plain `Vec<T>` columns over
PostgreSQL datums are enough for SUM/COUNT/MIN/MAX and add no dependency at
all. **The ADR is a gating deliverable of this release**, and the default
answer is the dependency-free one until Arrow is shown to be worth it.

### LT-9: Cost-Based Operator Scheduling

Reorder operators within a delta query's OpTree based on estimated cardinality
to minimize intermediate result sizes:

1. **Selectivity estimation:** Use PostgreSQL's `pg_statistic` to estimate
   filter selectivity and join cardinality
2. **Operator reordering rules:**
   - Push high-selectivity filters before joins (existing P2-7, extended)
   - Order join inputs by ascending estimated cardinality (smaller relation
     as build side)
   - Place DISTINCT before expensive projections
3. **Cost model:** Estimate total intermediate tuples for each valid ordering;
   choose minimum-cost plan
4. **Safety:** Only reorder when semantically equivalent (respects NULL
   handling, outer join placement constraints)

New diagnostic: `EXPLAIN (FORMAT JSON) SELECT pgtrickle.explain_delta_plan(pgt_id)`
shows the estimated cost and chosen operator order.

### PERF-O4: Parallel Delta Computation Fan-Out — **moved past v1.0**

The original plan was to "spawn parallel SPI connections (one per branch)" for
multi-source joins. SPI is not a connection pool: it executes inside the calling
backend's transaction and is neither thread-safe nor re-entrant across threads,
so this cannot be implemented as written. A real version needs either
PostgreSQL parallel workers driving the delta query (which the planner already
does inside a single delta statement) or the background-worker/`dblink` fan-out
that belongs with the decoupled-compute work.

It is therefore recorded in [v1.7.0](v1.7.0.md-full.md) rather than kept in the
pre-1.0 path with an implementation strategy that does not exist. Nothing else
in this release depends on it.

## Exit criteria

- [ ] `DiffContext` decomposed first; no behavioural change in the E2E suite
- [ ] ADR published deciding the vectorized-aggregate dependency question, and
      the v0.76.0 dependency-removal rationale explicitly addressed
- [ ] Vectorized aggregate path active for pure-aggregate stream tables; ≥5×
      throughput on the aggregate benchmark versus v0.87.6, with identical
      results verified against the FULL oracle
- [ ] Cost-based operator scheduling on by default; `explain_delta_plan()` shows
      the chosen order; no plan regression on the TPC-H suite
- [ ] Both paths validated by the DVM delta-invariant validator from v0.77.0