# v0.88.0 — Vectorized Aggregates & Delta Planning

> **Status:** Planned
> **Scope:** Large
> **User promise:** *"One PostgreSQL instance can handle a lot."*
> **Blocked by:** [v0.87.0](v0.87.0.md)
> **Renumbered and split:** previously the first half of v0.84.0 "Fast
> Incremental Engine". Incremental window functions moved to
> [v0.89.0](v0.89.0.md); parallel delta fan-out moved past 1.0.

## Theme

Make the delta path faster without changing what users see: a vectorized
execution path for pure-aggregate stream tables, and cost-based operator
ordering inside delta queries so intermediate results stay small.

This is one of two places where deep complexity is justified, because all of it
is **inside the engine**. Users get the same extension, the same PostgreSQL, the
same SQL — just faster. Nothing here changes deployment, topology or the API.

## Items

### ENG-1: `DiffContext` Decomposition (first)

Split the 15-field `DiffContext` struct into focused sub-contexts so the engine
work below lands in code that can be reasoned about:

- `CdcContext` — frontier LSNs, source OIDs, CDC columns, key columns
- `CacheContext` — CTE delta cache, scan-pushed predicates, bypass tables
- `OptimizationContext` — depth tracking, CTE counters, max limits

The top-level `DiffContext` composes these via delegation, so no call sites
change semantics. This lands **before** the items below, which each need only a
subset of the context.

### MT-8: Vectorized Aggregate Path

For pure-aggregate STs (GROUP BY with no joins), process deltas using columnar
batches:

1. Read change buffer rows into columnar batches (1024-row pages)
2. Vectorized hash aggregation: group-by keys → aggregate accumulators
3. SIMD-friendly SUM/COUNT/MIN/MAX operations on batch columns
4. Emit only changed groups as delta output
5. Apply via batched MERGE (INSERT/UPDATE per group)

Benefits:
- Large throughput gain for SUM/COUNT/AVG aggregates (auto-vectorization)
- Cache-friendly memory access patterns (column-at-a-time processing)
- Reduced SPI overhead (bulk operations instead of per-row)

Implementation:
- New `VectorizedAggregateOperator` in `src/dvm/operators/vectorized_agg.rs`
- Activated when OpTree is a single Aggregate node over a Scan (no joins)
- Falls back to standard SQL-based path for complex trees

**Dependency decision required first.** The original plan reached for
`arrow-array` and `arrow-compute`. v0.76.0 deliberately removed
`arrow-array`, `arrow-schema`, `parquet`, `object_store` and `bytes` to cut
compile time and attack surface. Re-adding two of them needs an ADR that either
reverses that decision explicitly, with the compile-time and audit-surface cost
measured, or selects a narrower implementation — plain `Vec<T>` columns over
PostgreSQL datums are enough for SUM/COUNT/MIN/MAX and add no dependency at
all. **The ADR is a gating deliverable of this release**, and the default
answer is the dependency-free one until Arrow is shown to be worth it.

### LT-9: Cost-Based Operator Scheduling

Reorder operators within a delta query's OpTree based on estimated cardinality
to minimize intermediate result sizes:

1. **Selectivity estimation:** Use PostgreSQL's `pg_statistic` to estimate
   filter selectivity and join cardinality
2. **Operator reordering rules:**
   - Push high-selectivity filters before joins (existing P2-7, extended)
   - Order join inputs by ascending estimated cardinality (smaller relation
     as build side)
   - Place DISTINCT before expensive projections
3. **Cost model:** Estimate total intermediate tuples for each valid ordering;
   choose minimum-cost plan
4. **Safety:** Only reorder when semantically equivalent (respects NULL
   handling, outer join placement constraints)

New diagnostic: `EXPLAIN (FORMAT JSON) SELECT pgtrickle.explain_delta_plan(pgt_id)`
shows the estimated cost and chosen operator order.

### PERF-O4: Parallel Delta Computation Fan-Out — **moved past v1.0**

The original plan was to "spawn parallel SPI connections (one per branch)" for
multi-source joins. SPI is not a connection pool: it executes inside the calling
backend's transaction and is neither thread-safe nor re-entrant across threads,
so this cannot be implemented as written. A real version needs either
PostgreSQL parallel workers driving the delta query (which the planner already
does inside a single delta statement) or the background-worker/`dblink` fan-out
that belongs with the decoupled-compute work.

It is therefore recorded in [v1.7.0](v1.7.0.md-full.md) rather than kept in the
pre-1.0 path with an implementation strategy that does not exist. Nothing else
in this release depends on it.

## Exit criteria

- [ ] `DiffContext` decomposed first; no behavioural change in the E2E suite
- [ ] ADR published deciding the vectorized-aggregate dependency question, and
      the v0.76.0 dependency-removal rationale explicitly addressed
- [ ] Vectorized aggregate path active for pure-aggregate stream tables; ≥5×
      throughput on the aggregate benchmark versus v0.87.0, with identical
      results verified against the FULL oracle
- [ ] Cost-based operator scheduling on by default; `explain_delta_plan()` shows
      the chosen order; no plan regression on the TPC-H suite
- [ ] Both paths validated by the DVM delta-invariant validator from v0.77.0