> **Plain-language companion:** [v0.37.0.md](v0.37.0.md)

## v0.37.0 — Scheduler Modularisation, pgVectorMV & OpenTelemetry

**Status: Planned.** Derived from [plans/PLAN_OVERALL_ASSESSMENT_7.md](plans/PLAN_OVERALL_ASSESSMENT_7.md) §4, §7, §9, §13; A15, A16, A43; F4, F10.

> **Release Theme**
> v0.37.0 completes the structural cleanup arc started in v0.36.0 by splitting
> the two remaining large files (`src/scheduler.rs` at 7,756 LOC and
> `src/refresh/merge.rs` at 3,472 LOC), adds pgVectorMV for AI/RAG incremental
> vector aggregation, threads OpenTelemetry traces through the refresh
> pipeline, and validates pg_partman + TimescaleDB compatibility.

---

### Architecture

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| A15 | Split `src/scheduler.rs` (7,756 LOC) into sub-modules | L | P1 |
| A16 | Split `src/refresh/merge.rs` (3,472 LOC) into sub-modules | M | P2 |

**A15 — Scheduler split.**
`src/scheduler.rs` carries: dispatch loop, tier scheduler, predictive cost
integration, parallel pool, `pgt_st_locks` Citus lease loop, DAG rebuild
trigger, and CDC backpressure. Split into:

- `src/scheduler/mod.rs` — main dispatch loop and `SchedulerState`
- `src/scheduler/tier.rs` — SLA tier assignment, deadline tracking, tier up/down
- `src/scheduler/cost.rs` — predictive cost model (`CostModel`, regression)
- `src/scheduler/pool.rs` — parallel refresh worker pool, `PoolWorker`, `WorkerHandle`
- `src/scheduler/citus.rs` — Citus `pgt_st_locks` lease acquisition loop

The split must be purely mechanical — no logic changes. All existing tests
must pass without modification. Each sub-module has its own `#[cfg(test)]
mod tests` block moved from `src/scheduler.rs`.

> **Implementation order:** A15 and A16 must land as the **first two commits
> of the v0.37.0 development cycle**, before any F4 or F10 work begins.
> Any concurrent feature work touching `scheduler.rs` or `merge.rs` during
> the split creates merge-conflict risk with 7,768 LOC in motion.

**A16 — Merge split.**
`src/refresh/merge.rs` is 3,472 LOC of MERGE-path codegen. Split along
PostgreSQL operator boundaries:

- `src/refresh/merge/delete.rs` — DELETE path codegen
- `src/refresh/merge/insert.rs` — INSERT path codegen  
- `src/refresh/merge/update.rs` — UPDATE path codegen
- `src/refresh/merge/conflict.rs` — conflict predicate generator
- `src/refresh/merge/columns.rs` — column-list builder

No behaviour change. Purely a maintainability improvement.

---

### Features

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| F4  | pgVectorMV: `vector_avg` incremental aggregate operator | L | P1 |
| F10 | OpenTelemetry W3C Trace Context propagation | M | P1 |

**F4 — pgVectorMV.**
AI and RAG pipelines maintain document-cluster centroids and IVF index
structures over pgvector embeddings. Recomputing aggregated vectors on a
schedule from scratch is expensive. Add vector-aware algebraic aggregate
operators to the DVM engine:

- `vector_avg` — element-wise mean, using running `(sum, count)` state with
  Welford's approach for numerical stability. Tested with HNSW and IVFFlat
  indexes on the downstream stream table.
- `vector_sum` — element-wise sum, for pre-averaging or custom normalization.
- Stores partial state in the aggregate reducer; derive the result on read.
- Compatible with pgvector 0.7+ `vector` type; fully differential.
- Gate behind `pg_trickle.enable_vector_agg = on` GUC (default `off`, requires
  pgvector extension present).

Also add:
- **Distance operator classifier:** The DVM validator now explicitly classifies
  pgvector distance operators (`<->`, `<=>`, `<#>`, `<+>`) as "FULL-fallback
  safe" — any stream table using these operators in WHERE/ORDER BY is
  documented to fall back to FULL refresh with an `INFO` log explaining why.
  (No differentiation rule exists for non-monotone, non-linear operators; this
  codifies the behaviour as a stable contract.)
- **Integration test:** `tests/e2e_pgvector_tests.rs` with user-taste centroid
  example: document table + embedding column, stream table computing
  `avg(embedding)` per user, HNSW index on the centroid stream table, assert
  correctness vs FULL refresh.
- **Cookbook:** `docs/tutorials/PGVECTOR_EMBEDDING_PIPELINES.md` — "Maintaining
  Centroids with pgVectorMV" with copy-paste examples for RAG.

**F10 — OpenTelemetry trace propagation.**
Add W3C Trace Context (`traceparent`, `tracestate`) propagation through the
refresh pipeline:

1. Application sets `SET pg_trickle.trace_id = 'traceparent: ...'` in the
   session before the DML that triggers the CDC capture.
2. pg_trickle reads the context from the session GUC at CDC capture time and
   stores it in the change-buffer row's `__pgt_trace_context` column
   (nullable, opt-in).
3. At refresh time, the orchestrator reads the trace context from the earliest
   change-buffer row in the batch and opens a child span for each refresh phase:
   `pgtrickle.cdc_drain`, `pgtrickle.dvm_plan`, `pgtrickle.merge_apply`,
   `pgtrickle.notify_emit`.
4. Spans are exported via OTLP/gRPC to the endpoint configured in
   `pg_trickle.otel_endpoint` GUC (default empty — disabled).

Schema change: optional `__pgt_trace_context TEXT` column in
`pgtrickle_changes.changes_<stable_name>`. Only added when
`pg_trickle.enable_trace_propagation = on` (default `off`).

---

### Test Coverage

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| A43 | pg_partman + TimescaleDB hypertable integration tests | M | P2 |

**A43 — Partition ecosystem integration tests.**
`src/cdc.rs` handles declarative partitioning but has not been verified
against pg_partman's runtime partition creation or TimescaleDB hypertables.
Add `tests/e2e_partman_tests.rs`:

- pg_partman `run_maintenance()` creates new monthly partition while a stream
  table is refreshing; assert no data loss
- TimescaleDB hypertable as source; assert chunk boundaries are handled as
  regular partition attach/detach events
- Upgrade test: stream table on pg_partman source survives `0.36.0 → 0.37.0`

Requires a Testcontainers image with pg_partman and TimescaleDB installed.
Gate behind a `partman-tests.yml` workflow running daily and on manual
dispatch.

> **Deferral note (assessment 2026-04-27):** A43 is a new CI workflow
> plus a multi-extension Docker image. It is the lowest-risk item in this
> release and may be deferred to v0.38.0 if capacity is constrained by the
> A15/A16 split or F4 pgvector CI setup work.

---

### Conflicts & Risks

- **A15** (scheduler split) is purely mechanical but touches 7,756 LOC.
  Approach: create the sub-module files, `pub use` re-export everything from
  `mod.rs`, compile + test, then incrementally move code. Land in a dedicated
  commit with `[no-behaviour-change]` in the message.
- **F4** (pgVectorMV) requires pgvector to be present in the CI image.
  Add pgvector to `tests/Dockerfile.e2e` and gate the tests behind
  `#[cfg(feature = "pgvector")]`. This is a **prerequisite for all v0.38–v0.40
  vector work** — treat adding pgvector to the E2E image as a tracked issue
  that must close before v0.37.0 exits.
- **F10** (OpenTelemetry) must not add latency to the critical refresh path
  when `enable_trace_propagation = off`. The implementation must be
  branch-free for the disabled case.
- **Dependency-chain risk:** v0.38.0, v0.39.0, and v0.40.0 all depend on F4
  landing correctly. If pgVectorMV slips, the entire pgvector integration arc
  slips. F4 is a hard gate for starting v0.38.0 work.

### Exit Criteria

- [ ] A15: `src/scheduler.rs` replaced by five sub-modules; all existing tests pass without change
- [ ] A16: `src/refresh/merge.rs` replaced by five sub-modules; all existing tests pass without change
- [ ] F4: `vector_avg` + `vector_sum` aggregates maintain correct centroids vs FULL refresh over 10k-iteration proptest; distance operators explicitly classified as FULL-fallback safe; pgvector HNSW index refreshes correctly on stream-table delta; cookbook published
- [ ] F10: OTLP spans visible in test-mode Jaeger for a complete refresh cycle; `traceparent` correctly propagated from session GUC through CDC → DVM → MERGE → NOTIFY
- [ ] A43: pg_partman + TimescaleDB integration tests pass in `partman-tests.yml` *(may defer to v0.38.0)*
- [ ] `tests/e2e_pgvector_tests.rs` passes (user-taste centroid example)
- [ ] pgvector added to `tests/Dockerfile.e2e`; `#[cfg(feature = "pgvector")]` gate wired in CI
- [ ] Minimal Criterion benchmark for `vector_avg` aggregate reducer added to `benches/` (microseconds/vector baseline for v0.38–v0.40 regression gate)
- [ ] Extension upgrade path tested (`0.36.0 → 0.37.0`)
- [ ] `just check-version-sync` passes

---
