# v0.37.0 — Scheduler Modularisation, pgVectorMV & OpenTelemetry

> **Full technical details:** [v0.37.0.md-full.md](v0.37.0.md-full.md)

**Status: Planned** | **Scope: Medium**

> Internal restructuring that makes the scheduler safe to evolve, first-class
> AI/RAG support via incremental vector aggregation, distributed trace
> propagation, and extended ecosystem compatibility.

---

## What is this?

v0.37.0 focuses on three themes: breaking up the two largest remaining
monolithic files so they are safe to evolve, adding AI/RAG workload support
via incremental vector-average maintenance, and threading OpenTelemetry traces
through the refresh pipeline so operators can correlate stream-table latency
back to the HTTP request that triggered it.

---

## Scheduler and merge modularisation

`src/scheduler.rs` is 7,756 lines — the single largest file in the codebase —
and carries the dispatch loop, tier scheduler, predictive cost integration,
parallel pool, `pgt_st_locks` lease loop, DAG rebuild trigger, and CDC
backpressure all in one file. Any change to one concern touches the entire
file in diffs and review.

v0.37.0 splits it along natural concern boundaries:

- `src/scheduler/mod.rs` — dispatch loop and main tick
- `src/scheduler/tier.rs` — tier assignment, SLA deadline tracking
- `src/scheduler/cost.rs` — predictive cost model integration
- `src/scheduler/pool.rs` — parallel worker pool
- `src/scheduler/citus.rs` — Citus lease acquisition loop

`src/refresh/merge.rs` (3,472 LOC) gets the same treatment, split along
PostgreSQL operator boundaries (delete path, insert path, conflict predicate
generator, column-list builder).

No behaviour changes — purely a maintainability improvement that makes every
future feature change incremental and reviewable.

---

## pgVectorMV: incremental vector aggregation for AI/RAG

AI and RAG pipelines commonly maintain document-cluster centroids or
IVF index structures over embeddings. Today this means recomputing all
aggregated vectors on a schedule from scratch. pg_trickle can maintain
centroids *incrementally* using `vector_avg` and `vector_sum` algebraic
aggregates that apply element-wise arithmetic in the L2 norm domain.

v0.37.0 adds `vector_avg` and `vector_sum` aggregate operators to the DVM
engine that are compatible with pgvector 0.7+ HNSW and IVF indexes. A stream
table over a document-embeddings table can maintain a live cluster centroid
that is always within one refresh cycle of the latest embeddings, without
recomputing the full set. The DVM distance-operator classifier is also
codified as a stable contract: any `<->`, `<=>`, `<#>` operators in a stream
table are explicitly documented to fall back to FULL refresh.

---

## OpenTelemetry trace propagation

When an HTTP request writes a row that triggers a stream-table refresh, there
is currently no way to correlate the two in a distributed trace. v0.37.0 adds
W3C Trace Context propagation:

1. The application sets `pg_trickle.trace_id` (a session GUC) to the W3C
   `traceparent` value before writing.
2. pg_trickle reads the trace context and carries it through the refresh
   pipeline, emitting OTLP/gRPC spans for each phase (CDC capture, DVM plan,
   MERGE apply, notification emit).
3. The trace spans are visible in Jaeger, Tempo, or any OTLP-compatible backend.

This closes the last observability gap in the v7 overall assessment's
recommendations for production readiness.

---

## Also in v0.37.0

- pg_partman and TimescaleDB hypertable integration tests — verify that
  pg_trickle's declarative partitioning support works with pg_partman's
  runtime partition creation and TimescaleDB chunks *(deferrable to v0.38.0
  if capacity is constrained by the scheduler split or pgvector CI setup)*
- pgvector added to `tests/Dockerfile.e2e` — prerequisite CI infrastructure
  for the entire v0.37–v0.40 vector arc
- Minimal `vector_avg` Criterion benchmark baseline (microseconds/vector)
  for future regression gating
- Any deferred items from v0.36.0 that did not meet the exit criteria

---

## Scope

v0.37.0 is a medium release. The scheduler and merge splits are
pure restructuring; pgVectorMV and OpenTelemetry are additive features with
no impact on existing stream tables.

> **Implementation discipline:** A15 (scheduler split) and A16 (merge split)
> must land as the first two commits of this release before any feature work
> begins. F4 (pgVectorMV) is a hard gate for v0.38.0 — the entire v0.38–v0.40
> pgvector arc depends on it.

---

*Previous: [v0.36.0 — Structural Hardening, Performance & Temporal IVM](v0.36.0.md)*
*Next: [v0.38.0 — Relay Phase 2a+2b: Cloud & Analytics Backends](v0.38.0.md)*
