# v0.37.0 — Scheduler Modularisation, pgVectorMV & OpenTelemetry

> **Full technical details:** [v0.37.0.md-full.md](v0.37.0.md-full.md)

**Status: Planned** | **Scope: Medium**

> Internal restructuring that makes the scheduler safe to evolve, first-class
> AI/RAG support via incremental vector aggregation, distributed trace
> propagation, and extended ecosystem compatibility.

---

## What is this?

v0.37.0 focuses on three themes: breaking up the two largest remaining
monolithic files so they are safe to evolve, adding AI/RAG workload support
via incremental vector-average maintenance, and threading OpenTelemetry traces
through the refresh pipeline so operators can correlate stream-table latency
back to the HTTP request that triggered it.

---

## Scheduler and merge modularisation

`src/scheduler.rs` is 7,756 lines — the single largest file in the codebase —
and carries the dispatch loop, tier scheduler, predictive cost integration,
parallel pool, `pgt_st_locks` lease loop, DAG rebuild trigger, and CDC
backpressure all in one file. Any change to one concern touches the entire
file in diffs and review.

v0.37.0 splits it along natural concern boundaries:

- `src/scheduler/mod.rs` — dispatch loop and main tick
- `src/scheduler/tier.rs` — tier assignment, SLA deadline tracking
- `src/scheduler/cost.rs` — predictive cost model integration
- `src/scheduler/pool.rs` — parallel worker pool
- `src/scheduler/citus.rs` — Citus lease acquisition loop

`src/refresh/merge.rs` (3,472 LOC) gets the same treatment, split along
PostgreSQL operator boundaries (delete path, insert path, conflict predicate
generator, column-list builder).

No behaviour changes — purely a maintainability improvement that makes every
future feature change incremental and reviewable.

---

## pgVectorMV: incremental vector aggregation for AI/RAG

AI and RAG pipelines commonly maintain document-cluster centroids or
IVF index structures over embeddings. Today this means recomputing all
aggregated vectors on a schedule from scratch. pg_trickle can maintain
centroids *incrementally* using a `vector_avg` algebraic aggregate that
applies Welford's online algorithm in the L2 norm domain.

v0.37.0 adds a `vector_avg` aggregate operator to the DVM engine that is
compatible with pgvector 0.7+ HNSW and IVF indexes. A stream table over a
document-embeddings table can maintain a live cluster centroid that is always
within one refresh cycle of the latest embeddings, without recomputing the
full set.

---

## OpenTelemetry trace propagation

When an HTTP request writes a row that triggers a stream-table refresh, there
is currently no way to correlate the two in a distributed trace. v0.37.0 adds
W3C Trace Context propagation:

1. The application sets `pg_trickle.trace_id` (a session GUC) to the W3C
   `traceparent` value before writing.
2. pg_trickle reads the trace context and carries it through the refresh
   pipeline, emitting OTLP/gRPC spans for each phase (CDC capture, DVM plan,
   MERGE apply, notification emit).
3. The trace spans are visible in Jaeger, Tempo, or any OTLP-compatible backend.

This closes the last observability gap in the v7 overall assessment's
recommendations for production readiness.

---

## Also in v0.37.0

- pg_partman and TimescaleDB hypertable integration tests — verify that
  pg_trickle's declarative partitioning support works with pg_partman's
  runtime partition creation and TimescaleDB chunks
- Any deferred items from v0.36.0 that did not meet the exit criteria

---

## Scope

v0.37.0 is a medium release. The scheduler and merge splits are
pure restructuring; pgVectorMV and OpenTelemetry are additive features with
no impact on existing stream tables.

---

*Previous: [v0.36.0 — Structural Hardening, Performance & Temporal IVM](v0.36.0.md)*
*Next: [v1.0.0 — Stable Release](../roadmap/v1.0.0.md-full.md)*
