# v0.38.0 — Embedding Pipeline Infrastructure & ANN Maintenance

> **Full technical details:** [v0.38.0.md-full.md](v0.38.0.md-full.md)

**Status: Planned** | **Scope: Medium**

> Embedding pipeline hardening via post-refresh hooks, automatic IVFFlat
> re-clustering on drift, vector-aggregate status monitoring, AI/RAG
> Docker image, and pgvector + pgai ecosystem integration.

---

## What is this?

Following on from pgVectorMV in v0.37.0, v0.38.0 adds operational primitives
for production embedding pipelines: post-refresh actions (ANALYZE, REINDEX,
drift-based re-clustering), vector-aware monitoring views, and a one-command
Docker image with pg_trickle + pgvector + pgai pre-installed for RAG
developers.

---

## Embedding pipeline operational hardening

When a stream table refreshes, it is often necessary to run housekeeping tasks:
analyzing statistics, rebuilding indexes, or re-clustering ANN structures that
have drifted. v0.38.0 adds explicit scheduling primitives:

- `post_refresh_action` — per-stream-table option: `none`, `analyze`, `reindex`,
  or `reindex_if_drift`. Runs asynchronously in a lower-priority tier after the
  refresh commits.
- `reindex_drift_threshold` — fraction (0.0–1.0, default 0.10) of rows that
  must change since last reindex to trigger `reindex_if_drift`. The scheduler
  tracks "rows changed since last reindex" in the catalog.
- `pgtrickle.vector_status()` — new monitoring view: per-vector stream table,
  shows last refresh time, embedding lag, ANN index age, approximate drift %,
  and segment count (for HNSW). Integrated with the v0.20+ self-monitoring
  system.

---

## pgvector + pgai integration

- **Always-Fresh RAG cookbook:** `docs/tutorials/PGVECTOR_RAG_COOKBOOK.md` —
  end-to-end pattern: pgai embeddings → stream table maintaining denormalised
  corpus (documents + tags + permissions joined) → BM25 + HNSW indexes on the
  flat table → application hybrid search.
- **Docker image:** `pg_trickle + pgvector + pgai + pgml` Dockerfile variant
  for one-command RAG deployment.

---

## Also in v0.38.0

- Any deferred items from v0.37.0 that did not meet the exit criteria
- Citus integration tests for vector aggregates (shard-additive centroid
  aggregation) — required exit criterion; defer with tracked issue if Citus
  infra is unavailable, but do not silently skip
- `docs/CONFIGURATION.md` overhaul: auto-generated GUC reference covering
  all 111+ GUCs, plus the new per-table stream-table options

---

## Scope

v0.38.0 is a medium release. The post-refresh-action primitives are scheduler
options (no engine change); the monitoring view is purely observational;
ecosystem integration (cookbook, Docker image) is additive. The CONFIGURATION
docs overhaul is new but tractable.

> **Hard prerequisite:** v0.38.0 development must not start until F4
> (v0.37.0 `vector_avg`) has landed with `tests/e2e_pgvector_tests.rs` green.

---

*Previous: [v0.37.0 — Scheduler Modularisation, pgVectorMV & OpenTelemetry](v0.37.0.md)*
*Next: [v0.39.0 — Hybrid Search & Sparse Vector Aggregates](v0.39.0.md)*
