# v0.35.0 — Citus: World-Class Distributed Source CDC

> **Full technical details:** [v0.35.0.md-full.md](v0.35.0.md-full.md)

**Status: Planned** | **Scope: Large**

> The culmination of pg_trickle's Citus integration. v0.35.0 delivers
> end-to-end incremental view maintenance over Citus distributed tables —
> per-worker change capture, distributed stream table storage, cross-node
> coordination, and a full Citus-specific test suite.

---

## What is this?

With v0.32.0's naming and frontier foundations in place, v0.35.0 implements
the full Citus integration. There are three distributed table problems that
need solving:

1. **Change capture** — triggers don't propagate to Citus workers, so
   pg_trickle can't use its normal trigger-based CDC for distributed tables.
2. **Stream table storage** — a distributed stream table can't be updated
   with a standard `MERGE` statement; Citus imposes restrictions on
   cross-shard operations.
3. **Coordination** — PostgreSQL advisory locks and `LISTEN/NOTIFY` are
   node-local; a cluster needs a different mechanism for cross-node
   mutual exclusion.

---

## Change capture on distributed tables

For distributed tables (where data lives on Citus worker nodes), pg_trickle
switches from trigger-based CDC to **WAL-based CDC over per-worker logical
replication slots**. Each worker gets a publication and a replication slot
for the source table. The pg_trickle scheduler on the coordinator polls
each worker's slot, decodes the change stream, and writes the changes into
the coordinator's local change buffer — exactly the same buffer used by
trigger-based CDC.

This reuses the WAL decoder already in `src/wal_decoder.rs`; the only new
code is a connection path to call `pg_logical_slot_get_changes()` on a
remote node. Reference tables and local tables continue to use the existing
trigger path unchanged.

---

## Distributed stream table storage

When a stream table materializes a view over distributed sources, it can
store its result in three ways:

- **Local** (default for small outputs) — a regular coordinator-side table,
  same as today.
- **Reference** — a Citus reference table, replicated to every worker.
  Ideal for dimension tables read by many queries.
- **Distributed** — a Citus distributed table, sharded by the stream
  table's row identity. Required for outputs too large to fit on the
  coordinator.

For distributed stream tables, the standard `MERGE` statement is replaced
by a `DELETE … WHERE row_id IN (…)` followed by an
`INSERT … ON CONFLICT DO UPDATE` — the pattern Citus supports everywhere.
The result is identical; only the internal SQL changes.

---

## Cross-node coordination

Advisory locks that guard refresh scheduling are node-local. v0.35.0
introduces a lightweight catalog-based lock table
(`pgtrickle.pgt_st_locks`) that provides cross-node mutual exclusion
without an external coordinator. The table uses `INSERT … ON CONFLICT DO
NOTHING` for lock acquisition and timestamp-based lease expiry for
failure recovery. Advisory locks remain the fast-path for the common
single-node case.

---

## Observability

A new `pgtrickle.citus_status` view surfaces the health of all
distributed source connections: which workers are reachable, the
replication lag per worker per source, slot positions, and any workers
in a failed state. This integrates with the existing Prometheus metrics
endpoint.

---

## Test suite

v0.35.0 ships a dedicated Citus E2E test harness:
`tests/e2e_citus_tests.rs` backed by a docker-compose environment
(one coordinator + two workers) and `tests/Dockerfile.e2e-citus`. The
test matrix covers 10 scenarios including local, reference, and
distributed sources; mixed CDC backends; outbox integration; DDL
propagation; and worker restarts.

The non-Citus benchmark suite shows zero regression — all Citus code
paths are guarded by the detection check introduced in v0.32.0 and
compile out entirely when Citus is absent.

---

## Scope

v0.35.0 is a large release. Phase 3 (per-worker slot CDC) is the deepest
new code; it reuses the WAL decoder infrastructure but adds remote
connection management. Phase 6 (test suite) requires building and
maintaining a multi-container Citus CI environment.

---

*Previous: [v0.32.0 — Citus: Stable Naming & Per-Source Frontier Foundation](v0.32.0.md)*
*Next: [v1.0.0 — Stable Release](v1.0.0.md-full.md)*