# v0.85.0 — Scheduler and Resource Resilience Gate

> **Status:** Planned
> **Scope:** Large
> **Releases after:** [v0.84.0](v0.84.0.md) — but shares no subsystem with
> [v0.82.0](v0.82.0.md)/[v0.83.0](v0.83.0.md) and may be developed in parallel
> **Driven by:** August 2026 pre-scaling hardening review
> **Renumbered:** previously planned as v0.81.4. It narrows or replaces
> documented self-healing behaviour, changes drain semantics, and imposes new
> hard maxima on API inputs — minor-release changes, not patch-release ones.

## Theme

Make the current in-process scheduler predictable under concurrency, multiple
databases, blocked queries, crashes, large catalogs, and adversarial input. This
is the final single-node resilience gate before the product arc begins in
v0.86.0.

## Non-Goals

- No external-worker protocol or adaptive external pool.
- No new monitoring backend.
- No scheduler feature expansion beyond making existing contracts reliable.

## Items

### OPS-81-1: Authoritative Worker Limits

Use one effective cluster limit, including PostgreSQL's
`max_parallel_workers`, for quota calculation and token acquisition. Make token
reservation, worker registration, exit, and reconciliation race-free by
tracking reserved/spawning and live workers separately or serializing state
changes. A sampled reconciliation must never erase or invent capacity.

### OPS-81-2: Database-Scoped Shared State

Key pause entries, scheduler health, and watermark baselines by database OID as
well as local `pgt_id`. Report bounded-capacity failures instead of returning
success when a pause entry cannot be stored. Waiting for selected nodes to pause
must not wait for unrelated work in another database.

### OPS-81-3: Persistent Drain Semantics

Keep dispatch disabled after `drain()` completes until an explicit resume or
documented restart boundary. `is_drained()` must distinguish no request,
draining, and drained, and must verify relevant workers rather than comparing
epochs alone. Maintenance cannot race a new job immediately after drain returns.

### OPS-81-4: Enforceable Refresh Deadlines

Add bounded, configurable lock and statement deadlines for scheduled refreshes
and apply them with transaction-local settings. Timeout outcomes must be typed,
retryable where appropriate, visible in health/history, and guaranteed to
release worker tokens and unblock drain.

### OPS-81-5: Complete Self-Healing and Error Accounting

Implement the documented temporary memory reduction, lock backoff, and reset
after successful cycles, or narrow the public contract to behavior that exists.
Inline and dynamic workers must share one persistent error-accounting,
remediation, and suspension path; in-memory dispatcher retries may not mask
repeated catalog-visible failure.

### OPS-81-6: Bounded Queue and Catalog Maintenance

Make queue-depth gauges reflect currently queued work by decrementing on claim,
cancel, spawn failure, and recovery. Periodically reconcile against catalog
state. Prune terminal jobs during normal uptime and limit history/cleanup work
per tick so a large retention backlog cannot monopolize dispatch.

### OPS-81-7: Scheduler-Safe Metrics and Alerts

Collect metrics only for an actual scrape rather than on every scheduler tick,
honor configured request deadlines, and prevent slow clients from delaying
dispatch. Standardize and document the bind address.

Build oversized alerts as valid structured JSON with character-boundary
truncation. Replace modulo-window reminders with last-emitted timestamps so
fast ticks cannot duplicate alerts and slow ticks cannot miss them.

### OPS-81-8: Resource Ceilings and Adversarial Assurance

Set conservative finite defaults and hard maxima for parse complexity, bulk
cardinality, pause arrays, and blocking control timeouts. Cancellation or client
disconnect must not strand scheduler state.

Make fuzz/property workflow names match reality: enumerate one target set for
smoke and nightly runs, exercise real SQL-facing parser/rewrite and CDC decode
entry points, assert property case counts, and detect variable-built dynamic SQL
in mandatory linting.

## Release Gate

v0.85.0 will not ship until all of the following pass:

- Multi-database stress with delayed worker startup and forced exits never
  exceeds configured worker limits and always recovers capacity.
- Identical `pgt_id` values in two databases have independent pause, scheduler
  health, and watermark state.
- Drain remains active across several scheduler intervals and resumes only via
  the documented action.
- Blocked locks and overlong refreshes terminate at configured deadlines,
  release tokens, record typed outcomes, and permit drain.
- OOM and lock-timeout injection proves remediation, reset, and eventual
  suspension in serial and parallel modes.
- Queue depth returns to zero and expired job/history cleanup progresses under
  continuous refresh load without unbounded tick latency.
- Idle metrics cause zero collection queries; slow scrapes stay within their
  budget; arbitrary Unicode alerts remain valid JSON below PostgreSQL limits.
- Oversized SQL/JSON/control inputs fail before expensive work, and every fuzz
  target runs in both smoke and nightly workflows.

The release requires a sustained multi-database soak with no leaked tokens,
stuck drain state, unbounded catalog growth, or unexplained refresh mismatch.