# v0.69.0 — DuckLake Sink Reliability & Security

> **Full details:** [v0.69.0.md-full.md](v0.69.0.md-full.md)

## What's New

v0.69.0 transforms the DuckLake sink from a best-effort post-refresh side effect
into a trustworthy, observable, and secure delivery mechanism. It resolves five
HIGH- and MEDIUM-severity findings from Assessment 13 that all cluster around
the DuckLake integration surface: sink delivery semantics, view registration
staleness, snapshot-ID race condition, unqualified schema resolution, and missing
observability.

### ARCH-002 / REL-001: DuckLake Sink Delivery State Machine (ARCH-002, REL-001)

The DuckLake sink now has durable delivery semantics. A new catalog table,
`pgtrickle.pgt_ducklake_sink_delivery`, tracks each write attempt with columns:
`delivery_id`, `stream_table_id`, `refresh_id`, `status`
(`PENDING` | `WRITING` | `DELIVERED` | `FAILED_RETRYABLE` | `FAILED_PERMANENT`),
`attempt_count`, `bytes_written`, `rows_written`, `started_at`, `finished_at`,
`last_error`.

`run_ducklake_sink()` is rewritten as a state-machine driver:
1. Inserts a `PENDING` row before attempting any S3 write.
2. Transitions to `WRITING` before the Parquet upload.
3. On success, transitions to `DELIVERED` and updates byte/row counts.
4. On retryable failure (network timeout, transient S3 error), transitions to
   `FAILED_RETRYABLE` and increments `attempt_count`; the scheduler will retry
   on the next tick up to `pg_trickle.ducklake_sink_max_retries` (default 3).
5. On permanent failure, transitions to `FAILED_PERMANENT` and does not retry.

A new `pg_trickle.ducklake_sink_failure_mode` GUC (`warn` | `error`, default
`warn`) controls whether a permanent failure propagates as an error to the
calling refresh or remains asynchronous.

### COR-005: DuckLake View Registration After Query-Only ALTER (COR-005)

`alter_stream_table()` now detects when a stream table has an active
`ducklake_sink_mode` and the incoming ALTER contains only a `query` change.
In that case, the DuckLake view row is updated with the new query definition
immediately after the pg_trickle metadata migration, without requiring the
operator to also re-specify sink parameters.

### COR-006: DuckLake Snapshot ID Serialization (COR-006)

`register_ducklake_data_file()` and the related snapshot catalog writes now
acquire `pg_advisory_xact_lock(hashtext('ducklake_sink_' || table_id::text))`
before any `MAX(snapshot_id) + 1` computation. This eliminates the race window
where two concurrent sink writers for the same DuckLake table could compute the
same next snapshot ID.

### SEC-002: Qualified DuckLake Catalog Resolution (SEC-002)

All DuckLake catalog SQL in `ducklake_sink.rs` now resolves the DuckLake schema
explicitly. A new `pg_trickle.ducklake_catalog_schema` GUC (default `main`)
holds the schema name. `ducklake_view_table_exists()` queries `pg_class` and
`pg_namespace` with the qualified schema rather than using
`information_schema.tables` with only `table_name`. All `INSERT`/`UPDATE`
statements use `quote_ident(schema) || '.' || quote_ident(table)` patterns via
the shared SQL builder.

### OBS-001: DuckLake Sink Health Metrics (OBS-001)

New SQL function `pgtrickle.ducklake_sink_status()` returns a set of rows — one
per stream table with an active DuckLake sink — with columns:
`stream_table_name`, `last_delivery_status`, `last_delivery_at`,
`last_bytes_written`, `last_rows_written`, `failed_attempts`, `last_error`.

Prometheus counters added:
- `pg_trickle_ducklake_sink_attempts_total{stream_table}`
- `pg_trickle_ducklake_sink_successes_total{stream_table}`
- `pg_trickle_ducklake_sink_failures_total{stream_table,kind}`
- `pg_trickle_ducklake_sink_bytes_written_total{stream_table}`
- `pg_trickle_ducklake_sink_upload_seconds{stream_table}` (histogram)

### DEP-002: Arrow/Parquet/ObjectStore Dependency Policy (DEP-002)

`Cargo.toml` gains structured comments for the DuckLake dependency group
documenting the upgrade cadence (major versions on DuckLake protocol changes,
minor versions for CVE remediation within 30 days), supported object-store
schemes (`s3`, `gcs`, `az`), the binary-size impact (approx. 12 MB stripped),
and the CI coverage requirement (E2E DuckLake tests on every PR).
`deny.toml` includes the Arrow/Parquet dependency group in the
version-lock advisory section.

## Breaking Changes

- `pg_trickle.ducklake_sink_failure_mode = 'error'` causes sink failures to
  abort the refresh. The default remains `'warn'` (backward-compatible).
- The `ducklake_catalog_schema` GUC defaults to `main`. Deployments where
  DuckLake uses a different schema must set this GUC explicitly.