# Changelog

All notable changes to pgclone are documented in this file.

## [4.3.0]

### Added
- **Consistent-snapshot clones** — every clone now reads the source under a `BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY` transaction, so all per-table COPY commands within a single clone observe the same point-in-time view of the source database. This eliminates the foreign-key violations and partial-state anomalies that v4.2.x and earlier could produce when cloning a live OLTP source. Enabled by default for `pgclone.table()`, `pgclone.schema()`, `pgclone.database()`, `pgclone.table_async()`, `pgclone.schema_async()` (sequential and parallel pool modes).
- **Cross-connection snapshot sharing** for multi-connection paths — schema and database clones, and parallel pool mode, use `pg_export_snapshot()` / `SET TRANSACTION SNAPSHOT` so every libpq connection involved in a single clone (per-table sub-calls, FK retry, views, matviews, functions, triggers, and every pool worker) binds to one shared snapshot. Same correctness model as `pg_dump -j`.
- **Snapshot-coordinator background worker** (`pgclone_pool_coordinator_main`) — a dedicated bgworker spawned by `pgclone.schema_async(... '{"parallel": N}')` that opens its own source connection, exports the snapshot, publishes the ID into shared memory for pool workers to import, and sits idle in transaction until every importer has bound, then COMMITs.
- **Opt-out option** — `'{"consistent": false}'` in any options-JSON disables the wrapping for callers who want lower source-side lock pressure or prefer the v4.2.x behaviour.

### Changed
- `pgclone.table()` source connection now runs at REPEATABLE READ READ ONLY for the entire clone (was: no transaction, except for an inner READ ONLY wrap when `WHERE` was set). The old per-table inner `BEGIN TRANSACTION READ ONLY` is now a no-op when an outer snapshot transaction is already open, preserving SQL-injection containment in both call paths.
- `pgclone.schema()` keeps its initial source connection open across all sub-phases as a snapshot keeper. Previously it closed the connection after the table-list read and each sub-phase opened independent connections that could see different points in time.
- `pgclone.database()` follows the same keeper pattern at one level higher; the snapshot ID is propagated to every per-schema sub-call and on through to per-table sub-calls.
- `pgclone.schema_async(... '{"parallel": N}')` now launches `N + 1` background workers (one coordinator plus N pool workers) when in consistent mode. The coordinator job appears in `pgclone.jobs_view` like any other job for full progress visibility.

### Tradeoffs documented
- Long-running clones now hold an open transaction on the source for the entire clone duration, which delays VACUUM cleanup of dead tuples and WAL recycling proportional to clone time. For very long clones on busy sources this may cause table/index bloat and WAL accumulation; opt out with `'{"consistent": false}'` if that is more important than cross-table consistency.
- Hot-standby sources are supported on PostgreSQL ≥ 10 (where `pg_export_snapshot()` works on hot standby).

### Internal
- New `CloneOptions.consistent` (bool) and `CloneOptions.snapshot_id[64]` fields plus parser support for `"consistent": false` and `"snapshot_id": "..."` in options JSON.
- New helpers `pgclone_begin_repeatable_read()`, `pgclone_commit_source()`, `pgclone_export_snapshot()`, `pgclone_begin_with_imported_snapshot()`, `pgclone_setup_source_txn()`, `pgclone_setup_source_txn_done()` in `src/pgclone.c`. Mirror helpers `bgw_begin_repeatable_read()`, `bgw_commit_source()`, `bgw_export_snapshot()`, `bgw_begin_with_imported_snapshot()` in `src/pgclone_bgw.c`.
- New `PgclonePoolQueue` shared-memory fields: `consistent`, `snapshot_ready`, `snapshot_failed`, `snapshot_imported_count`, `snapshot_expected_workers`, `launch_complete`, `snapshot_id[64]`, `coordinator_job_id`.
- New `PgcloneJob.consistent` field so the single-worker async path knows whether to wrap in REPEATABLE READ READ ONLY.

## [4.2.0]

### Added
- **Pre-flight validator (`pgclone.preflight(source_conninfo, schema_name)`)** — read-only sanity check that surfaces issues that would otherwise fail mid-clone. Returns a JSON document with three role-based summary arrays (`errors` / `warnings` / `info`) and a per-check object covering: source/target connection, PostgreSQL versions and major-version compatibility, schema existence on both sides, USAGE/SELECT on source and CREATE on target, estimated source size, current target database size, object counts (tables/views/sequences/indexes), name conflicts on the target schema, extensions installed on source but missing on target, owner/grantee roles missing on target, and non-default tablespaces missing on target. `ready` is `true` only when zero errors are recorded.
- **Loopback test coverage** — new preflight assertions in `test/test_loopback.sh` (function registration, JSON shape, `ready` boolean, fabricated name-conflict surfacing, missing-source-schema error path, STRICT NULL handling, read-only catalog invariant).

### Changed
- **CI runs `test/test_loopback.sh` directly** instead of an inline subset hand-rolled in `.github/workflows/ci.yml`. Going forward, every assertion added to the loopback script is exercised in CI automatically — closing the gap that hid v4.1.0 schema-diff tests from the matrix.

### Internal
- New isolated translation unit `src/pgclone_preflight.c`. Like `src/pgclone_diff.c`, this file does not share helpers with `src/pgclone.c` — the feature is fully additive and trivially auditable.
- Both source and target connections run inside `BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY`. The function never issues DDL or DML on either side.

## [4.1.0]

### Added
- **Schema diff (`pgclone.diff(source_conninfo, schema_name)`)** — read-only DDL drift detection between source and the local target. Returns a JSON document with summary counts plus per-category arrays of `only_in_source` / `only_in_target` / `modified` for tables (with per-column `type` / `not_null` / `default` drift), indexes (excluding those backing constraints), constraints, user-defined triggers, views and materialized views, and sequences. Both source and local connections run inside `BEGIN ... READ ONLY` transactions; the function never executes DDL or DML on either side.

### Internal
- New isolated translation unit `src/pgclone_diff.c`. The diff feature does not share helpers with `src/pgclone.c`, keeping the surface area additive and trivially auditable.
- Catalog queries explicitly use `ORDER BY ... COLLATE "C"` to guarantee identical sort order on both sides regardless of the local lc_collate setting, so the merge-walk comparison is deterministic.

## [4.0.1]

### Fixed
- **Schema clone dependency ordering** (issue #3): `pgclone.schema()` now creates objects in dependency-respecting order — sequences → tables (no triggers) → FK retry → views → matviews → functions → triggers — instead of the previous order which cloned functions before tables. SQL-language functions whose body references a table in the same schema no longer fail with `relation "schema.table" does not exist` at `CREATE FUNCTION` time.
- **Unqualified relation references in extracted DDL** (issue #3): all source libpq connections now `SET search_path = pg_catalog` immediately after connect. This forces `pg_get_triggerdef()`, `pg_get_expr()` (column DEFAULTs), and `pg_get_indexdef()` to emit fully schema-qualified relation names. Previously, source DBs with an application schema on their default `search_path` (a common production pattern) produced DDL like `CREATE TRIGGER ... ON city_street ...` and `DEFAULT nextval('documents_to_resend_id_seq')` that failed when replayed on the target loopback connection. Affects sync paths and both bgworker async paths (sequential and pool).

### Internal
- New `pgclone_normalize_session()` helper in `src/pgclone.c` invoked from `pgclone_connect()`; equivalent inline `SET search_path` calls added at the two `PQconnectdb(job->source_conninfo)` sites in `src/pgclone_bgw.c`.
- `pgclone_schema()` now passes `triggers=false` through to each per-table `pgclone_table()` call and runs a single trigger pass at the end after functions are cloned.

## [4.0.0] — BREAKING

### Changed
- **Schema namespace**: All pgclone functions now live under the `pgclone` schema, created automatically by the extension
  - `pgclone_table(...)` → `pgclone.table(...)`
  - `pgclone_schema(...)` → `pgclone.schema(...)`
  - `pgclone_database(...)` → `pgclone.database(...)`
  - `pgclone_database_create(...)` → `pgclone.database_create(...)`
  - `pgclone_table_async(...)` → `pgclone.table_async(...)`
  - `pgclone_schema_async(...)` → `pgclone.schema_async(...)`
  - `pgclone_progress(...)` → `pgclone.progress(...)`
  - `pgclone_cancel(...)` → `pgclone.cancel(...)`
  - `pgclone_resume(...)` → `pgclone.resume(...)`
  - `pgclone_jobs()` → `pgclone.jobs()`
  - `pgclone_clear_jobs()` → `pgclone.clear_jobs()`
  - `pgclone_progress_detail()` → `pgclone.progress_detail()`
  - `pgclone_jobs_view` → `pgclone.jobs_view`
  - `pgclone_discover_sensitive(...)` → `pgclone.discover_sensitive(...)`
  - `pgclone_mask_in_place(...)` → `pgclone.mask_in_place(...)`
  - `pgclone_create_masking_policy(...)` → `pgclone.create_masking_policy(...)`
  - `pgclone_drop_masking_policy(...)` → `pgclone.drop_masking_policy(...)`
  - `pgclone_clone_roles(...)` → `pgclone.clone_roles(...)`
  - `pgclone_verify(...)` → `pgclone.verify(...)`
  - `pgclone_masking_report(...)` → `pgclone.masking_report(...)`
  - `pgclone_version()` → `pgclone.version()`
  - `pgclone_table_ex(...)` → `pgclone.table_ex(...)`
  - `pgclone_schema_ex(...)` → `pgclone.schema_ex(...)`
  - `pgclone_functions(...)` → `pgclone.functions(...)`
- Extension control file now specifies `schema = pgclone`
- **Upgrade path**: This is a breaking change. Users must `DROP EXTENSION pgclone; CREATE EXTENSION pgclone;` to upgrade from v3.x. All application queries must be updated to use the new `pgclone.` prefix.

## [3.6.0]

### Added
- **GDPR/Compliance Masking Report**: `pgclone_masking_report(schema)` generates an audit report listing all sensitive columns, their masking status, and recommendations
  - Detects sensitive columns using shared `sensitivity_rules` (~40 patterns across 10 categories)
  - Checks if a masked view (`table_masked`) exists for each table
  - `mask_status`: `MASKED (view)` or `UNMASKED`
  - `recommendation`: either "OK - masked via view" or "Apply mask strategy: X"
  - Returns SET OF (schema_name, table_name, column_name, sensitivity, mask_status, recommendation)
  - Useful for GDPR, HIPAA, SOX compliance audits — "prove no PII exists unmasked"

### Changed
- Refactored sensitivity rules into shared `SensitivityRule` struct and `pgclone_match_sensitivity()` helper — used by both `pgclone_discover_sensitive` and `pgclone_masking_report`
- Version bumped to 3.6.0

## [3.5.0]

### Added
- **Clone Verification**: `pgclone_verify()` compares row counts between source and target databases, table by table, returning a side-by-side comparison
  - Two overloads: `pgclone_verify(conninfo)` for all schemas, `pgclone_verify(conninfo, schema)` for a specific schema
  - Returns SET OF (schema_name, table_name, source_rows, target_rows, match)
  - Match indicators: `✓` (equal), `✗` (different), `✗ (missing)` (table not on target)
  - Uses `pg_class.reltuples` for fast approximate counts without full table scans
  - Works with regular and partitioned tables
- 5 new pgTAP tests (84 total): verify function runs, row counts match after clone, correct column count in result

### Changed
- Version bumped to 3.5.0

## [3.4.0]

### Added
- **Clone Roles with Permissions and Passwords**: `pgclone_clone_roles()` clones database roles from source to local, including encrypted passwords, role attributes, and all privilege grants
  - `pgclone_clone_roles(conninfo)` — clone all non-system roles
  - `pgclone_clone_roles(conninfo, 'role1,role2')` — clone specific roles (comma-separated)
  - Clones role attributes: LOGIN, SUPERUSER, CREATEDB, CREATEROLE, REPLICATION, INHERIT, CONNECTION LIMIT, VALID UNTIL
  - Copies encrypted passwords from `pg_authid` (requires superuser on both source and target)
  - If target role already exists: updates password, attributes, and applies permissions additively
  - Clones role memberships (GRANT role TO role)
  - Clones schema-level privileges (USAGE, CREATE) via `aclexplode()`
  - Clones table-level privileges (SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER)
  - Clones sequence privileges (USAGE, SELECT, UPDATE)
  - Clones function/procedure EXECUTE privileges
- `test_reader`, `test_writer`, `test_admin` test fixture roles with graduated permissions
- 6 new pgTAP tests (79 total): role creation, LOGIN attribute, CREATEDB attribute verification

### Changed
- Version bumped to 3.4.0

## [3.3.0]

### Added
- **Dynamic Data Masking via Views**: Role-based masking policies that preserve original data while presenting masked views to unprivileged users
  - `pgclone_create_masking_policy(schema, table, mask_json, privileged_role)`: creates a masked view (`table_masked`), revokes base table SELECT from PUBLIC, grants view access to PUBLIC, grants direct table access to the privileged role
  - `pgclone_drop_masking_policy(schema, table)`: drops the masked view and restores base table access to PUBLIC
  - Uses the same `pgclone_build_mask_expr()` engine as clone-time and static masking — all 8 strategies work
  - Queries local `pg_attribute` for column names, applies mask expressions to matching columns
- 7 new pgTAP tests (73 total): policy creation, view existence, masked data verification, privileged role bypass, policy drop, view cleanup

### Changed
- Version bumped to 3.3.0

## [3.2.0]

### Added
- **Static Data Masking on Local Tables**: `pgclone_mask_in_place(schema, table, mask_json)` applies masking to already-cloned tables via UPDATE statements — no source connection needed
  - Uses the same mask JSON format as clone-time masking: `{"email": "email", "name": "name", "ssn": "null"}`
  - Reuses `pgclone_build_mask_expr()` for consistent masking between clone-time and post-clone paths
  - Wraps user-provided JSON into clone options format for unified parsing
  - Returns summary: `OK: masked N rows in schema.table (M columns)`
- 7 new pgTAP tests (66 total): clone-then-mask workflow, verify original data removed, names/SSNs masked, row count preserved

### Changed
- Version bumped to 3.2.0

## [3.1.0]

### Added
- **Auto-Discovery of Sensitive Data**: `pgclone_discover_sensitive(conninfo, schema_name)` scans the source catalog for columns matching ~40 sensitive data patterns and returns suggested mask rules as JSON
  - Pattern categories: email, name, phone, SSN/national ID, financial (salary/income), secrets/credentials (password/token/api_key), address, date of birth, credit card, IP address
  - Case-insensitive matching with LIKE-style wildcards against column names
  - Output is a JSON object grouped by table, ready to paste into the `"mask"` option: `{"employees": {"email": "email", "salary": "random_int", "ssn": "null"}}`
  - Scans regular tables and partitioned tables (relkind `r` and `p`)
- 6 new pgTAP tests (59 total): discovery function runs, detects email/full_name/phone/salary/ssn columns

### Changed
- Version bumped to 3.1.0

## [3.0.0]

### Added
- **Data Masking / Anonymization**: Column-level data masking applied during cloning via the `"mask"` JSON option
  - 8 masking strategies: `email`, `name`, `phone`, `partial`, `hash`, `null`, `random_int`, `constant`
  - `email`: preserves domain, masks local part (`alice@example.com` → `a***@example.com`)
  - `name`: replaces with `XXXX`
  - `phone`: keeps last 4 digits (`+1-555-123-4567` → `***-4567`)
  - `partial`: configurable prefix/suffix retention (`Johnson` → `Jo***on` with prefix=2, suffix=2)
  - `hash`: deterministic MD5 hash — preserves referential integrity across tables
  - `null`: replaces with NULL
  - `random_int`: random integer in configurable `[min, max]` range
  - `constant`: fixed replacement value
  - All strategies are NULL-safe (NULL inputs produce NULL outputs, except `constant` and `random_int`)
- Masking is applied server-side as SQL expressions inside `COPY (SELECT ...) TO STDOUT` — no row-by-row processing overhead
- Fully composable with existing `columns`, `where`, `indexes`, `constraints`, `triggers` options
- `MaskRule` struct and `pgclone_build_mask_expr()` internal functions for extensible mask strategy architecture
- 15 new pgTAP tests (53 total): email masking, name masking, null masking, hash masking, constant masking, combined masks with WHERE filter
- `test_schema.employees` test fixture table with realistic sensitive data (names, emails, phones, salaries, SSNs)

### Changed
- `CloneOptions` struct extended with `masks[]` array and `num_masks` counter
- `pgclone_parse_options()` handles `"mask"` JSON key with both simple string values (`"email"`) and object values (`{"type":"partial", "prefix":2, "suffix":3}`)
- `pgclone_copy_data()` queries source catalog for column names when masks are active (without explicit column list) to apply per-column mask expressions
- Version bumped to 3.0.0

## [2.2.1]

### Added
- **WHERE clause SQL injection protection**: Two-layer defense against malicious input in the `"where"` JSON option
  - Layer 1: Keyword validation rejects DDL/DML keywords (`DROP`, `INSERT`, `UPDATE`, `DELETE`, `CREATE`, `ALTER`, `TRUNCATE`, etc.) and semicolons before the query is sent
  - Layer 2: Source connection runs inside `BEGIN TRANSACTION READ ONLY` when a WHERE clause is present — PostgreSQL itself blocks any write operations even if validation is bypassed
  - Word-boundary-aware matching avoids false positives on column names like `created_at`, `update_count`, `drop_rate`
- 4 new pgTAP tests (37 total): semicolon rejection, DROP keyword rejection, INSERT keyword rejection, false-positive safety with `created_at` column

## [2.2.0]

### Changed
- **Worker Pool Architecture**: Parallel cloning (`"parallel": N`) now uses a fixed-size worker pool instead of spawning one background worker per table
  - Exactly N workers are launched, each pulling tasks from a shared queue
  - Dynamic load balancing: faster workers automatically handle more tables
  - Resource usage reduced from O(tables) to O(N) for bgworkers and DB connections
  - No longer risk exhausting `max_worker_processes` on large schemas

### Added
- `PgclonePoolQueue` struct in shared memory for task queue management
- `pgclone_pool_worker_main()` background worker entry point
- Pool worker test in `test/test_async.sh` (TEST 8)
- `PGCLONE_MAX_POOL_TASKS` limit (512 tables per pool operation)
- Guard against concurrent pool operations

### Removed
- Per-table background worker launch in parallel mode (replaced by pool)
- Per-table job slot allocation in parallel mode (pool workers share fewer slots)

## [2.1.4]

### Changed
- Local loopback connections now use Unix domain sockets (from `unix_socket_directories` GUC) instead of TCP `127.0.0.1`
- `pg_hba.conf` `trust` entry for `127.0.0.1` is no longer required for async operations — default `local all all peer` is sufficient
- Falls back to TCP `127.0.0.1` automatically if Unix sockets are unavailable

### Fixed
- Security: removed unnecessary `trust` authentication requirement for background worker connections

## [2.1.3]

### Fixed
- Async bgworker: COPY pipeline error handling — failures now logged with `PQerrorMessage`, source COPY result consumed on error path to prevent connection leak
- Async bgworker: `WaitForBackgroundWorkerStartup` added to all async functions — jobs no longer stuck in 'pending' state
- `pgclone_schema_async` parallel mode: fixed hardcoded `jobs[0]` write that corrupted slot 0 in shared memory
- `pgclone_schema_async` parallel mode: parent job now correctly transitions to COMPLETED after child workers finish

### Added
- `pgclone_clear_jobs()` function to free completed/cancelled job slots from shared memory
- Async test suite (`test/test_async.sh`) covering `pgclone_table_async`, `pgclone_schema_async`, `pgclone_progress`, `pgclone_jobs_view`, `pgclone_clear_jobs`
- `CONTRIBUTING.md` — development setup, code guidelines, PR process
- `SECURITY.md` — vulnerability reporting, security considerations
- Documentation restructured: `docs/USAGE.md`, `docs/ASYNC.md`, `docs/TESTING.md`, `docs/ARCHITECTURE.md`, `CHANGELOG.md`

## [2.1.2]

### Added
- Elapsed time column in `pgclone_jobs_view`
- `elapsed_time` field in progress detail output

## [2.1.1]

### Changed
- Visual progress bar in `pgclone_jobs_view` replaces verbose NOTICE messages
- Per-table/per-row NOTICE messages moved to DEBUG1 level

## [2.1.0]

### Added
- `pgclone_jobs_view` — query async job progress as a standard PostgreSQL view
- `pgclone_progress_detail()` — table-returning function for detailed progress

## [2.0.1]

### Added
- `pgclone_database_create()` — create a new database and clone into it
- Automatic pgclone extension installation in the target database
- Idempotent behavior: clones into existing database if it already exists

## [2.0.0]

### Added
- True multi-worker parallel cloning with `"parallel": N` option
- Each table gets its own background worker
- Parent worker monitors child workers via shared memory

### Changed
- Shared memory layout expanded for parallel job tracking

## [1.2.0]

### Added
- Materialized view cloning during schema clone (with `"matviews": false` opt-out)
- Exclusion constraint support (cloned alongside PK, UNIQUE, CHECK, FK)
- Materialized view indexes and data are preserved

## [1.1.0]

### Added
- Selective column cloning with `"columns": [...]` JSON option
- Data filtering with `"where": "..."` JSON option
- Automatic filtering of constraints/indexes referencing excluded columns

## [1.0.0]

### Added
- Async clone operations via background workers (`pgclone_table_async`, `pgclone_schema_async`)
- Job progress tracking (`pgclone_progress`, `pgclone_jobs`)
- Job cancellation (`pgclone_cancel`)
- Job resume from checkpoint (`pgclone_resume`)
- Job cleanup (`pgclone_clear_jobs`)
- Conflict resolution strategies: error, skip, replace, rename

## [0.3.0]

### Added
- Background worker infrastructure
- Shared memory allocation for job state
- `_PG_init` with shmem hooks

## [0.2.0]

### Added
- Index cloning (including expression and partial indexes)
- Constraint cloning (PRIMARY KEY, UNIQUE, CHECK, FOREIGN KEY)
- Trigger cloning with trigger functions
- JSON options format for controlling indexes/constraints/triggers
- Boolean parameter variants (`pgclone_table_ex`, `pgclone_schema_ex`)

## [0.1.0]

### Added
- Initial release
- `pgclone_table()` — clone a single table with or without data
- `pgclone_schema()` — clone an entire schema
- `pgclone_functions()` — clone functions only
- `pgclone_database()` — clone all user schemas
- COPY protocol for fast data transfer
- `pgclone_version()` — version string
- Support for PostgreSQL 14–18
- pgTAP test suite (33 tests)
- Docker Compose multi-version test infrastructure
- GitHub Actions CI pipeline