# Contributing

pg_sift needs PostgreSQL 17, 18 or 19 with the server headers (`pg_config` on `PATH`) and gcc or clang.

```sh
make                                # builds pg_sift.so; PG_CONFIG=/path/to/pg_config picks another installation
make installcheck                   # regression tests against a running server that can load pg_sift
make check                          # quick tier: every suite in throwaway clusters, about 1.5 minutes on 8 cores, 2.5 on 4
scripts/ci.sh affected              # the same, only the jobs your diff against origin/main (with the working tree) can break (scripts/areas.tsv); use it before a push
scripts/ci.sh affected --focus      # while developing: of the changed functions of src/, the regression tests that execute them (scripts/focus/); the full set is the push
make check-full                     # full tier: ASan build, long soak and fuzz runs, all mutants
```

`make check` and its scripts need bash 4 or newer: on macOS `brew install bash` and put it first on `PATH` (the stock bash 3.2 stops with `needs bash 4+`). The memory checks of the leak gate (`/proc`) and the CPU column of `scripts/ci.sh --times` are Linux only. A host that cannot run a stage says so and skips it: the Perl TAP tests (Homebrew's PostgreSQL has no `src/test/perl`), the compatibility tests that build a release older than 3.2.0 (x86-64 only) and the benchmark generators (Python 3.12).

A pull request that changes a source that starts parallel workers or handles snapshots (the files `ASSERT_MARK` of `scripts/ci/affected.sh` matches, e.g. `src/agg/par/worker.c`) runs the regression tests that run those paths on a server built with assertions (`ASSERT_JOBS`, about 27 tests, a few minutes) beside its other checks; the label `assertions` runs the same for any pull request. Say in the pull request text that the family passed on an assertions build (that job or `make pg-assert` locally).

`scripts/pgtest.sh install && scripts/pgtest.sh start` gives a throwaway cluster (ports 56100-56299, nothing installed system-wide); `scripts/pgtest.sh psql` connects to it and `scripts/pgtest.sh destroy` removes it.
The local gate runs only the PostgreSQL of `pg_config` on `PATH`; GitHub CI also runs 17. PostgreSQL 17 installs sift into the server's own directories (`SIFT_SUDO` when they are not writable), so run it on a private copy of the installation
(`cp -a /opt/pgsql-17 /tmp/pg17`, or the `postgresql-old-upgrade` package of Arch) with `PATH=/tmp/pg17/bin:$PATH SIFT_TMP=/tmp/sift-pg17 SIFT_PORT=56180 scripts/ci.sh affected`, or one job of it with `SIFT_CI_JOBS`. That build has no Perl TAP modules: the `tap-*.pl` jobs need a server built with `--enable-tap-tests`.
`make check` cuts the suites into jobs that one lane per core (at most 8, `SIFT_LANES=N`) takes from a queue; `SIFT_CI_JOBS=REGEX`
runs only the jobs whose name matches (`SIFT_CI_JOBS='installcheck-facets|tap-001' make check`).
`scripts/coldiff.sh` is the differential test of the aggregate pushdown: plain SQL aggregates with `sift.enable_agg` off and on, results compared,
with fixtures, scenarios and a self-test (also part of `make check`, docs/AGG.md).
The scratch directory (`$TMPDIR/sift-ci-PID`, `SIFT_CI_TMP` picks another) peaks at about 1.1 GB (the same with `SANITIZE=1`; `CI_BUDGET_MB=900` brings it to 1 GB at the price of about a quarter more time, as the two big WAL tests then run one after the other). A lane starts a job when
the estimated scratch of the running ones plus its own fits `CI_BUDGET_MB` (`EST_MB` in `scripts/ci/jobs.sh`), and a job's data goes when it passes.
`SIFT_CI_STATS=1 make check` prints the peak by component (WAL, tables, temp files), lane and job, and the memory of the servers; run it when
you add a heavy test, and give the job an estimate. The clusters are small (1 MB WAL segments, 48 MB `max_wal_size`, 32-64 MB buffers); the
crash, recovery and replication tests (`WAL_CONF` in `test/t/lib.bash`, `tap.sh` for the Perl ones) keep the default WAL segments and
settings, since the WAL is what they test.
`scripts/replication.sh` (also part of `make check`) tests physical and logical replication on temporary clusters;
`scripts/replication.sh --container` runs it with every node in its own docker container (`PG_IMAGE`, default `postgres:18`).
`SIFT_TESTING=1` (the default for `pgtest.sh` and `make check`) also builds the test-only `pg_sift_test` extension and the `sift._test_*` functions; a normal `make install` never installs them. Switching it needs `make clean`.
The regression tests in `TESTING_REGRESS` (Makefile) need those functions and are skipped by a release build; every other one
(except the slow ones of `SLOW_REGRESS` in `scripts/ci/plan.sh`) runs against the testing build, and the deep tier runs it against the release library too (`SIFT_TESTING` unset, built in a copy of the tree,
job `release-library-regress`, `scripts/release-library-check.sh`; the quick tier runs a smoke of it and the tests that refer to a test-only name behind a guard). Tests of limits, privileges and the planner belong in the second group (do not use
`sift._test_*` or the `sift.test_*` settings in them), so they run against the library users get. `SIFT_TESTING= scripts/pgtest.sh install`
builds it by hand.

Every suite runs under one leak gate, `scripts/leakcheck.sh`, which no test refers to: the server logs of every
cluster and node are scanned for what PostgreSQL logs when a transaction or backend leaks (buffer, relcache, TupleDesc,
snapshot, lock and file references, `you don't own a lock`), for assertion failures, crashes and sanitizer reports; a
`SANITIZE=1` server also reports malloc leaks as each process exits (LeakSanitizer, `scripts/lsan.supp`); every cluster is
checked when it stops (no session left in a transaction or holding locks, no prepared transaction, no temporary file left,
no long-lived process grown); the soak fails on a backend whose memory keeps growing. `make check` ends with the scan of
everything it ran. A test that crashes or leaks on purpose declares it with `scripts/leakcheck.sh expect COUNT REGEX`.

The SQLite extension in `sqlite/` is tested on its own, without PostgreSQL: `sqlite/test.sh` (about 40 s; `SIFT_SANITIZE=1 sqlite/test.sh` adds ASan/UBSan and the TSan stress test) needs
`cc`, `make`, the SQLite development files (3.38 or newer) and python3 with loadable `sqlite3` extensions and FTS5. `make -C sqlite isa-check` proves the baseline objects use no POPCNT or
vector instruction; `scripts/package-sqlite.sh` builds and tests the release tarball in containers. It compiles the portable core of `src/` (the files of `src/portable.list`) unmodified
against the host implementation in `sqlite/host/`.

**Errors say what to do.** An ERROR with SQLSTATE 54000 (limit), 0A000 (unsupported) or 55000 (wrong state) carries an `errhint` or `errdetail`; no ERROR is XX000. A condition the code
rules out is `SIFT_INVARIANT(cond, "text")` (`core/host.h`, an `Assert` in an assert build); anything a user or the data can reach is an `ereport` with its own errcode (XX001/XX002 for
damaged data, 40001 for a retry). `scripts/check-errors.sh` enforces it in the lint job and `scripts/ci.sh`; a test-only hook outside a `*_test.c` file says `check-errors: allow` in a
comment on the call.

**An expected output holds no failed verdict.** A test that compares two plans or two engines prints the verdict in a column (`agree`, `same`, `equal`, `match`, `ok`, `identical`, `equivalent`, `passed`, or a name made of them); `scripts/check-expected-oracle.py` (the `expected-oracle` step of `scripts/lint.sh`) fails an `f` or an empty cell (a NULL: the comparison did not say yes) in such a column of `test/expected/*.out` and the word `FAIL`, `MISMATCH` or `DIFFERENT` in a result cell, because a file accepted with one pins the disagreement as the right answer. An `f` or NULL that is the answer (a function named `match` asked a question it must refuse, a case with nothing to compare) is listed in `scripts/expected-oracle.allow` with its reason and its exact number of cells.

**Messages have a row.** Every NOTICE, WARNING and INFO of `src/` and `sql/` is one row of `docs/messages.tsv` (the exact text, where and when, the decision, its benefit and trade-off with numbers quoted from `docs/options.tsv` or a cited file, how to switch it off), and `docs/MESSAGES.md` is generated from the rows. `scripts/gen-messages.sh --skeleton` prints the row of a message that has none, `scripts/gen-messages.sh` writes the page, and `scripts/gen-messages.sh --check` (lint) fails on a message without a row, a row without a message, a changed text, DETAIL or HINT, and a number that is in no cited measurement.

**No source file passes 1000 lines.** A longer C file, header, script, SQL or Python file is split into a directory of context files (`src/<area>/<context>/*.c` and a small private header), a pure move first, then the change. `scripts/check-file-size.sh` enforces it in the lint job; documents, expected outputs and result tables are not counted, and the full install scripts of released minors (`sql/pg_sift--X.Y.sql`, frozen) are the one allowance.

**Portable files include only the host header.** A file of `src/portable.list` includes the C library, compiler intrinsics, other files of the list and
`src/core/host.h`, never a PostgreSQL, SQLite or other host header: new code there uses the vocabulary of `core/host.h` (documented at the top of that file) and, when
it needs something from the host, adds it to that vocabulary and to every host (`sqlite/host/host.h`), not a `#include`. `scripts/check-portable.sh` fails
a file that breaks the rule; it is a stage of `scripts/ci.sh` (so of `make check`) and the first step of `sqlite/test.sh` (the SQLite job). A change to a portable file
that uses another host API or a global mutable variable needs the host or the Makefile to follow, so run `sqlite/test.sh` as well as `make check` when you touch
them. Code shared by two hosts goes into the portable core, not into both. `make check` and `scripts/ci.sh` never
build or run `sqlite/`. Its changes are listed in [sqlite/CHANGELOG.md](sqlite/CHANGELOG.md); its version is `sqlite/VERSION`, which must equal the
release tag.

`src/` has one folder per area. Sources include each other by path from `src/` (`#include "bm25/query.h"`); a file is named for what it holds, without the `sift_` prefix or its folder's name, and goes to the folder of its area (`scripts/restructure-src.sh --check` fails for a file outside them or a stale path).

`scripts/check-neutral.sh` keeps the repository generic. It fails a tracked file (name or text), a commit of the branch (subject or body) or the title or body of the pull request that holds a name of the private vocabulary (product, machine and application-domain names; it is read from `NEUTRAL_VOCAB_FILE` or the `NEUTRAL_VOCAB` secret, and without it only the structural rules run), a name that the naming vocabulary of `test/README.md` replaced, an address of a private range, a local domain name or the home directory of a person. Data, tests, benchmarks and docs use generic names (orders, events, docs, facts, `dim_a_id`, `tenant_id`). A generic use of a listed word goes into the private vocabulary with a path, a text pattern and the reason; the inet and cidr test data may use `10.0.0.x` and `192.168.x.x` (`scripts/neutral.allow`). `scripts/ci.sh affected` runs the guard first, whatever the diff touches; `.github/workflows/neutral.yml` runs it on every pull request, also when only the title or body is edited (they reach the script through the environment).

Before sending a change:

- run `make check` and add a regression test (`test/sql/NAME.sql` plus `test/expected/NAME.out`) or a TAP test
  (`test/t/`) for a fix or a new behavior. For a bug, write the test first and see it fail before you fix it;
- keep the build free of compiler warnings;
- start the commit subject with the module or feature it changes: `query: parse fuzzy terms with a distance above 2`.

Contributions are licensed under the [PostgreSQL License](LICENSE) like the rest of the project.

## Removing a deprecated option

The options that a release deprecates (`order_columns`, `point_columns` and `key_cut_rows`, removed in 4.4) go in one commit each, with the setting that keeps them alive in tests (`sift.test_col_legacy_options`, `grep` for it in `src/col/am/` and `scripts/pgtest.sh`), the regression tests that name them or set that setting (`test/sql/`, `grep -l`), the bench cases that name them (`bench/cases/`), the advice rows of `docs/messages.tsv` and the preflight rows of `scripts/upgrade_check.sql` and the option rows of `sift.upgrade_commands()` in `sql/`. A test that checks a layout only the removed option wrote is deleted with it; a test that only passed the option drops the argument and keeps its answers.

## Merging

`scripts/land.sh PR` lands one pull request: it merges the tip of the PR's base into the branch (conflicts only in lists both sides add to, in the CHANGELOG and in generated files are resolved; any other conflict is exit 2, `SOURCE CONFLICT`: merge the tip yourself), runs `scripts/lint.sh`, the privacy gate and the check that a new regression test has a file `scripts/focus/NAME.tsv`, pushes only when needed, waits for the required checks (`gate` and `neutral`, or the branch protection list) and merges by rebase, or by squash when the branch holds a merge commit. Exit codes: 0 merged, 1 required checks red, 2 source conflict, 3 merge commit failed, 4 a gate is red, 5 the PR deletes files (`ALLOW_DELETE=1` after checking they are its own), 6 closed, 75 try again (checks pending, the tip or the branch moved). `--dry-run` prints the plan and pushes and merges nothing. Landings take turns on one kernel lock (`land.lock`), which goes away with its holder however it ends; nothing matches process names. State and settings are outside git, in `$XDG_STATE_HOME/pg_sift/land` (`LAND_DIR`): the lock, `MERGED.md`, and `privacy.deny`, the private list of names the privacy gate refuses (one regular expression per line; `LAND_DENY` names another file; without it the gate fails closed). `scripts/land.sh --selftest` is part of `scripts/lint.sh`.

## Releasing

Write `docs/releases/vX.Y.Z.md` with what is about the release (the sections up to Compatibility, from the notes of the previous release), run `scripts/release.sh X.Y.Z --notes` to append the Packages, SQLite extension and MariaDB extension sections that the release workflow checks (`scripts/release-components.tmpl`: the same text in every patch release, the version filled in; edit the file after for a release that changes them), keep the release notes under `## Unreleased` of the CHANGELOGs, push `main`, then
`scripts/release.sh X.Y.Z --dry-run` and `scripts/release.sh X.Y.Z`. It takes the green full CI run of the commit, checks the upgrade from the previous release and
`AUR_SSH_KEY`).