pg_sift 2.0.2: a decoder hardening against damaged index data and a fix for `VACUUM` running next to queries. Install the new files: the extension SQL does not change (no `ALTER EXTENSION pg_sift UPDATE` step), the on-disk format does not change (no `REINDEX`), the results do not change.

## Highlights

- **Security: a damaged index can no longer make a query read or write outside memory it owns.** The query path follows the lengths, counts and offsets that a term blob stores without the exhaustive checks of `sift.verify`, and used them unchecked. Bytes that are not what the writer wrote (a bad disk or controller, a torn page, a file edited with the server stopped, a physical copy from a damaged or hostile source) could make a backend read past the blob, write past a stack array, or return one group without end. Fuzzing the decoder found 35 distinct crash inputs on 2.0.1 in five minutes; 2.0.2 is clean on all of them, on the six reproductions of the audit and on about 4 million further executions.
  - **Affected:** any sift index whose pages are damaged. **There is no remote vector**: no SQL role can write index pages; it takes damage on disk or write access to the data directory.
  - **What to do:** install 2.0.2. No `REINDEX` is needed for healthy indexes. A damaged index now fails the query with `XX001` (`sift: corrupt posting data: ...`, naming what is wrong) instead; rebuild it with `REINDEX`, and use `sift.verify` to find out which are damaged. The checks are made once per blob header, per group and per block, not per posting. `sift.verify` itself also read one nibble past a tf stream whose bits counted more nibbles than it holds; fixed.
  - **Cost** (user instructions per operation against 2.0.1, `bench/steps/cpu_gate.sh`, interleaved, 7 rounds): queries 0.995x to 1.006x (the cells with a tight interval 1.000x to 1.003x), `VACUUM` after deleting 80% of the rows 1.007x (it opens every term blob of the segment it rewrites), `CREATE INDEX` 1.004x, `COPY`, inserts, `sift.merge` and `sift.promote` 1.000x to 1.003x.
- **Fixed: a lazy `VACUUM` running next to queries could reuse the index blocks of a segment while a query was still reading them.** A `VACUUM` (manual or autovacuum) is left out of snapshots and of the xid horizon, its own xid included, so a query that began after the `VACUUM` had taken its xid saw that xid as passed, and the blocks the `VACUUM` released with it became reusable while the query still held the directory that listed them. The query then failed with `sift index "t" is corrupted` (`expected page type 4 generation 107, found type 8 generation 0`); the page checks every read makes turn the reuse into an error, and no wrong result was seen. Reproduced on 2.0.1 within seconds by 24 pgbench clients (inserts, deletes and searches of stable rows with a content check) with `VACUUM` in a loop, and not without the `VACUUM` loop; with the fix the same run passes.
  - **Affected:** indexes that are searched while `VACUUM` or autovacuum runs on their table and rewrites segments (dead documents, merges by the `VACUUM`). Only the reuse of released blocks changes, never the contents of a segment: the error is transient (retry), nothing is stored wrong, no `REINDEX`.
  - **Now:** a `VACUUM` releases with the next xid, so the blocks it frees are reusable once a later transaction has started; its own truncation of the free end of the file (an exclusive lock keeps every query out) counts them as reusable at once, so the file still shrinks in the `VACUUM` that freed the space.
- Documentation: sample queries and documents use neutral wording; no change for users.

## Compatibility

- PostgreSQL 17 and 18; PostgreSQL 19 is experimental (CI builds it from git on every push). No extension SQL change (it stays 1.2) and no on-disk format change: existing indexes keep working, with no `REINDEX`. After installing the new files nothing else is required.

## Packages

x86-64 packages for PostgreSQL 18 are attached below (`SHA256SUMS` covers every asset); other versions build from source with `make USE_PGXS=1`.

- Debian 13, Ubuntu 24.04 (PGDG `postgresql-18`): `sudo apt install ./postgresql-18-pg-sift_2.0.2-1_amd64.deb`
- RHEL, Rocky, Alma 9 and 10 (PGDG `postgresql18-server`): `sudo dnf install ./pg_sift_18-2.0.2-1.el9.x86_64.rpm`
- Arch Linux: AUR `pg_sift`.

Every change: [CHANGELOG.md](https://github.com/brauliobo/pg_sift/blob/v2.0.2/CHANGELOG.md).

## SQLite extension

pg_sift for SQLite 2.0.2 (`sift`, a loadable extension; a separate component from the PostgreSQL extension above). Rebuilt on the shared core of the PostgreSQL 2.0.2 release: no change to the storage format or the SQL interface.

- **Shared engine:** the posting decoder is the one hardened for PostgreSQL. A table file with damaged bytes now fails the query with a data-corruption error instead of reading outside the term it decodes; the damaged table still has to be rebuilt from its source. The unit and feature tests pass under ASan and UBSan, and the stress test under TSan and ASan. No speed-up or slow-down is claimed: the figures above are PostgreSQL's. The `VACUUM` fix does not apply: SQLite has no shared page store.
- Needs SQLite 3.38 or newer; Linux x86-64, glibc 2.34 or newer. Built on Debian 12, loaded into the SQLite of Debian 12, Ubuntu 24.04 and AlmaLinux 10 before release.
- Storage format 2: nothing to upgrade from 2.0.1.

Download `sift-sqlite-2.0.2-linux-x86_64.tar.gz` (below, covered by `SHA256SUMS` with the packages), unpack it and `.load ./sift` from the `sqlite3`
shell (`sha256sum --ignore-missing -c SHA256SUMS`). Documentation: [sqlite/README.md](https://github.com/brauliobo/pg_sift/blob/v2.0.2/sqlite/README.md).
Every change: [sqlite/CHANGELOG.md](https://github.com/brauliobo/pg_sift/blob/v2.0.2/sqlite/CHANGELOG.md).

## MariaDB extension

pg_sift for MariaDB 2.0.2 (`sift`, a UDF library and two SQL files; a separate component from the extensions above). Rebuilt on the shared core of the PostgreSQL 2.0.2 release: no change to the index format or the SQL files.

- **Shared engine:** the posting decoder is the one hardened for PostgreSQL. Segment files with damaged bytes now fail the query with a data-corruption error instead of reading outside the term; `CALL sift_rebuild` rebuilds such an index from its table. The functional, live and stress tests pass on MariaDB 13.0.2. No speed-up or slow-down is claimed: the figures above are PostgreSQL's. The `VACUUM` fix does not apply: MariaDB has no shared page store.
- One library per MariaDB series, because it calls the server's own symbols and refuses to load into another series: 10.11, 11.4, 11.8, 12.1 and 13.0; Linux x86-64. Each was built in the stock `mariadb:<series>` image against that server's headers and loaded into a server of the same image before release. `BUILD-INFO` in the tarball lists the server version and the glibc each library needs; a server whose glibc is older than that of its library does not load it.
- Index format 2: an index of 2.0.1 opens as it is, nothing to rebuild.

Download `sift-mariadb-2.0.2-linux-x86_64.tar.gz` (below, covered by `SHA256SUMS` with the packages), unpack it, copy `lib/<series>/libsift_udf.so` of your server's series
(`SELECT VERSION()`) into its `plugin_dir`, then `SOURCE sql/functions.sql` once per server and `sql/sift.sql` once per database (`sha256sum --ignore-missing -c SHA256SUMS`).
Documentation: [mariadb/README.md](https://github.com/brauliobo/pg_sift/blob/v2.0.2/mariadb/README.md).
Every change: [mariadb/CHANGELOG.md](https://github.com/brauliobo/pg_sift/blob/v2.0.2/mariadb/CHANGELOG.md).