# Changelog

This file is the changelog of the PostgreSQL extension. The SQLite extension (`sift` for SQLite, artifact `sift-sqlite`) has its own: [sqlite/CHANGELOG.md](sqlite/CHANGELOG.md).

## Unreleased

## 4.3.2 - 2026-11-11

- Fixed: pg_sift: a correlated sub-select over a join with a derived table that PostgreSQL does not pull up (`GROUP BY`, `DISTINCT`, `UNION`, `HAVING`), filtered by a column of the outer row on a column of the derived table, could return the count (or sum) of an earlier outer row for the later ones when the aggregate scan ran it: the scan read the derived table once and replayed it for every outer row. The scan now leaves such a sub-select to PostgreSQL's plan, which asks the derived table once for each outer row (seen from 4.0.5 to 4.3.1).


## 4.3.1 - 2026-11-11

- Fixed: pg_sift: on a server with a publication that publishes deletes and covers the extension's tables (`CREATE PUBLICATION .. FOR ALL TABLES`), every `DROP` of a relation (`DROP INDEX`, `DROP TABLE`, a plain table too) failed with `cannot delete from table "table_option_values" because it does not have a replica identity and publishes deletes`: the event trigger that removes the options of a dropped table deletes from `sift.table_option_values`, which had no replica identity. Releases 4.2.0 to 4.3.0 are affected. `ALTER EXTENSION pg_sift UPDATE` makes the unique key index of the table its replica identity (no other step); a fresh install has it. The other tables of the extension have a primary key. Held by `publication_all_tables` (the update from 4.2.0 and from 4.3.0, a fresh install, a subscriber that has the tables).
- Fixed: pg_sift: a read through a `sift_col` index with `track_changes` (an aggregate, a filtered count, a top-N over groups) inside the statement that deletes or updates the key of rows, or before its statement triggers have run, counted the rows the statement ended: an `AFTER DELETE` statement trigger of the table whose name sorts before the index's own (`sift_track_*`), an `AFTER ROW` trigger, a function the statement calls (a `WHERE` clause, a `MERGE` condition) and a `BEFORE` trigger of the same statement read the log before the tombstones of the statement were written (50,000 rows against 47,500 in a statement trigger that deleted 2,500). The index reads the heap for the table while a statement that ends row versions of it runs, so the answers equal the heap's; the next statement, an `AFTER` trigger with a later name and the commit were correct. Affects 4.0.0 to 4.3.0 (test `col_track_own_statement_reads`).
- Fixed: pg_sift: sessions that run one query at once could load the rows of its joined table more than once (the shared lookup): a session that looked for the rows just before another kept them, and asked for the load's claim just after that one dropped it, found no claim to wait for and loaded the table again (the answers were right); it now looks for the rows again once it holds the claim
- Fixed: pg_sift: a `VACUUM` that vacuums the b-trees of a table with parallel workers no longer warns `maintenance failed: cannot assign transaction IDs during a parallel operation` for a `sift` (BM25) index, counts no failure for it and no longer leaves its segments to the retry: the merge by workers needs a transaction id before a parallel mode of its own, which the leader of that `VACUUM` cannot take, so one process merges the segments in the same `VACUUM`.

## 4.3.0 - 2026-11-10

- Fixed: pg_sift: on PostgreSQL 19 the parents that have no child (a `LEFT JOIN` with a `WHERE` on a column of the child being `NULL`, `NOT EXISTS`: an anti join) were counted by a serial scan of the parents with the whole child as its lookup, where 17 and 18 scan the child with the parents as its lookup and use workers; the child-first scan is also planned for the anti join, and for `NOT EXISTS` on every major
- Changed: pg_sift: the choice of the second order of the order arrays (`order_array_group = 'auto'`) prices the table of groups that every segment writes (24 bytes a group, or about 4 from 1,024 groups with no NULL key in the segment) where it priced one table for the index: an entity of many values with few rows apiece is no longer refused for a table it does not write (300,000 rows, 60,000 values: refused, now chosen, the index grows 4.3 bytes a row), and one with NULL keys is refused where its tables would be the index. The choice is the same at 100, 1,000 and 93,000 values over 1,000,000 and 5,000,000 rows.
- Improved: pg_sift: the table of groups of a grouped order array (`order_array_group`) of 1,024 groups or more and no NULL key is packed to the bits of the distance of a group from the first and of the start of its entries, about 4.4 bytes a group where it took 24 (column format minor 44; a block with NULL keys keeps the 24): on 5,000,000 rows and 93,000 values the index with the group is 65.0 MB, was 102.7 MB (39.5 MB without it), the build 0.2% more instructions, a top-N of one group 5% more and an `IN` of two 6% fewer instructions, all of them 24 to 37% fewer blocks; an index built before keeps its tables until `REINDEX`.
- Improved: pg_sift: the second order of the order arrays (`order_array_group`) names the entries of a chunk by their position in the plain order of the column where the positions of a group are close (at most 1,024 apart on the mean) and that takes at most 75% of the bytes of keys and heap addresses (`gord_pos_pct`, `gord_pos_gap`; no option): on 5,000,000 rows the second order by an integer of 100 values grows the index +20.2% over the plain order where it grew +59.6%, 1,000 values +14.2% where +30.6%, the build is 7.7% more instructions than by keys and addresses, and `ORDER BY ts LIMIT` and `IN` of the group read half the blocks or fewer; a group of 93,000 values or more keeps the old form. Column format minor 44; an index built before keeps its arrays until `REINDEX`.
- Improved: pg_sift: a top-N, an `OFFSET` or a range by an order array of a `sift_col` index that tracks its changes (`track_changes`, the default) asks the tombstone log about the heap pages of the entries it takes, not about the whole superblock of each: over a table whose rows were updated and not vacuumed, the keys are spread over the whole heap and the superblock masks read the tombstones of the whole table for a few rows; the rows are the same.
- Improved: pg_sift: a top-N, an `OFFSET` or a page of an integer or time key with an `IN` list on it (`ORDER BY u LIMIT 100` with `u IN (...)`) reads the entries of the order array of each value of the list after those of the one before, a seek apiece, in place of a range from the first value to the last that tests every entry between (a list of a few values over a large table read the whole array) or, for a list of more than a dozen values, the units; a list of a thousand values or more still reads the units.
- Improved: pg_sift: a read of a group array (`order_array_group`) of a `sift_col` table with updated and vacuumed rows lists the dead rows of the units whose pages it reads, not of the whole segment sorted for every statement: the top rows of a group no longer cost time in proportion to the rows of the table.
- Improved: pg_sift: a chunk of an order array may be seekable (a fence key every 32 entries), so a seek or a scan decodes 32 entries and not 512, and a walk keeps the handles of the segments for the backend's later statements and decodes only the addresses it consumes: 5,000,000 rows, a plain array on a timestamp, top-N descending 1.38 M to 0.88 M instructions, `OFFSET` ascending 1.03 M to 0.87 M, a range 2.47 M to 1.37 M, an `IN` list 3.05 M to 2.56 M. A range with both bounds and `LIMIT 20` seeks the far end of each segment's range only when the scan gets there (26 segments: 1.138 M to 0.898 M instructions and 149 to 64 blocks read cold; a b-tree 0.265 M and 23), a cold seek reads the header of an order block alone, reads the directory when a seek first needs it and starts no read-ahead after a lookup (blocks read cold: range 258 to 188, top-N 227 to 160, `OFFSET 100` 250 to 187; a b-tree 23, 23 and 123), and an equality or an `IN` list of ids decodes the addresses of its entries and not those of the whole chunk (200,000 rows: 64 keys 2.68 M to 1.47 M instructions, 16 keys 0.82 M to 0.54 M, one key 0.18 M to 0.16 M). The seekable form takes 9.33 bytes a row for 1,000,000 random keys where the plain one took 9.86 (a b-tree 9.31). Column format minor 44, which a release before 4.3 refuses; an index built before keeps its chunks until a merge or `REINDEX` writes them again.
- Improved: pg_sift: the header page of a segment of a `sift_col` index keeps a copy of the small ranges that a plan and the first read open (the schema, the unit directory, the point directories, the order headers, the zone directory, the pre-aggregate and set headers), so planning an index with many segments reads fewer blocks: over 5,000,000 rows in 18 segments a plan reads 81 index pages where it read 146, the first ordered range query 98 where 121 (116 where 193 with the index and heap evicted) and a top-N 75 where 106; the index grows by 0.26%, a warm statement runs 1 to 4% more instructions, and build, promotion, merge and `VACUUM` times do not change. The statistics of an index count the compact chunks from the headers alone, and the share of units without a histogram or a rollup is read only when the cost of a pre-aggregate answer needs it: the first statement of a new backend over 5,000,000 rows in 26 segments plans from 238 blocks where it read 416 (buffer hits 1,021 to 279), and over 1,000,000 rows from 81 where it read 112. No format change.
- Improved: pg_sift: a merge of the segments of a `sift_col` index keeps to `maintenance_work_mem` (a forced merge of the 5,000,000 row test table at 64 MB peaked at 97 MB, now 59 MB; at 256 MB 261 MB, now 229 MB): a unit decoded for a merge no longer leaves a buffer of its rows in the merge's memory until the merge ends, an input with no unit in a step gives back the arrays of its largest unit, the arrays that are held count with the writer's memory when a segment is cut, and the partitions of a merge are sized to the memory a writer may take within `maintenance_work_mem` (what the build already assumed: the backend's own 30 MB and 1.4 times the estimate), so a merge at 64 MB writes more, smaller segments than before.
- Improved: pg_sift: a forced merge at the default `maintenance_work_mem` of 64 MB writes 15 segments of the 5,000,000 row test table where it wrote 17 (8 at 88 MB, 2 at 256 MB), at a peak of 59 MB (82 MB at 88 MB, 229 MB at 256 MB): a merge streams its inputs, so its memory is the writer's estimate and 36 MB, not 1.4 times the estimate and 30 MB as a build's is. The partitions of a merge are cut at their share of the bytes still to write, so none is a short last one, and a second force finds them and writes nothing.
- Changed: pg_sift: `CREATE INDEX` and `REINDEX` group the second order of the order arrays (`order_array_group = 'auto'`) also by an integer column that a non-unique b-tree of the table runs the shape of (the column and a date or timestamp of the index as its first two key columns, in either order), where they grouped by an entity only: `WHERE col = x ORDER BY ts LIMIT n` on a table that has that b-tree no longer reads the units after the b-tree is replaced by the sift index. The NOTICE and `sift.index_choices` say whether a b-tree, an entity or the hint chose the group; an index built before keeps its arrays until `REINDEX`.
- Changed: pg_sift: `key_order = 'col'` (one column) no longer sets `point_tid_columns` on the column: its directory made the index 71 to 95% larger (4.89 to 9.34 MB at 1,000,000 rows, 23.98 to 46.74 MB at 5,000,000) and made an `IN` list of 100 values 5.7 to 11.8 times cheaper (a b-tree is 1.7 to 3.8 times cheaper still), an equality up to 1.6 times and nothing else; set `point_tid_columns` by hand for such lists. Indexes built before keep their directory.
- Changed: pg_sift: a serial id (a nearly unique integer that follows the heap order, correlation 0.99 or more, with no unique b-tree of its own) gets an order array chosen by the build, kept whole, and no point directory: runs of consecutive heap addresses take 0.4 bytes a row where the directory took 3.3. 5,000,000 rows, an index of an id, a group, a timestamp and an amount: 63.3 MB to 49.1 MB; 1,000,000 rows 9.0 MB to 6.1 MB. An equality on the id takes 0.46 M instructions and 5 cold blocks with the directory and 0.51 M and 2 with the array (a b-tree 0.12 M), the other classes are within 3%; a list of many ids costs more with the array (`id IN` of 64 keys over 1,000,000 rows: 3.7 M instructions in 94 cold blocks against 1.0 M in 155), so set `point_tid_columns = 'id'` where the reads are long id lists. A table with a primary key keeps the directory of the key for the writes that supersede rows; an index built before keeps its directory until a merge decides again for the segments it writes, or `REINDEX` for all of them.
- Deprecated: pg_sift: the `sift_col` option `order_columns` is accepted and ignored: a build writes no sorted run in its units, and the order arrays (`order_array_columns`, chosen by the build) answer a top-N, a range, an `OFFSET` and an `IN` list over any number of units for the same bytes where the runs read 4 to 6 times the instructions beyond a top-N of 32. A `key_order` no longer asks for the runs. An index that carries the option keeps reading the runs it has, and `order_array_columns` no longer has to name columns of `order_columns`; `CREATE INDEX` and `REINDEX` say so in their NOTICE and in `sift.index_choices`; 4.4 removes the option.
- Deprecated: pg_sift: the `sift_col` option `point_columns` is accepted, and a build lists its columns in the directory of heap addresses (`point_tid_columns`), which reads no unit for an equality or a short `IN` list where the directory of rows read the unit of each row, for the size (within 3%). A release before 3.7 cannot read the segments written so. `CREATE INDEX` and `REINDEX` of an index that carries it say so in their NOTICE and in `sift.index_choices`; the segments already written are read as before; 4.4 removes the option.
- Deprecated: pg_sift: the `sift_col` option `key_cut_rows` is accepted and ignored: the units a `key_cut` build, promotion or merge writes aim at the rows of an average superblock, which kept the number of units and the planner's per-unit costs and gave no query a gain at any size measured. `CREATE INDEX` and `REINDEX` of an index that carries it say so in their NOTICE and in `sift.index_choices`; the segments already written are read as before; 4.4 removes the option. `sift.test_col_legacy_options` (testing only) makes a build honor the deprecated options until then.
- Improved: pg_sift: a joined table whose join columns are in two `sift_col` indexes, none of them holding all, is loaded from the leaves of both (as the classes of its keys or as rows) where it was read from its heap: the two indexes have the same units while no row was inserted into the table after one of them was made, and the load reads the heap again when they do not
- Improved: pg_sift: a `CREATE INDEX`, a merge and a rewrite of a `sift_col` index keep to `maintenance_work_mem` and write larger segments in it: the point directories and order arrays of a segment are built from the memory the rows already took (an integer column of 1,000,000 rows peaked at 89 MB, now 56 MB; the four columns of the 5,000,000 row test table at 64 MB of `maintenance_work_mem` 134 MB, now 68 MB, and at 256 MB 253 MB, now 233 MB with 2 segments, not 5), the segment is cut at the memory it will reside in (the writer, the batches and the allocator's) and glibc keeps the arrays it frees from staying in the process, so fewer and larger segments are written and a read has fewer to open.
- Fixed: pg_sift: `sift.col_merge(index, true)` on an index whose segments are already the partitions its bytes make (all of the partition size, none with a dead row, no more of them than the merge would write) does nothing: it used to read and write all of the index again each time it was called (62 MB of WAL for each call on a 5,000,000 row index), so a second call, or a loop of them, is now free.
- Fixed: pg_sift: a `CREATE TABLE AS` or `SELECT INTO` of a statement that the extension plans as it is and with its sub-selects pulled up failed with `XX000` `unrecognized node type` (and could read freed memory) the first time that statement was planned in a server, when the pulled-up plan won: the plan that lost left the statement's own tree with nodes of the memory it was planned in, which the event triggers of the extension copy afterwards; the planner now plans a copy of such a statement
- Fixed: pg_sift: a `NOT IN` of a list the planner hashes (a test PostgreSQL evaluates for each distinct value of the column) is priced for what it costs the scan: a call of the subplan and a probe of its hash for each distinct value (68 ns, priced at the 36 of any other expression), the distinct values counted before the test rejects any, and a miss of the cache for each tuple of a value met before once the values are many (1,000,000 rows in random order: 31 ns a tuple at 100,000 values, 183 ns at 632,000), where the scan was priced as if the values were few. A column of 1,000,000 rows and 447,000 values, `count(*) ... NOT IN (a list of 1,000)`, was priced 8,028 units for the scan against 27,135 for the normal plan and ran 156 to 258 ms against the plan's 142; it is priced 43,682 now and PostgreSQL keeps it; a column of 1,000 values (13 ms against 95) and of 100,000 (20 to 51 ms) keep the scan. With workers the scan is still chosen: PostgreSQL prices its parallel plan 1.5 times faster than the serial one and it runs 3.2 times faster (45 ms against 76), a known limit in docs/AGG.md. Test `agg_filter_hashed_subplan_price`. No format change.
- Fixed: pg_sift: a table of keys that only tells whether a key exists (EXISTS, NOT EXISTS, IN over sparse keys) is priced as the keys alone it is loaded as, without the directory of its keys that the load does not build, where the price charged each key a miss of that directory: a fact of 1,000,000 rows and a table of 1,000,000 sparse keys was priced at 327,178 units for the scan (the normal plan 90,166) and ran as the normal plan in 532 ms; priced for what it does the scan is 31,647 units and runs in 139 ms with the keys read from the heap, and 43,641 units and 68 ms with a column index on the table (the normal plan 90,166 units, 532-616 ms). Test `agg_join_keys_only_price`. No format change.
- Improved: pg_sift: a top-N, an `OFFSET`, a range or an `IN` list by an order array (`order_array_columns`) walks the arrays also when some segment of the index has none of the key (an index given the option after it was built, and the overlays written without them): the walk takes the live rows of such a segment as a source sorted in memory, as it does those of the write buffer, made from the chunk of the key and the heap addresses of its units only, for each scan, and the planner prices the rows of those segments for each one it reads and prices no walk past the cap of `SIFT_CGORD_FREE_MAX` rows (65,536 of a number or a time, a sixteenth of that for a text key), where it reads the units as before. Before, one such segment sent the walk of every row of the index back to the units. Test `col_topn_gorder`. No format change.
- Fixed: pg_sift: a lookup table that another lookup of the scan keeps rows of (the datasets of one tenant, reached through the dimension that names them) is priced for the rows its index finds for the keys the other keeps, as the executor loads it, where the price read the dependence from a copy of the join that had no key expression and charged the whole table: a fact of 1,000,000 rows joined to a dimension of 500,000 rows, a set of 1,000 and the 40 tenants, restricted to one tenant, read 22,609 rows of the dimension and was priced for 501,001; the price of the scan falls from 117,112 to 25,051 units (the normal plan 13,673), nearer the 26 ms the scan takes against the normal plan's 141 ms. Test `agg_join_chain_load_price`. No format change.
- Improved: pg_sift: a serial `CREATE INDEX` or `REINDEX` of a `key_cut` index on a table of more than one run (it needed more than three before) is built by ranges of the key, so a table of one to three runs no longer leaves runs that overlap in the key for the maintenance to rewrite after the build: the build writes the index once, and it is smaller.
- Improved: pg_sift: `min`, `max`, `sum` and `avg` of a column of a joined table (the lookup of a join) are added tuple by tuple for the integer, date, timestamp and `float8` columns, with PostgreSQL's own comparison and arithmetic, where each tuple was an entry in a set of the pairs of its group and the id of its value, whose size past every cache of a column with a value in each row (and, for `sum` and `avg`, whose pairs were then added one transition call each) was most of the scan. 5,000,000 children of 250,000 parents, each parent with its own value, grouped by a child column, one backend, minimum of three, `perf stat`: `min` and `max` of an integer column 2,681 M instructions and 1,142 ms to 1,700 M and 525 ms; `sum` of an `int4` column 2,400 M and 1,244 ms to 1,474 M and 514 ms (an `int8` sum raising `bigint out of range` as PostgreSQL does); `sum`, `min` and `avg` of a `float8` column 2,583 M and 786 ms to 2,057 M and 390 ms. A column of 100 values (a set of the size of the cache) was not slow and keeps its time. The planner prices these as it priced a column of few values. Test `agg_mapped_direct_values` (the answers are the normal plan's for NULL values, children without a parent, ties, a NaN and an infinity). No format change.
- Improved: pg_sift: the values of a column kept as a dictionary of numbers (a `numeric`, `bigint`, `timestamp` or `date` column of a few thousand values or fewer, and the codes of a text column) are looked up by their codes with one gather of four values on AVX2 and AVX-512, where the scalar loop took seven instructions a row for each row a scan reads. A sum and a count per group of 5,000,000 rows with a filter that keeps a quarter of them, one backend, `perf stat`: 263 M instructions to 235 M, the same buffers; the scans that split the rows between workers pay the lookup once for each participant, so they gain it as many times. The ARM and scalar tables keep the loop. Test `agg_dict_values_gather` (the answers at the scalar and AVX2 levels equal the normal plan's for filters that keep every row, a quarter, an eighth, a hundredth and none). No format change.
- Improved: pg_sift: a parallel aggregate whose scan joins one table by a plain integer, date or timestamp key splits the rows of the scan by that key (`Rows Split By: the key of a lookup join`): each participant reads every unit and takes the rows whose key falls in its share, so a key is resolved (its id and its rows in the joined table) by one participant, where each resolved every key it met and the work of the keys grew with the workers. 5,000,000 children of 100,000 parents, `count` per parent attribute through `RIGHT JOIN`, `perf stat` of the postmaster's processes: instructions with two workers are 1.11 times those of none (they were 2.04 times with the rows split), with four 1.21 times; wall time 116 ms (no workers), 62 ms (two), 58 ms (four). The planner prices the key split by the read of the key column that every participant repeats and picks it where it is cheaper than the row split (a scan of millions of rows); `sift.test_split` is a test setting. Test `agg_par_join_split` (NULL keys, children without a parent, parents without children, repeated keys, the leader taking the shares of workers that did not start). No format change.
- Changed: pg_sift: a joined table's `uuid`, `interval`, `bytea`, `inet` and `numeric` of more than 18 digits columns are read from the leaves of its `sift_col` index as classes of the values they take, where they sent the whole table to the heap (`Lookup Heap Reason`). 1,000,000 parents, 2,000,000 children, `GROUP BY` an `interval` column of 1,000 values: 1,099 ms to 28 ms (one backend, lookup caches off). A column with more values than a table of classes holds (4,096 tuples) is still read from the heap. Test `agg_lookup_wide_classes`.
- Improved: pg_sift: a `LEFT JOIN` of two tables that both have a `sift_col` index is planned also as the scan of the child with the preserved parent as its lookup (it was planned so only when the parent had no index), and the cheaper plan runs: the count of children per parent, the sum with conditions in the ON clause, and the grouped counts of a parent attribute take between a third and two thirds of the time they took with the parent scanned (a million children of a hundred thousand parents). The child grouped first is kept when it costs less, not only when it costs half; `sift.agg_child_first = off` turns both off. A statement such as `count(e.id)` over the nullable side now runs in the scan. Test `agg_left_child_scan`. No format change.
- Improved: pg_sift: the rows of a joined table kept for the next statement, the classes of a dimension and its shared map are valid by one rule (`agg/lookvalid.c`): the stamp of the journal of the table's connected index when it vouches for the snapshot, else the LSN of every page of the heap, taken before and after the read; and they are named by one key (`look_key`). No change of behavior and no format change.
- Improved: pg_sift: the tests of a joined table's columns that the load of its classes computes (`FILTER (WHERE dd.a = 1 OR dd.t IS NULL)`) are evaluated once for each tuple of the values they read and kept, not for each row of the table: 4,000,000 keys and three tests take 195 ms to class where they took 495 (the planner prices a test at half of what it did). The answers are those of the normal plan (test `agg_lookup_gang`). No format change.
- Improved: pg_sift: an overlay of a `sift_col` index (a segment under half the rows of the largest) is also written without the global-order arrays, while the overlays together hold at most a tenth of the rows of the index and no more than a walk of the arrays takes from them in memory (65,536 rows; a sixteenth of that for a text key; `order_array_columns`), which span the rows of the whole index; the arrays of the larger segments cover over nine tenths of the values, as the order planner asks, and the overlay is read by its rows. Fewer index pages and less index WAL are written for each row ingested, and the file is smaller; a per-key ordered probe is no longer slower with four overlays alive than with one. No format change.
- Improved: pg_sift: the segments that promotions of the write buffer leave over a `sift_col` index (those under half the rows of the largest) and the segments a merge of them makes are written without point directories, whose arrays span the rows of the whole index and cost more than the rows of an overlay (their order arrays stay), and get no net pre-aggregate sidecar for their dead rows (the next merge drops them); a merge into a larger segment writes the full set from the rows. Their merges take the smallest overlays that come to twice the largest (it was one and a half times), and more than four alive in an index that is read are merged whatever their sizes. Fewer pages are written for each row ingested; no format change (a segment stamps the format minor of what it holds), `docs/USAGE.md`.
- Improved: pg_sift: the classes of the keys of a big dimension table (an index of 640,000 rows or more by default, 8 times `sift.agg_par_min_rows`) are made by workers together when `max_parallel_workers_per_gather` is above 0 and the statement is a read-only `SELECT`: the leader and up to 14 workers take the index's superblocks as they come, and the classes, their numbers and the array of the keys are those of a load by one process. 5,000,000 keys: 138 ms in one process, 87 ms with four workers (minimum of 7). `EXPLAIN ANALYZE` shows `class workers`; test `agg_lookup_gang`. No format change.
- Improved: pg_sift: a scan of a `sift_table` reads the directory, write buffers, tombstones and versions it opened with through a state of its own, and the workers of a parallel aggregate scan open the table on the directory and write buffers the leader read, where each read the table's current ones; the answers do not change, a later VACUUM or invalidation can no longer reach a scan that has begun
- Improved: pg_sift: the rows of a write buffer of a `sift_col` index that tracks changes stay in shared memory (`sift.col_wbuf_ref`, on): a row goes to the buffer as a 16-byte reference to its heap tuple, its values are kept in a shared arena by the backend that inserted it and a reader takes them from there, where they were written in the pages of the buffer. 500,000-row index, 48 batches of 2,000 rows: index pages written 410 to 199 bytes a row with updates, deletes and `VACUUM`, 100 to 32 bytes a row for inserts alone; index WAL 506 to 162 bytes a row (inserts alone 76 to 15.5); WAL of all relations 5,970 to 5,617 bytes a row; the index file 37.9 to 31.4 MB for inserts alone and 64.1 to 42.3 MB with a snapshot held, and a `VACUUM` that waits for the lock of the cut of the free end does not wait the 5 seconds of its retry, because the free end it has to cut is 5.0 MB without references and 0.6 MB with them. The costs: an insert +0.2% instructions, a promotion +1.7%, a read of the buffer within 1% warm, the first statement of a backend +4% to +5% at 47,000 buffered rows, and the first read of a buffer after a restart, on a standby or on a promoted standby builds its entries from the heap once (about 11,900 instructions a row: 12,000 rows 65 M to 209 M instructions, 8 ms to 24 ms). The arena takes 88 bytes of shared memory a buffered row, in 1 MB segments made on demand up to 128 MB for the server, and `sift.col_arena_status()` shows it; `SET sift.col_wbuf_ref = off` writes the rows with their values as before. A write buffer that holds a reference is column format minor 44, which a release before 4.3 refuses.
- Fixed: pg_sift: the participants of a parallel unranked scan of a `sift` (BM25) index bind to the directory the leader exported before the workers start, where only a ranked scan did and the others opened the directory each for itself: the rows of the scan are those of its snapshot whatever a promotion, a delete or a `VACUUM` does after the leader exported. Isolation test `bm25_par_dir`.
- Fixed: pg_sift: the workers of a parallel scan of a `sift_col` index open their sources on the end of the heap the leader counted as well as on its directory, where each counted it again: the items of the scan are ranges of superblocks and mean what the leader's directory and heap end say.
- Improved: pg_sift: a lookup join to a table kept as classes of its keys writes the output row of each tuple of a chunk in one pass over the keys (one read of the class and one of the row of the class), where it made an entry for each tuple and read the entry back; a chunk with a key the table has no row for (an inner join) is resolved as before.
- Improved: pg_sift: the join to a table kept as classes of its keys resolves a chunk of the scan with a read of the class and a read of the entry of each row, and the aggregates and group keys that read the table's columns resolve the rows of it they have not met in a chunk alone, where one such row made the whole chunk resolve row by row. Same answers as the normal plan.
- Improved: pg_sift: a column of an inner-joined table that its ON clause equates to a column of the scanned table (an integer or date key) is read as the scanned column in the aggregates and group keys of a scan: `count(DISTINCT d.id)` over `d.id = f.d_id` counts a leaf of the scan where it scanned the joined table and looked the rows of the scanned one up (1,000,000 rows loaded), and the join to the table is then one whose columns are classes of its keys. Same answers as the normal plan.
- Fixed: pg_sift: a lookup join to a table kept as classes of its keys (an `int4` key and columns with few distinct tuples of values) answered a query that reads the key column of that table itself (`sum(t.k)`, `count(DISTINCT t.k)` over `t.k = f.k`) with the key of the first row of each class, one value for all the keys of it, where the normal plan gives each key's own: such a join is no longer kept as classes and is loaded row by row. The classes were used only with the lookup caches off or a table with a page that is not all-visible, and by the dimension map of this release with the caches on.
- Added: pg_sift: the reloption `connect` of a `sift_col` index, `WITH (connect = 'fk -> parent(key)')`, connects the index to the `sift_col` index of the table it joins (`docs/COL-CONNECT.md`): the child index depends on the parent index (`DROP INDEX` of the parent needs `CASCADE`), the parent index keeps a journal of the keys its updates, deletes and key changes ended (column format minor 44, which only 4.3 and later read), and the text is dumped, restored and resolved again by `REINDEX`. Nothing reads the connection yet.
- Added: pg_sift: a semi, anti or inner join to the parent of a connected `sift_col` index that reads no column of the parent (`EXISTS`, `NOT EXISTS`, a join that counts) is answered from the flag the segments keep that the parent row was found, less the keys the journal of the parent names and plus those of them the parent holds now, where the join loaded the parent; the ordinary join runs, with the same answer, when a write buffer holds rows, a segment has no stamp or, for a child with rows whose key the parent lacked, a parent row was inserted since (`docs/COL-CONNECT.md`, "Found"; a clean child, as a foreign key keeps it, is not slowed by writes to the parent); `sift.agg_connect_found` turns it off, EXPLAIN ANALYZE shows `Found Proofs`, `Found Keys Outside The Proof`, `Found Fallback`, `Parent Leaf Reads` and `Parent Heap Fetches`
- Added: pg_sift: `SELECT count(*) FROM parent WHERE [NOT] EXISTS (SELECT 1 FROM child WHERE child.fk = parent.key AND ..)` with a unique `int2`, `int4` or `int8` parent key is planned also as the count of the distinct keys of the child's rows that have a parent (`NOT EXISTS`: the parents' count less it), where the parents were scanned and the child loaded into a table for the join; the plan that costs less is kept, and a connected child answers the parent from its found flag (bench cases `invert_exists_child_filter` and `invert_not_exists_child_filter`; `docs/COL-CONNECT.md`, "Parent to child"); `sift.agg_child_first` turns it off
- Changed: pg_sift: the aggregates of a table left-joined to ONE table (`parent LEFT JOIN child ON child.fk = parent.key GROUP BY parent.attr`), not only to a join of tables, are made for each key of the child first and added up over the parents when that plan costs less (bench case `child_first_left_join_groups`; `sift.agg_child_first` turns it off)
- Added: pg_sift: `sift_conn_inserted_since` (`col/conn.h`) tells a reader that holds a stamp of a connected parent index whether a row could have been inserted since it: false only when every segment says its inserters finished before the stamp and the write buffer holds no live record from it on (an aborted transaction does not count, the caller's own insert and a transaction still open do). Tests: `test/sql/col_conn_inserted.sql`, `test/isolation/specs/col_conn_inserted.spec`.
- Added: pg_sift: the extension version is 4.3 (`ALTER EXTENSION pg_sift UPDATE` from 4.2 or a patch release of it); `sift.col_connections` lists the connected indexes (child, parent, the text of the option) with the state of the key journal of the parent, and `sift.col_connection_state(index)` tells it for one index.
- Improved: pg_sift: the classes of the keys of a joined dimension table (the table kept as classes of its keys, and the dimension map of this release) are made for the `text` (dictionary), `varchar`, `numeric`, `float4`, `float8` and `time` columns and the expressions a `sift_col` index holds or a test computes, as they were for the integers, dates, timestamps and booleans: a join to such a dimension was loaded row by row (text, numeric) or as classes that no other backend could share (an expression). Strings are told apart by their bytes, a float by its bits (-0 and 0 are two classes), a numeric by its scaled integer; the answers are those of the normal plan (tests `agg_dimmap_types`, `agg_dimmap_values`). No format change.
- Improved: pg_sift: the classes of the keys of a joined dimension table (an `int4` key and the columns a query reads, with few distinct tuples of values) are kept in shared memory (`sift.agg_dimmap_kb`, 64 MB by default, 0 turns it off) once a backend has read them from the leaves of the `sift_col` index of the table, and every later query of any backend reads them in place, where the classes kept by the lookup caches need a table whose every page is visible to every transaction. A map is read while the table is what it was: when the `sift_col` index of the dimension keeps a journal of its changed keys (a child index is connected to it with `connect`), a write to the dimension makes the next statement read the leaves again and keep a map of its own, without a vacuum and whatever the size of the table; otherwise a write, committed, rolled back or the reader's own, is a table with no map until it is vacuumed again. A map that does not fit the budget is not kept; the answers are those of the normal plan. `EXPLAIN ANALYZE` shows `Dimension Maps Shared` and `Dimension Maps Built`. A grouped count and sum of 1,000,000 facts joined to a dimension of 100,000 keys with 60 groups takes 76 M instructions (12 ms) with the map or the classes kept by the lookup caches, and 1,915 M instructions in the normal plan.
- Added: pg_sift: the reloption `connect` names several connections (`'a_id -> dim_a(id); b_id -> dim_b(id)'`, up to 8 in an index; the same parent may be connected by two columns), each with its own parent index, dependency and journal check (`DROP INDEX .. CASCADE` of any parent drops the whole child index), and a connected index holds one boolean leaf for each connection (`<column>#found`, the parent row was found) in column format minor 44, which 4.2 and older refuse (`docs/COL-CONNECT.md`). `sift.col_connections` has a row for each connection. Nothing reads them yet. Tests: `col_conn_declare`, `col_connect_dump.sh`.
- Added: pg_sift: the writer of a segment (build, promotion, merge) of a connected index fills the carried leaves from the parent rows visible to a snapshot taken after the stamp of the parent's key journal, and the segment keeps the stamp (`docs/COL-CONNECT.md`); a parent that does not fit `maintenance_work_mem` or has no journal yet leaves them NULL; `col_verify` leaves the carried leaves out; the journal vouches only for snapshots taken after its creation.
- Added: pg_sift: the automatic rewrite refreshes the segments of a connected index whose found flags aged (reason `stale connection` of `sift.col_maintenance_due`): a segment never filled, one whose parent index was rebuilt, or one whose parent keys changed since its stamp are `sift.col_connect_refresh_pct` percent (10) of the parent's rows; the rewrite stays within the WAL budget of `sift.col_auto_merge` (`docs/COL-CONNECT.md`).
- Improved: pg_sift: connecting a `sift_col` index to the index of its parent table holds the writers of the parent table out only from the end of the child's scan to the end of the transaction, and only for the first child of a parent (the journal exists for the next ones), where the lock was taken before the scan. Held by the isolation test `col_conn_lock`.
- Added: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index connect it by themselves (`sift.col_auto_connect`, on) to the `sift_col` index of the table each validated single column integer foreign key of its columns references, when that index tracks the referenced key, and say so in a NOTICE that names the foreign key and says how to carry columns. An automatic connection carries no parent column (the key mapping: a found flag leaf of a few hundred bytes and the stamp of each segment); `carries` is the user's choice. The choice is written into the option as `connect = 'auto: ..'`, which a dump and a restore keep without the constraint; `connect = 'off'`, `SET sift.col_auto_connect = off` and an explicit `connect` turn it off or win, `connect = 'auto'` asks for it with the setting off, and `sift.col_connections.automatic` tells. A foreign key that does not qualify says why in a NOTICE when the referenced table has a `sift_col` index. Tests: `col_conn_auto`, `col_connect_dump.sh`.

## 4.2.2 - 2026-11-09

- Fixed: pg_sift: a `VACUUM` of a `sift_table` that waited for the table's maintenance lock went on with the segments it had read before the wait. When the holder promoted or merged write buffers meanwhile, the `VACUUM` did not mark the dead rows of the new column groups as removed and then dropped the records of their deletion, so the old versions of updated rows, and rows that had been deleted, were visible again to every snapshot (a manual `VACUUM`, autovacuum, the background worker of `sift.table_auto_defrag` or two `VACUUM`s of one table; 4.0.0 to 4.2.1, `sift_table` only). Every maintenance entry now takes the lock and reads the segments in one step. The error `sift index "..." is corrupted ... expected page type/generation` that `VACUUM` or `sift.table_maintain()` raised in the same race aborts that call and left the tables examined consistent (a few cases); it does not by itself mean that rows reappeared. The update does not repair rows that came back: see the notes of 4.2.2.
- Added: pg_sift: `sift.table_find_stale_versions(tbl regclass, sample integer DEFAULT 10)` returns the number of primary key values that are on more than one row of a `sift_table` and up to `sample` of them (one sequential scan, `ACCESS SHARE`, no writes). It cannot see deleted rows that came back.
- Fixed: pg_sift: assert builds stop where the segment switch of a merge of a `sift` index finds none of the segments it replaces in the directory.
- Fixed: tests: the update test of 4.2.1 starts from 4.2.0, not from the newest release tag, and the landing script fetches the tip of the base without failing when another landing moves it.

## 4.2.1 - 2026-11-09

- Known issue: pg_sift: the tap-shell test `col_index_replica` (test 7, the standby answers the units of a merged segment from the exceptions) may fail on some runners, because the standby plan can vet units by heap; the failure is in the test, fixed in 4.2.2, and no data or answer is affected.
- Fixed: pg_sift: the library takes no relcache callback slot again: a backend has ten for all its libraries and a session that ran out of them died with `FATAL: out of relcache_callback_list slots`; the shared-CTE source cache, the tombstone log cache, the `track_changes` trigger memo and the removal of the triggers of a deleted index (4.2.0 and the 4.2.1 builds before this one) took up to four. They follow `pg_class` through syscache callbacks, which have 64 of their own, and the triggers of a deleted index are noticed by the object access hook. `scripts/check-relcache-slots.sh` fails the lint on a call.
- Fixed: pg_sift: code that held a string of an index's options (or the state it caches) across a lock or a catalog lookup read freed memory when that lookup reloaded the index's cache entry, which a `VACUUM (ANALYZE)` or an `ALTER INDEX` of another session makes it do: the planner's check for a point lookup of a `sift_col` index, the choices of a build, the automatic key cut, point and path decisions of a build or a rewrite, and the `bm25` insert path, ranked scoring and build. They copy it before the first lock (a crash, or a wrong plan or answer; 3.6.0 to 4.2.0).
- Fixed: pg_sift: a parallel aggregate whose lookup join runs a child plan in the workers (`JOIN LATERAL (SELECT .. FROM t WHERE t.id = x LIMIT 1)`) that reads a `sift_col` index of `t` missed the rows the transaction itself had inserted into `t` after its first one (they are held back in the backend until the index is read, and a worker has none to flush). The leader flushes them before it launches the workers (3.6.0 to 4.2.0).
- Fixed: pg_sift: a plan that scans a `sift_col` index under a `Gather` (an index scan in a worker, from a plan made before the rows were inserted: a prepared statement or a function) missed the rows the transaction itself had inserted after its first one: they wait in the backend and a worker has none to flush. A statement that needs parallel mode flushes them (and the held-back tombstones) when it starts (3.6.0 to 4.2.0).
- Fixed: pg_sift: a parallel aggregate over a `sift_col` index missed the rows the transaction itself had inserted (every row of a statement after its first, up to the next read of the index) when the leader did not scan (`parallel_leader_participation = off`, or workers that were first to run): the rows wait in the backend until the transaction reads the index and a worker has none to flush, so the answer lacked them from the second transaction of a session on (the first read of a session flushed them). The scan flushes them when it opens (3.6.0 to 4.2.0).
- Fixed: pg_sift: a join to a parent table whose keys are spread over a range larger than the lookup budget (`sift.agg_lookup_max_kb`) is priced as a probe of a hash table for each tuple, not as the array of classes of a dense key that the executor cannot build for it; the price was a fifth of the time (a 2,000,000-row parent with keys up to 2 billion, 4,000,000 tuples: 25.9 thousand cost units for 4.2 s, now 111 thousand), so a plan that does not pay was chosen against the normal plan's.
- Fixed: pg_sift: a lookup table that repeats a key is priced as rows, not as classes of its keys (the load refuses such a table), and a table read as rows pays the cache misses of the directory of its keys (98 ns a row for 400,000 keys of 4,000,000 rows, 543 ns for 4,000,000 keys): three FILTER aggregates over a join of two tables of 4,000,000 rows chose the scan that loads the table with 2,500,000 distinct keys, 6.4 to 8.4 s against the normal plan's 4.9 s, and now scan the other table, 420 to 530 ms.
- Fixed: pg_sift: a join to a table kept as classes of its keys answered wrongly when a test read the join's key next to the table's columns (`count(*) FILTER (WHERE f.k > 9000 AND t.a IS NOT NULL)` over `t.k = f.k` counted none of the rows it keeps, a `CASE` or a `GROUP BY` of an expression of the key too): the entry of such a test was taken by the output row of the table, which is the row of the first key of the class, so the key read as that one. A test of the table's columns alone still takes its entry by the row. Not in 4.2.0 (the tables kept as classes are new in this release). Test `agg_lookup_tests`, shapes 10 to 14.
- Fixed: pg_sift: the check of a `sift_table` (`sift.col_verify`) no longer reports "The summary of the versions of unit 0 is not what its vectors hold" for a group that VACUUM froze after a committed `COPY`: VACUUM clears the flag of the version section of a group whose rows are final and leaves the section in the file, and the check compared that section with rows that read as frozen. Scans were never affected: they read such a group as frozen. Test: `test/sql/tbl_verify_frozen.sql`.
- Fixed: pg_sift: an `EXISTS` or `NOT EXISTS` over a `sift_table` (a table of another access method) is no longer taken by the scan as a probe of the table's btree for each row of the scan: the probe was priced as an Index Only Scan of a heap, an entry for 3 operators, where every entry of such a table is a fetch of its own through the access method (14,000 instructions, 2.8 us), so the scan did the nested loop of the normal plan and no less, in one process where the normal plan used its workers. The entry is priced at what it takes and the normal plan wins (50,000 dimension rows against a 1,000,000-row `sift_table`, `NOT EXISTS`: 917 M instructions and 244 ms by the probe, 853 M and 83 ms with the normal plan's worker; 835 M and 141 ms without workers; the same probe of a heap table stays, 277 M instructions against 420 M). Test `agg_probe_table_am`.
- Fixed: pg_sift: the rewrite of a `sift_col` index chooses the key again from the data (`key_cut`, the order arrays, the point directories of the options the build chose) whichever segments its first step takes; it did so only in a step that took every segment, so an index whose first step took the overlays alone (two overlays and a dirty main segment) kept the choice of its build, in a session's `sift.col_maintain` and in a worker alike, and the step that finds the heap locked by a VACUUM is left to the next call instead of being made without the choice.
- Fixed: pg_sift: a `key_cut` build of a table with a value that holds a large share of the rows (a third of 3,200,000 rows: one segment of 1,067,520 rows, 42 times an even split; a value of 100 million rows would hold gigabytes of rows and arrays) reads the rows of that value back in runs of the memory of one, and the segments are those of an even key (47,360 rows at most).
- Fixed: pg_sift: a list page of one group under a test of another column (`WHERE tenant_id = 17 AND st < 100 ORDER BY date DESC LIMIT 20`) walks the order array of the group when most rows pass the test (2.68 to 1.81 M instructions, 25,000 rows to a group; 3.63 to 1.96 M with the primary key ending the order) where the plan read the units the zones leave, which it priced by the rows the filters pass, not by the rows the units hold.
- Fixed: pg_sift: a `sift_col` index whose tombstone log has a column format this build does not read is refused with an error and a hint when the log is opened (a write to the table, a read of the log), where an older build appended tombstones to it without the records a newer build keeps in the log; a database written by a newer release is not written by an older one by mistake. Test: `test/sql/col_log_newer_format.sql`.
- Fixed: pg_sift: the default `key_cut` of `CREATE INDEX` and `REINDEX` prices the zones that every other leaf of the index would lose to the cut, not only those of a leaf that follows the order of the table: a leaf whose zones skip some of the units (its equality keeps a third of them) costs the cut what the skipping was worth, by the share of its rows that have a value, so a cut by a scattered date no longer wins against the lookups of a key that is mostly stored together.
- Fixed: pg_sift: the workers of a parallel aggregate read the directory of the index the leader read, where each process read its own copy: a promotion of the write buffer, or a merge, between two opens moved rows from one item of the scan to another (the units of a key-cut segment are numbered by their key range, the batches of the write buffer by the superblock of the heap), and a statement that ran while rows were being appended counted some rows twice and some not at all. The same holds for the partitions of a partitioned table. No format or option changes.
- Fixed: pg_sift: a scan of a `sift_table` filtered the pages it read with the write buffers and the tombstones that the backend held at that moment, where its source had read both when it opened: when the table's cache was invalidated in between (by a VACUUM of the table, noticed when the scan locked a joined table), the rows deleted or updated that a write buffer still held came back, and an aggregate over the table joined to a lookup answered more rows than the snapshot holds, in a parallel scan whose leader had opened before the VACUUM, and in a serial one. The scan filters its pages with the state it opened with; sequential and group scans of the table too.
- Fixed: pg_sift: the workers of a parallel sequential scan of a `sift_table` read the tombstones of the table as they were when each worker opened, where the leader had taken the directory and the write buffers when it prepared the scan: a VACUUM in between dropped the tombstones of rows that the leader's write buffers still held, and the scan counted rows that the snapshot has deleted or updated. The workers read the tombstone log of the generation the leader's directory names.
- Fixed: pg_sift: the heap check of a BM25 index on a `sift_table` (`count(*)`, scans, facets) judged the pages with the tombstones of the cache entry the backend held when it asked, where its scan had read them when it began: when a VACUUM of the table invalidated the entry in between, and an older snapshot kept the log entries of rows deleted before the query's snapshot, the new entry had read none and those rows were counted. The check uses the tombstones its own scan read.
- Fixed: pg_sift: the addresses of the dead rows of a segment, listed for the scans of its global order (the skip of an `OFFSET` in a key-cut segment, the probe of the heap by a compact order block), are listed only when they fit `work_mem`; a segment with more has none listed: its unit cannot say, the heap confirms the entries, and the compact block is not used, where the list took memory by the dead rows of the segment.
- Fixed: pg_sift: the index-only scan in the order of the first column, or of the first two, holds at most `work_mem` of what followed the table: the rows it sorts itself (an order with compact chunks, or the ends only) go to a tuplesort that spills, the rows it keeps for the restore of a merge join to a temporary file, and the groups of a pair are merged as they are read; the group table of an order block is read through the buffers past a small size, and `sift.col_verify` reads the groups in windows.
- Fixed: pg_sift: a worker of a parallel aggregate sends its table in messages of about the limit `blob_bytes`, a slice of its groups each, where a table with a `count(DISTINCT)` of more than a hundred million pairs was one message larger than a buffer can hold; a group whose partial result alone is more than four times the limit ends the query with an error that names `max_parallel_workers_per_gather`.
- Fixed: pg_sift: the skip of an `OFFSET` over an order read group after group (`ORDER BY group, key`) checks the entries of one group at once up to the limit `skip_pending`; a longer group is walked with the groups after it, where it allocated arrays as long as the group, however long it was.
- Fixed: pg_sift: the node that groups the rows of a plan (`Custom Scan (SiftAggRows)`) ends with an error that names `work_mem`, `hash_mem_multiplier` and `sift.agg_rows` when its groups pass the bytes a scan that cannot spill may take, where its table of groups, which the planner can only guess for a join, grew with no bound until the server ran out of memory.
- Fixed: pg_sift: the group table of a scan that cannot spill (a two level aggregate, grouping sets, an ordered scan, a generic aggregate) is bounded at run time by the same bytes the planner leaves such a scan to the normal plan for (`group_nospill_factor` times hash memory), counting the groups at the bytes the planner prices them and not the sets and values the aggregates keep beside the table (so a scan that ran in 4.2.0 does not end for them): past them it ends with an error that names `work_mem` and `hash_mem_multiplier`, where an estimate made on old statistics or a prepared plan let it grow until the server ran out of memory.
- Fixed: pg_sift: `WHERE group = x AND <test of another column> ORDER BY key DESC, id DESC LIMIT n` on an index with `key_order = 'group, key'` (or the automatic grouped order array) read the units of the group through their sorted runs instead of walking the group's order array, because the plan priced the walk of the entries of every group and not the pinned one: the list page of one entity with the primary key ending the order and a soft-delete test now walks the array, as the page of `ORDER BY key DESC` alone did.
- Fixed: pg_sift: `sift.key_cut_decision` says what `CREATE INDEX` decides: it takes the same two steps (the first look at 8 superblocks, then the full sample) where it always read the full sample, so on a table whose first look ends the decision the function and the build gave different answers (a cut by an arrival timestamp reported, none built); its `sampled` column is the step that decided.
- Fixed: pg_sift: `VACUUM` of a table with two or more large b-trees, which runs the cleanup of the `sift_col` index in the leader under parallel mode, failed with `cannot assign transaction IDs during a parallel operation` when the index had a tombstone log that a `REINDEX CONCURRENTLY`, a restore or `pg_upgrade` left incomplete (the repair of the log, the baseline, needs a transaction id). The repair now waits for a `VACUUM` without parallel workers (`VACUUM (PARALLEL 0)`, autovacuum, the maintenance worker) and the reads check the heap until then. Held by `col_vacuum_parallel_baseline`.
- Fixed: pg_sift: the default `key_cut` of `CREATE INDEX` and `REINDEX` weighs a leaf that follows the order of the table by the rows that have a value in it: the zones a cut would widen are a cost of the cut, priced like a benefit and scaled by the share of non-`NULL` rows, where any such leaf ended the cut of every key; a soft-delete timestamp filled in 0.2% of the rows no longer decides the whole index, and a filled leaf still does. `sift.key_cut_decision` names the leaf when it alone turns the cut down.
- Fixed: pg_sift: the merge, the rewrite and the VACUUM of a `sift_col` index wrote a segment (or a partition) of as many rows as its bytes allowed, and the writer of a segment keeps the keys and addresses of every row for the point directories and the order arrays: an index of narrow rows (a few bytes of segment for each row) held several times `maintenance_work_mem` in one pass, and one of hundreds of millions of rows could ask for an array over 1 GB. The partitions are now sized to `maintenance_work_mem` by the memory a row takes in the writer, and a writer is cut at that memory or at a fixed number of rows (`SIFT_CPOINT_BUILD_ROWS`), the segments that are left taking what the rows left need. An index of narrow rows is kept as more, smaller partitions than before. No format change.
- Fixed: pg_sift: the automatic rewrite of a `sift_col` index waits for the snapshots that hold back the blocks of its last pass once those blocks reach one pass (`sift.col_auto_merge_pass_mb`) or half the index, whichever is less, where it waited only at half the index: a held horizon (a standby with `hot_standby_feedback`, a long query, the transactions of `REINDEX CONCURRENTLY`) took the file well past the live bytes plus a pass before a rewrite stopped, and now holds it to the live bytes plus one pass. Test: `test/t/col_auto_merge_held.sh`.
- Fixed: pg_sift: a `VACUUM` of a `sift_col` index with `sift.col_auto_merge` on gives the free space back to the file system when no rewrite is coming, with a standby streaming or not: it left the segments that end the file to the next rewrite whenever copying them down wrote more than half of the blocks it gives back, so a file left loose by the last pass of a build's runs, or by a horizon that held the blocks of a rewrite and passed, stayed at its size for good (a primary with a streaming standby cut no block however often it was `VACUUM`ed). The file is back near the live bytes in a few `VACUUM`s (the first moves the segments, the cut follows once the horizon has passed what moved); an index whose rewrite is due still leaves the move to it. Tests: `test/t/col_vacuum_space_idle.sh`, `test/t/col_auto_merge_space.sh`, `test/t/col_vacuum_defer_gate.sh`.
- Fixed: pg_sift: the sweep of the orphan blocks of a `sift` index (after a killed write) built an array with an entry for each orphan block (gigabytes for the orphans of a large index); it builds one with an entry for each run of consecutive blocks, the whole extents a write leaves (test `store_sweep_runs`)
- Fixed: pg_sift: `CREATE INDEX CONCURRENTLY` and `REINDEX CONCURRENTLY` of an index on a `sift_table` held a TID for every row of the table while it validated the index; it holds the TIDs of the first pass a batch at a time within `maintenance_work_mem` and scans the table once for each (test `tbl_validate_batches`)
- Fixed: pg_sift: a scan of a `sift_table` that reads a wide column (a high-precision or unconstrained numeric, money, an interval) whose distinct values in the rows of one unit take more than an allocation may hold ended with `invalid memory alloc request size`; it fails with `54000` naming the column (test `tbl_wide_dict_limit`)
- Fixed: pg_sift: `sift.workers`, `sift.col_index_status` and the functions that open a `sift_col` or `sift` index by name (`sift.col_segments`, `sift.col_sections`, `sift.col_extents`, the maintenance calls and the like) read the metapage of an index that `CREATE INDEX CONCURRENTLY` or `REINDEX CONCURRENTLY` had not finished (or that a failed build left) and failed with `invalid page layout` (block 0 with lower, upper and special 0) or `could not read blocks`, on the primary and on a standby, where the build had not yet written the metapage (it writes it last, past shared buffers); the two views failed for every index of the database while it lasted. The views list only indexes that are ready, and the functions say `sift index ... is not ready to be read` until the build ends. The WAL of the build is complete and in order before the catalog commit (the files of the standby equal the primary's after every kind of build); the view update is in `pg_sift--4.2--4.2.1.sql`. Test `col_standby_building_index`.
- Fixed: pg_sift: the choices `CREATE INDEX` makes before it reads the table hold no memory by rows and no wrapping product at any size of the table: the elements of an array column are sampled one at a time (an array of a gigabyte held more elements than a `Datum` array may), at most 16 array columns are looked at for `paths = 'auto'` (128 MB of sets, where 128 columns took a gigabyte), a map of thousands of keys in a document is found by its key in log time where the sample of the paths scanned every key it had met, and the shares of units that decide `key_columns = 'auto'` are 64-bit; `test/sql/col_decide_scale.sql` runs the choices with a row estimate of 257 million, a billion, four billion rows and the most a float holds.
- Fixed: pg_sift: the tombstone log of a `sift_table` is read and sorted in plain arrays of 32 bytes an entry: past 16.7 million entries (33.5 million on the first look of a backend) no backend could open the table, and VACUUM, which reads the log the same way, could not shrink it. The free list of the store (past 33 million records), the bounds of a prefix over many terms, the ties of a keyed `bm25` search, the values of a facet and the orphan sweep failed the same way at 1 GB; they are huge allocations now. No format change.
- Fixed: pg_sift: a build or a read that holds an array of 8 bytes or more for each row, group or entry of a segment (the fallback of an ordered index scan, the group table of an order block, the dead keys of a key-cut segment, the groups of a row-by-row aggregate, the pairs of a `count(DISTINCT)` that is dumped, the merge of old key-cut segments, the rows of one key over 67 million) ended with `invalid memory alloc request size` at 1 GB; they are huge allocations now. A build whose heap has more rows than its segments hold says so before the scan. No format change.
- Fixed: pg_sift: a `CREATE INDEX` or `REINDEX` of a `sift_col` index with `key_order = 'group, date'` and no key cut ended after the whole scan with `invalid memory alloc request size` on a table of hundreds of millions of rows: the writer cut a segment at its memory budget without counting the keys and addresses it keeps for the point directories and the order arrays, so one segment held every row and built its arrays at the end. The budget now counts them, with the arrays the end of a segment builds from them; a segment is also cut at a fixed number of rows (`SIFT_CPOINT_BUILD_ROWS`), whatever the budget, and a budget too small for the rest of the table in the segments the directory has left grows to what they need, where the last segment held what no budget cut; an array over 1 GB is a huge allocation (`core/rowmem.h`). A segment of an index with many order arrays and point directories is cut earlier than before, at the budget (`maintenance_work_mem`) the build was always meant to keep. No format change.
- Fixed: pg_sift: a `CREATE INDEX` or `REINDEX` of a `key_cut` index with a `maintenance_work_mem` of 1.5 GB or more wrote runs that overlap in the key, as with `sift.col_build_key_ranges = off`, because a first run over 768 MB turned the build by key ranges off: the automatic rewrite then cut them by key into new blocks while the old ones waited for the snapshots that could read them (a streaming standby with `hot_standby_feedback`, a long transaction, the `REINDEX CONCURRENTLY` itself), and the file stood 1.8 times its live size after the first pass and 2.8 times with a snapshot held through three (300,000 rows, 13 columns), where the build by ranges ends at 1.01 times. The runs of the ranges are now cut at 768 MB of memory, whatever the first run held (`docs/OPTIONS.md`, `sift.col_build_key_ranges`). No format change: REINDEX to apply.
- Fixed: pg_sift: the default `key_cut` of `CREATE INDEX` and `REINDEX` cuts the units by the key whose lookups gain most, where it cut by the first key-shaped column of the index that qualified (a column of three values listed before the entity won, and its lookups read 146 and 45 of 235 units where the entity's read 235), the leading column of a `CLUSTER` of the table counting one unit of the default formula more; a column that is all `NULL` is no longer a leaf that follows the order of the table and blocks the cut of every other key; and `sift.key_cut_decision` reports the reads of the leaf it names, not of the first key of the index.
- Fixed: pg_sift: a `key_cut` `sift_col` index in partitions rewrote all of itself each time the segment that promotions of new rows grew to reached half a partition: it overlaps every partition in the key, as the runs of a build do, and was taken for them, so the rewrite skipped `sift.col_auto_merge_wal_pct` and took the index in windows of `sift.col_auto_merge_pass_mb`. Only overlapping segments that make up at least an eighth of the index (the bytes whose removal leaves none overlapping) are a build's runs now; a grown overlay waits for the WAL budget like any other churn. 24 inserts of 2,000 rows (and 10% updates) into a 1,000,000-row index: 21.7 KB of index pages written per inserted row in 4 rewrites, 4.3 KB with none (1.1 KB once the rewrite of the build is behind); test `col_key_cut_grown_overlay`.
- Fixed: pg_sift: `DROP INDEX CONCURRENTLY` of a `sift_col` index with `track_changes` (the default) failed with `must be first action in transaction`, so a migration had to turn the option off first (4.0.0 to 4.2.0). The index now depends on its two triggers, where they depended on it, and the transaction that deletes the index removes them: `DROP INDEX CONCURRENTLY`, `DROP INDEX` and `ALTER TABLE .. DROP COLUMN` leave no trigger behind (a rollback leaves the index and both; a table that is in use at the end of a concurrent drop is not stopped: a background worker removes the triggers a moment after its readers are gone, and they do nothing meanwhile), a writer beside the drop skips the index once the drop has marked it dead, and `REINDEX CONCURRENTLY` still carries the triggers over to the index that stays. `ALTER EXTENSION pg_sift UPDATE` (4.2.1, `sift.col_track_convert()`) converts the indexes of earlier releases: until it runs such an index still refuses the concurrent drop (its triggers are found, and its log is trusted, meanwhile; an index made or dropped before the update behaves as after it). `DROP TRIGGER` of one of the two triggers is refused: the index depends on it. The update takes `SHARE UPDATE EXCLUSIVE` on each table it converts, so it waits for a concurrent drop or rebuild of an index of the table instead of racing it.
- Changed: pg_sift: the automatic `key_cut` (and the cut a `key_order` implies) leaves a key alone when its statistics, or the sample without them, say one value holds more rows than a segment holds (2^26 rows); the NOTICE of the build and `sift.index_choices` name the key and the rows (`key-cut-hot-value`), and `key_cut = 'column'` builds the cut anyway.
- Changed: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index with `key_order = 'group, key'` and no `key_cut` cut the units by the group (`key_cut = 'auto: group, key'`, said by the NOTICE of the automatic choices) where the group's lookups read many units, as the plain rule decides for any key: the order arrays alone left the count, the range and the group-by of a group reading every unit like an index with no layout. `key_cut = 'off'` or a column of your own keeps the units by block range; `sift.key_cut_decision` reports the same.
- Changed: pg_sift: `VACUUM` of a `sift_table` holds its dead rows a batch at a time within half of `maintenance_work_mem` (`autovacuum_work_mem` in an autovacuum worker): the TIDs of the write buffers and of the log, the roots of the HOT chains that go on and the runs of aborted rows of the groups were three lists that grew with the table; when they are full the batch clears the liveness of its group rows and every index goes through its TIDs, then the next batch starts, as the index passes of a heap `VACUUM`; `VACUUM VERBOSE` reports the batches. The index validation of `CREATE INDEX CONCURRENTLY` holds its TIDs within `maintenance_work_mem` exactly (the list doubled past it). Test `tbl_vacuum_batches`.
- Changed: pg_sift: the recount of the dead postings of a `sift` index segment (`VACUUM` once enough of its documents were cleared since the last count, and `sift.verify`) holds the dead rows of the superblocks a window at a time within half of `maintenance_work_mem`, and reads the dictionary again for each window; it held them all until the end of the call. Test `bm25_dead_windows`.
- Changed: sift for SQLite: `sift_facets` holds the pages with a match, the values counted and the rows of the answer within `sift_work_mem()` (a new function in kB, which also bounds the hits of a ranked search without a `LIMIT`) and fails naming it past it; it reads the postings of a term in batches of a superblock (the stream of the merge) where it gathered a whole term first, with its positions, and copies each value once. Test `test_facets_stay_within_work_mem`.
- Changed: pg_sift: what a statement holds for the matches of a search, which follows the data, has a budget: `sift.facets` (the values of the keyword index and the matches), a ranked search without a `LIMIT` (a hit for each match) and a search ordered by keys and then by `sift.score()` (the rows of the first key and the rows tied with its last) hold it within `work_mem` and fail with `54000` naming `work_mem` past it, where they took what the data asked for until the server ran out. The planner leaves the keyed scan out when the rows of its first round do not fit `work_mem`, and the ranked scan serves the query (test `bm25_budgets`)
- Changed: pg_sift: `sift.verify` and `sift.segments` no longer hold an array for every document of a segment or for every posting of the hottest term, nor the whole position stream of a term: they look the length of the documents of a posting up in the superblocks the term touches, kept within `maintenance_work_mem` as the merge keeps them, and read the position records through a window, so they need the same memory for a segment of any size (test `verify_doc_cache`). The heap check of `sift.verify(index, true)` streams the postings it adds to its filter instead of gathering a term
- Changed: pg_sift: the tombstone log of a `sift_table` that one backend keeps in memory (64 bytes an entry, the entries of its own statement included) is bound by `sift.tomb_log_limit`, 8 GB by default: a read or a statement past it fails with `54000` naming the setting and `VACUUM`, which is not bound by it and drops the entries of the rows it cleared, and the background pass is asked for when the log reaches a quarter of the limit, where it waited for enough reads of the table to pay for it (test `tbl_tomb_log_limit`). Earlier releases kept the whole log whatever its length
- Changed: pg_sift: `sift.col_sections` lists the point directories (`point_dir`) and the global order arrays (`order_arrays`) of the segments in rows of their own, where `dict_other` held both: `dict_other` is now what is left of the dictionary section (headers and records). The rows still add up to `pg_relation_size`.
- Changed: pg_sift: VACUUM of a table with `sift.col_auto_merge` on no longer copies the segments of a `sift_col` index down into the holes of the rewrite before (a second write of the whole index, 133 MB of 133 MB in the measured run) while a rewrite of the index is coming; it cuts a free end only, and the rewrite takes the holes, so a rewrite writes the index once. With no rewrite coming the VACUUM moves them (see the next entries). `sift.col_auto_merge = off` keeps the copy; test `col_auto_merge_space`.
- Improved: pg_sift: the `float4`, `float8` and `time` columns of a joined table are read from the leaves of its `sift_col` index, as the integers, dates, timestamps, booleans, numerics and texts were (a table that has one of them in its join read the whole table from the heap), and `time` tells classes apart too. Measured per row of a 100,000-row table joined from 1,000,000 rows, one backend, instructions: `float8` and `time` 2,253 where the heap took 2,813 and 2,827. A table that is still read from the heap though it has a `sift_col` index says why: `EXPLAIN (VERBOSE)` prints `Lookup Heap Reason: column u (uuid) is not a plain leaf (a wide type, a lossy leaf or a path) of index ..` (and `DEBUG1` logs it). The specials of the floats (NaN, the infinities, -0, a denormal, NULL) and the times come out as the heap's. Test `agg_lookup_leaf_types`.
- Improved: pg_sift: `count(x)` of a column of a joined table keeps a flag for each joined tuple, not the id of its value (a column with a value in every row made a table of the size of the lookup table, and the scan of a parent LEFT JOINed to its children, counted for each parent with the ten largest, ran 905 ms against the normal plan's 502 ms: it was chosen and lost); `min`, `sum` and `count(DISTINCT)` of such a column, which do need the ids, are priced for that table. Measured with perf stat on 1,000,000 child rows of 100,000 parents, default `work_mem`, one backend with the lookup caches off, minimum of five: the scan runs 562 M instructions and 2,139 M cycles (it ran 1.1 to 1.65 G instructions), the normal plan 1,642 M and 3,197 M; every block in shared buffers (13,685 hit, none read).
- Improved: pg_sift: the classes of a dense table of more than 524,288 keys are asked for 64 keys ahead, as the tuples of a join resolve against them (the array is past the cache of a core): a group by a parent attribute of 4,000,000 tuples against a parent of 4,000,000 keys ran 102 ms before and 48 to 59 ms after, minimum of five, three runs; a parent of 100,000 keys, whose classes are in the cache, is not prefetched and keeps its 9 ms.
- Improved: pg_sift: a join to a table kept as classes of its keys resolves the rows of a chunk in one loop of two array reads each (the class of the key, the entry of the class), where the general loop tested each row for a NULL, for the value before it and for the kind of its leaf. A group by a column of the joined table over 1,000,000 rows and 100,000 keys takes 44 M cycles and 101 M instructions where it took 72 M and 119 M, three `FILTER` aggregates over its columns 88 M and 220 M where it took 124 M and 237 M (the same 898 buffers hit). A version of the loop through the gather kernels was measured and is slower (73 M and 146 M cycles): the work is the random reads of the class array, which a gather does not make cheaper. Tests `agg_lookup_tests` (a NULL key among them), `agg_lookup_dense`.
- Improved: pg_sift: distinct counts of one parent key under FILTERs of the parent's own columns (`count(DISTINCT parent.id)` and `count(DISTINCT parent.id) FILTER (WHERE nullif(btrim(parent.a), '') IS NOT NULL)` of a join on the key, grouped by a column of the rows) share one set of ids: the scan groups by the outcome of each filter and counts each id once, and the statement adds up the counts of the groups each filter keeps, where the scan kept a set of ids for each count. The plan is kept where it costs less than the one of a set for each count; the answers are exact and equal to those of the normal plan.
- Improved: pg_sift: a report of one list (an `EXISTS` over a table the scan reads through its own btree conditions) reads the parent table for the keys of that list alone, where it classed or loaded the whole table, as fast as the normal plan's own index lookups.
- Improved: pg_sift: the tests of the columns of one joined table (a `FILTER` or `CASE` condition, a boolean group key: `dd.a = 1`, `dd.b < 5`, `dd.c IS NULL`) are columns of its lookup table, which are computed once for each row of the table from the leaves of its index and not for each tuple of values the scan reaches, and a table whose tests take few outcomes is kept as classes of its keys by them, not by the tuples of its columns (7,000 here, past the 4,096 a table may tell apart). Three `FILTER` aggregates over the attributes of a parent table of 1,000,000 rows from a table of 10,000,000 rows take 124 ms and 2,324 M instructions where the normal plan takes 1,846 ms and 15,748 M (the scan took 3.8 s and 12,547 M, as it was forced, and the planner kept the normal plan), and from 1,000,000 rows by a parent of 100,000 keys 22 ms and 247 M instructions where it takes 325 ms and 1,789 M. A test that may raise an error, the ON clause of the table, and a query with a right or full join are left to the combos as before; the kept tables are named by the text of the tests. `EXPLAIN ANALYZE` shows `rows classed` (test `agg_lookup_tests`, case `lookup_dense_tests`).
- Improved: pg_sift: the key column of an inner join in the argument of an aggregate (`count(DISTINCT parent.id)` of `parent.id = child.parent_id`) is read as the scan's column it equals, so a distinct count of it keeps a set of the column's values like any other leaf, not a table of the joined rows for each aggregate: seven such counts with FILTERs of the parent's columns over 20,200,000 rows, 26 s of CPU before and 4.4 s after.
- Improved: pg_sift: a conjunction of tests that an index holds as one boolean leaf (`lat IS NOT NULL AND lon IS NOT NULL`) is read from the leaf, and so is the disjunction of the opposite tests (`lat IS NULL OR lon IS NULL`) as its complement, in a FILTER or a WHERE of the scan and of a joined table.
- Improved: pg_sift: the expression of a lookup table that its index holds as a column may be a test or a condition (a boolean leaf of `nullif(btrim(col), '') IS NOT NULL` per text column), the `IS NULL` of the same text is the complement of that leaf in the scan and in the lookup, and the key of an inner join is the leaf it equals (`count(DISTINCT parent.id)` reads no column of the parent): the FILTERs of a presence report written as the columns' tests are read from the index.
- Improved: pg_sift: a lookup table that is kept as classes of its keys is read from the cache of an earlier statement as classes of its keys, with the caches at their defaults, where the caches loaded the table row by row and the classes were made only with them off. A `GROUP BY` of the level (60 values) of the parent of 1,000,000 rows by a parent table of 100,000 keys takes 105 M instructions and 7 ms where it took 246 M and 41 ms, and three `FILTER` aggregates over two of its columns 262 M where they took 475 M; the first statement makes the classes (12 ms), the next ones read them from the backend's cache or the shared memory. The kept table of a join that reads the key as a column is another entry, loaded row by row. `EXPLAIN ANALYZE` shows `rows classed` for the first statement and `cached tables` for the next (test `agg_lookup_dense_kept`, case `lookup_dense_kept`, the cache size `sift.agg_lookup_cache_kb`).
- Improved: pg_sift: the classes of a lookup table kept as classes of its keys are priced by the range and the NULLs of the leaves in the index, and the groups of a key computed from them are no more than the classes, where PostgreSQL's statistics of an expression that was never analyzed made the table millions of classes and the plan serial or the normal one: a count by the presence mask of a table not analyzed takes 0.32 s, as one that is.
- Improved: pg_sift: an expression of the columns of an inner-joined lookup table that a `sift_col` index of the table holds as a column (a presence mask of seven text columns) is one column of the lookup table, read from the leaf of the index, where it was a combination of the values of seven columns evaluated for each distinct tuple; with the classes of keys, a count grouped by the mask of 7.6 million parents over 20 million rows takes 0.25 s, where it took 8.8 s. Fixed: a table read as classes of keys gave the first key of a class for a read of the key as a column of the table; it is loaded row by row now.
- Improved: pg_sift: a `count(DISTINCT)` set that is a hash set stops at 262,144 pairs and goes to the sort of the pairs, where it grew to `hash_mem` (a table of hundreds of megabytes that missed the cache on every pair, and was rebuilt as it doubled): a distinct count of 20 million sparse values in 3,340 groups takes 0.66 s with `work_mem = '4GB'`, where it took 1.35 s (0.75 s with 4 MB, which is why a larger memory was slower). The array of pairs is stripped of repeats at a million pairs, so a set of many repeats does not grow to `hash_mem` before it loses any.
- Improved: pg_sift: a semi or anti join (`IN`, `EXISTS`, `NOT EXISTS`) by an `int8` key is answered from a byte table over the keys of the joined table, as one by an `int4`, `int2` or date key was, and keys that span more than `sift.agg_semi_lut_size` (4 MB, 4,194,304 values) are a hash set of them while that fits the same memory (262,144 keys), where both took the generic lookup join (a hash probe and a cache entry for every distinct value of the column). `NOT EXISTS` of a set of 1,000 members by a `bigint` key of 1,000,000 rows takes 99 M instructions and 15.8 ms where it took 592 M and 375 ms (PostgreSQL's plan 1,607 M, 84 ms), and of 10,000,000 rows, whose 5,000,000 keys span more than the byte table, 1,133 M and 56 ms where it took 9,789 M and 2,816 ms (16.5 times the instructions of the 1,000,000 rows for 10 times the rows, now 11.4; PostgreSQL's plan 16,016 M, 803 ms). Rows equal the normal plan's (tests `agg_semi_lut_int8`, `agg_semi_lut`, `agg_join_key_types`; case `semi_lut_bigint`; the matrix cell `set_anti_1000` of `bench/steps/join_matrix.py`). An `int8` leaf whose keys the scan's key filter has tested keeps needing no test of its own. Switched off with `sift.agg_semi_lut = off`.
- Improved: pg_sift: a lookup table of one `int4` key whose columns are integer, date, timestamp or boolean leaves of a `sift_col` index of it with few distinct tuples of values (a flag, a code, a mask of what is present) is kept as classes of its keys: the load classifies the key and the columns of each row and the scan makes an entry for each class, not for each distinct key. A `count(DISTINCT parent_id)` of 20,220,914 rows by the mask of the parent of 7,641,231 rows takes 0.85 s where it took 6.9 s with the table read from the leaves (20.2 s from the heap), and the planner prices the join, shares the table with workers and chooses the fact table as the scanned one by that load. Switched off with `sift.agg_lookup_columns = off`; `EXPLAIN ANALYZE` shows `rows classed` (test `agg_lookup_dense`, case `lookup_dense_classes`).
- Documented: pg_sift: a text column in the `WHERE` of a query that selects integer columns only must be in `return_columns` for the index to answer it as an `Index Only Scan` (PostgreSQL requires every column a query reads to be returnable), and the aggregate scan needs no such option.
- Documented: pg_sift: a `key_cut` build of a table with many keys ends in segments of at most twice the rows of an even split over its 128 (3,200,000 rows, 5,000 keys: 47,360 rows a segment, the arrays of its end under 2.4 MB), the slices of a parallel build are of one size, and a key with one value holding a large share of the table keeps that value's rows in one segment; the test `col_key_cut_segment_cap` holds the first two.
- Known limit: pg_sift: a ranked search of a `sift` index that has to hold every match (no `LIMIT` the search can take, or a filter after the search) fails with `54000` naming `work_mem` when its hits pass it, where earlier releases held them whatever their number; raise `work_mem` or add a `LIMIT`. Spilling the hits to a sort is planned
- Known limit: pg_sift: `sift.facets` of a keyword index with very many distinct values fails with `54000` naming `work_mem` when the values it holds pass it, where earlier releases held them whatever their number; raise `work_mem` or narrow the query. Reading the dictionaries as a stream is planned
- Known limit: pg_sift: the marks an ordered scan keeps for a run of equal keys, and the runs of an aggregate or a `count(DISTINCT)`, are temporary files that are read back by a seek, which the file layer of PostgreSQL does not make past 1 GB of one file: a statement that writes more fails with a read error of the temporary file (a run of tens of millions of equal keys). A seek by file and offset is planned
- Known limit: pg_sift: `parent LEFT JOIN child` with an aggregate over the child's columns, with a `sift_col` index on both tables, runs as the normal plan or as a scan of the parent that loads the child as rows; with the index on the child alone the scan reads the child and is several times faster (index the child).
- Tooling: pg_sift: `scripts/check-memory.sh` flags a plain `palloc` or `repalloc` whose size multiplies a count of rows, values, documents, units or blocks without a stated bound; `sift.test_limits = 'array_bytes=N'` lowers the limit of the helper those arrays go through, so a test fails a build that sizes an array by the table on a few rows.

## 4.2.0 - 2026-11-07

- Fixed: pg_sift: the background worker of a `sift_col` index ended with the merges undone when it found the maintenance lock with an inserter (which promotes and leaves the merges to the worker) at its first look, and a seal inside `sift.maintenance_naptime` of the request that started it asked for no worker of its own, so the segments stayed until the next write after the window or a `VACUUM` (a burst of inserts on a slow or busy server left tens of segments; present since 3.7.0, in every 4.0 and 4.1 release): the worker now stays until `sift.maintenance_naptime` after it started and looks again every so often, with a last look after the window, as the worker of a `sift` index already did; it still takes only the lock that is free, so none waits behind a long `VACUUM`.
- Fixed: pg_sift: a `DELETE` or `UPDATE` of a table beside `CREATE INDEX CONCURRENTLY` (or `REINDEX CONCURRENTLY`, or `pg_repack --only-indexes`) of a second `sift_col` index with `track_changes` failed with `sift index ... is corrupted` (block 0 with lower, upper and special 0) and so did the build: the trigger of the first index looked at the other indexes of the table and read the metapage of the one being built. The build writes its file past shared buffers, data pages first and the metapage last, so the read kept the hole of block 0 in shared buffers although the file ended complete. The triggers, and the planning of a window over the table (top rows per group), now look only at indexes that are ready for inserts, as the executor does. Test `col_build_unready`.
- Fixed: pg_sift: the directory entry of a write buffer carried four bytes of uninitialized stack (the alignment hole before its 8-byte fields) into the directory page, so two indexes built alike differed byte for byte (and leaked process memory into the file and its WAL); the hole is now a named member that every entry zeroes, and the layout and size are unchanged.
- Fixed: pg_sift: `sift.table_maintain()` asks for the lock of the cut of the free end once and never waits for a reader: it waited up to 5 s (or until `statement_timeout`) when a reader held the table, where its contract is to take only what is free now; `VACUUM` keeps its retry every 50 ms for up to 5 s (test `test/t/017_tbl_autodefrag.pl`)
- Fixed: pg_sift: a background worker that looked the table of an index up before the transaction that created the index committed locked the index without the table, the reverse of the order of `DROP TABLE` (table first); a server built with assertions stopped the worker, and it locks the table first now (test `merge_ranked` of the isolation suite on PostgreSQL 19)
- Fixed: pg_sift: `GROUP BY k % 3 HAVING k % 3 IN (SELECT ..)` keeps its HAVING on every PostgreSQL: from 18 the grouped expression looked like a column of the grouping entry and the condition moved to the rows, against the rule of the notes of 4.2 (an expression or a cast of the key keeps its HAVING); a `varchar` key cast to `text` and compared with a sub-select did the same on 17 and ran as a semi join on 18 where the scan refused it (test `agg_having_where`)
- Fixed: pg_sift: the first write into an empty `sift_table` (the metapage and the directory) reaches the file only after its WAL is durable: it went ahead of it, and a crash before the WAL flush left pages the recovery and a standby never had, so the standby stopped with "WAL contains references to invalid pages" and the table answered "is corrupted" or the primary failed its next recovery (test `tbl_first_writer_crash`)
- Fixed: pg_sift: on PostgreSQL 18, `WHERE id IN (SELECT id FROM t ..)` over the table itself by its unique key, where self-join elimination keeps the scan of the subquery and removes the outer one, said `SiftAgg not used: outer_join` (a label for an outer join the query does not have); it names the merge, as when the other scan is removed (slug `from_shape`: "PostgreSQL merged a joined table into another scan of the same table"). The query keeps the normal plan as before (test `agg_semi_same_table`); the cost of that normal plan and the way to get the scan are in `docs/AGG.md`.
- Docs: pg_sift: `docs/TUNING.md` says when to pick which of two pairs of options that look alike, with measured instructions, bytes and build cost: `point_tid_columns` (the default `auto`) is cheaper than `point_columns` in every shape measured, and `point_columns` is for an index a release before 3.7 must read; `order_array_columns` is the layer on top of the sorted runs of `order_columns` (it names columns of `order_columns`; `auto` builds both), 1.1 to 8 times cheaper again for 44% to 47% more index
- Docs: pg_sift: the description of `sift.col_format_minor` in `SHOW`, `pg_settings`, `docs/OPTIONS.md` and `docs/FORMATS.md` no longer calls 15 "the newest 3.3 writes": 15 is the default and the highest the setting takes, and an option with a newer layout (`key_cut`, `point_tid_columns`, ..) stamps its own higher minor
- Changed: pg_sift: `sift.workers` lists the `sift_table` tables the caller can read next to the indexes (`index` holds the table, `kind` is `sift_table`, with its directory entries, sealed write buffers and whether maintenance runs; the background of a table keeps no note of a failure, so `failures` is 0 and the columns that note one are NULL, as is `due`), and `sift.worker_state` takes a `sift_table`; the update replaces the view with the same columns, so a view of yours that reads it still works
- Changed: pg_sift: `sift.help(slug)` answers the slug of a refusal of `sift.explain_why` and of the notice `SiftAgg not used` as well (kind `refusal`; the reason as the message, what to change as the hint), where it knew only the ids of `docs/MESSAGES.md`; `sift.help()` without an argument lists the messages as before, and an unknown slug names the two places that list them
- Changed: pg_sift: `sift.merge`, `sift.promote` and `sift.verify` take a `sift_col` index too and run `sift.col_merge`, `sift.col_promote` and `sift.col_verify` on it (one name for the three kinds of maintenance; the `col_` names stay, with the same arguments and results); an index of another access method is refused as before
- Changed: pg_sift: `CREATE INDEX` and `ALTER INDEX .. SET` of a `sift` index with the reloption `mutable_limit` (the 2.0 spelling of `write_buffer_limit`, still accepted) say `NOTICE:  reloption "mutable_limit" is the older name of "write_buffer_limit"` with the hint to use the new name; nothing else changes, and the stored option and `ALTER INDEX .. RESET (mutable_limit)` work as before
- Fixed: pg_sift: the `sift_col` options `calendar_columns` and `rollup` take the words their docs promised: `calendar_columns = 'auto'` is the default (every date, timestamp and timestamptz column) like the empty value, and `rollup = 'auto'` or `'default'` is the byte-budget rule (`docs/OPTIONS.md` said `'default'` returned to it; the parser refused it as a leaf name); `docs/TUNING.md` lists which options take `auto`
- Changed: pg_sift: `sift.agg_lookup_max_rows` (deprecated since 3.2; `sift.agg_lookup_max_kb` replaces it) warns once per session, at the first aggregate plan that its limit decides, with `WARNING:  sift.agg_lookup_max_rows is deprecated and will be removed in 5.0`; it keeps limiting the rows as before, and `RESET sift.agg_lookup_max_rows` ends the warning
- Improved: pg_sift: a `VACUUM` of a `sift_col` index whose `AccessExclusiveLock` stays refused for the whole wait (readers that overlap all the time, a writer) gives the free blocks back online, as it does with a standby that streams and without one: it seals and promotes a write buffer above the end the file can have, moves the segments above that end down and cuts the free end under an `ExclusiveLock` (readers are never kept out, a writer waits for the cut only: longest statement 18 ms in a run of two overlapping readers and a writer); a file of 42.3 MB for 10.8 MB live was 10.9 MB after a few `VACUUM`s with them running, 28.2 MB with the code before. The step of the background worker that freed the blocks does the same, and the `VACUUM` says what it did once: `index "t_idx": a lock is held on it (waited 5000 ms): 2628 blocks moved, 0 given back online`.
- Fixed: pg_sift: a `VACUUM` of a `sift_col` index that is due for a rewrite no longer leaves the free blocks of the file to it when the automatic rewrite waits for its WAL budget (`sift.col_auto_merge_wal_pct`) or for the snapshots that hold the blocks of the last rewrite, or when only the write buffer is due: it moves the segments down and cuts the free end itself (a file of 27.0 MB for 11.4 MB live stayed there before and is 11.6 MB after one `VACUUM`; under two overlapping readers the file stayed at 2.0 times the live bytes for 56 s of writes and is 1.19 times on average after).
- Improved: pg_sift: the background worker of a `sift_col` index runs each step (a promotion, a merge, a pass of a rewrite) in a transaction of its own, where it kept one transaction and one snapshot for all of them, so the blocks a step freed were not reusable by the next (the xid that released them and the snapshot of the worker held the horizon back) and every step took new blocks at the end of the file: after 40 rounds of churn with the cut off the file is 11,015 blocks for 7,408 live where it was 14,608 for 7,822, and with the cut on the largest file of the run is 10,314 blocks where it was 16,242. Test: `test/t/col_worker_step_reuse.sh`.
- Improved: pg_sift: a `VACUUM` of a `sift_col` index or a `sift_table` that cannot cut the free end of the file says why, once, with the free blocks that stay in it: a lock another session holds on the table after the wait, a standby that streams, or a snapshot that may still read the blocks. `VACUUM (VERBOSE)` reports it at `INFO` (an index access method is told `DEBUG2` whatever the options, so the statement is read for `VERBOSE`) and an autovacuum that logs reports it at `LOG`, for example `index "u_idx": not truncating, a lock is held on it (waited 5000 ms): 1809 free blocks stay`. Test: `test/t/col_cut_skip_log.sh`.
- Tooling: pg_sift: `scripts/options-released.tsv`, the defaults of the last release that `docs/OPTIONS.md` compares with ("new in", "was .. in", the table "What a user gets after the update"), is the snapshot of 4.1 again; it still said 3.7, so the update table listed what a 3.7 user gets for options that 4.0 and 4.1 have shipped. The upgrade notes that were about 3.7 are gone from the cells of those options. `scripts/gen-options.sh --check` (the `options` lint) fails when the snapshot is not the newest release before the version of the tree, which is the step the version bump after a release does with `scripts/gen-options.sh --snapshot` in the tree of the tag; `--snapshot` itself failed (it read `SIFT_COL_FORMAT_MAX_READ` from a header it moved out of).
- Docs: pg_sift: `docs/REFERENCE.md` files the objects where a reader looks for them: `sift.index_advice`, `sift.index_choices`, `sift.key_cut_decision`, `sift.cluster_advice`, `sift.suggest_paths` and `sift.approx_count_distinct` are in "Column index" (they were in "Search and ranking"), `sift.help` is in "Diagnostics and upgrade", and `sift._table_option_names`, `sift._table_option_stored`, `sift.tblref_in`, `sift.tblref_out`, `sift.col_unit_agg` and `sift.col_unit_hist` (the support of other functions, the input and output of a type, two debug hooks) are listed under "Internal", where `scripts/gen-reference.sh` now puts every name that starts with `_`.
- Docs: pg_sift: `docs/QUICKSTART.md` section 5 listed `sift.enable_col_scan` as off (it is on since 3.3) and `sift.col_auto_merge` and `sift.table_bulk_groups` (both on) among the "opt-in switches": it is now two tables, what is off until you turn it on and what is on with its switch, and describes `sift.col_format_minor` as the setting that fixes the column format the segments are written in. `docs/COL-TAM.md` no longer says that the options of a `sift_table` are dumped as reloptions: they are the rows of `sift.table_option_values`.
- Docs: pg_sift: 24 settings that docs/OPTIONS.md filed as class `default` are class `diagnostic`, because their own advice is "leave it on, switch it off to compare or to find out which optimization gave a wrong result": `sift.agg_dense`, `agg_dist_stream`, `agg_float_step`, `agg_join_filter`, `agg_key_skew`, `agg_left_lateral`, `agg_lookup_columns`, `agg_lookup_pushdown`, `agg_preagg`, `agg_preagg_distinct`, `agg_probe_direct`, `agg_probe_rewrite`, `agg_rows`, `agg_semi_lut`, `agg_share_lookups`, `agg_super`, `agg_two_level`, `agg_fuse`, `agg_case_filter`, `topn_walk`, `topn_zone_keys`, `topn_runs`, `vis_page_cache` and `prefetch`. Nothing changes in the library: the class decides where the setting is listed (the 99 defaults are what a server runs on, the 38 diagnostic ones are in the generated table of `docs/TUNING.md`).
- Docs: pg_sift: `docs/TUNING.md` is the page for a user who has not read the source: the ten things worth touching (`sift.enable_agg = off`, `sift.explain_why`, `sift.index_choices`, `track_changes`, `key_order`, `paths` with `sift.suggest_paths`, `sift.workers`, `sift.upgrade_commands`, the three BM25 options that change what matches, the per-table options of a `sift_table`) with one statement each, the diagnostic options (a table generated from `docs/options.tsv`, `scripts/gen-options.sh`), the three places where one name (`write_buffer_limit`, `dead_rewrite_pct`, `dict_lookup`) has three defaults, the limits of the background work of the three kinds side by side, the words that mean several things (`key`, `rollup`, `field`), the words each list option accepts, and which function to call for the space, the segments and the maintenance state of each kind. README, USAGE and QUICKSTART link to it.
- Changed: pg_sift: the developer method `unit_rewrite` of `sift.eval_methods` is removed with its writer (one segment generation per unit at build and promotion, and the rewrite of a unit by VACUUM; `sift.test_col_unit_gate`, a test setting, goes with it): it cost more than it saved in every measured case and was never on by default. `SET sift.eval_methods = 'unit_rewrite'` (in a session, `ALTER DATABASE .. SET`, `ALTER SYSTEM` or `postgresql.conf`) now fails with `ERROR:  invalid value for parameter "sift.eval_methods": "unit_rewrite"` and `DETAIL:  Unknown evaluation method: unit_rewrite`; remove the name before updating. Segments written with the method stay readable (the flag is still honoured): VACUUM treats them like the others, the automatic merge and rewrite leave them alone, and a forced `sift.col_merge` folds them. Nothing else is written differently.
- Changed: pg_sift: the name `latest_hint` of `sift.eval_methods` is gone with the code behind it, a per-backend array of the latest version of a key that the supersede lookup of `track_changes` could try first: it won nothing in any measured case and was never on by default. `SET sift.eval_methods = 'latest_hint'` (in a session, `ALTER DATABASE .. SET`, `ALTER SYSTEM` or `postgresql.conf`) now fails with `ERROR:  invalid value for parameter "sift.eval_methods": "latest_hint"` and `DETAIL:  Unknown evaluation method: latest_hint`; remove the name before updating. Nothing is written differently.
- Changed: pg_sift: the name `snapshot_masks` of `sift.eval_methods` is gone: it was accepted and had no code behind it. `SET sift.eval_methods = 'snapshot_masks'` (in a session, `ALTER DATABASE .. SET`, `ALTER SYSTEM` or `postgresql.conf`) now fails with `ERROR:  invalid value for parameter "sift.eval_methods": "snapshot_masks"` and `DETAIL:  Unknown evaluation method: snapshot_masks`; remove the name before updating. Nothing is written differently.
- Changed: pg_sift: the developer method `skip_trigger_lookup` of `sift.eval_methods` is gone because it is always on: the statement trigger of `track_changes` leaves out the look at an UPDATE that kept the key, as it did by default, and the sign of `-name` (which turned a default method off) is no longer accepted. `SET sift.eval_methods = 'skip_trigger_lookup'` and `'-skip_trigger_lookup'` now fail with `ERROR:  invalid value for parameter "sift.eval_methods": "skip_trigger_lookup"` and `DETAIL:  Unknown evaluation method: skip_trigger_lookup` (`-skip_trigger_lookup` for the second); remove the name from `ALTER DATABASE .. SET`, `ALTER SYSTEM` and `postgresql.conf` before updating, or the setting is refused where it is read. Nothing is written differently.
- Fixed: pg_sift: `VACUUM` of a `sift_col` index or a `sift_table` gives the free end of the file back when a reader or a writer holds the table for a moment: the cut asked for its `AccessExclusiveLock` once and gave up, so on a table that is read all the time the free blocks stayed in the file (a merge or a rewrite leaves it up to twice the live bytes). It now asks again every 50 ms for up to 5 seconds without queueing, like the truncation of a heap; a shorter `lock_timeout` ends the wait, and the background worker still asks once. Test: `test/t/col_cut_lock_wait.sh`.
- Improved: pg_sift: a correlated `min()` or `max()` sub-select of a table is the first row of its entity in the order of the column, where it was a pass over the rows of the entity for each row of the query around it, or the table grouped and joined: `(SELECT max(ts) FROM events WHERE entity = e.id)` in a select list or a condition is planned a second time as `ORDER BY ts DESC LIMIT 1` over the non-NULL values (PostgreSQL does that itself for a b-tree, but only for the query at the top), and the planner keeps it where it runs in the scan and costs less. The scan reads one entry of the order array of the entity (`order_array_group`) or one unit; a loose condition of the sub-select that passes few rows of the entity (`AND status = 7`) refuses the walk at planning. The rows are those of the normal plan, NULLs, empty entities and NULL keys included (`test/sql/col_topn_minmax_sublink.sql`).
- Changed: pg_sift: 4.2 changes ten things worth reading before the update (`docs/UPGRADING.md`, "What 4.2 changes"; only (7), (8) and (9) ask for a step, to remove a setting before the update; the new library reads the indexes of 4.1 and older as they are): (1) the index keeps a presence bit for the `jsonb` columns with paths that may be NULL: `CREATE INDEX` and `REINDEX` of a `sift_col` index write a presence leaf for each (no option, no NOTICE; no column format minor changes); an index built before it has none, and 3.2, 4.0 and 4.1 refuse an index that has it (`sift_col index "x" does not match its column layout`, hint `REINDEX the index.`): to go back, `REINDEX` the index with the older release; there is no switch. (2) The write buffer of a BM25 `sift` index is format 6; the first insert after the update raises the index to minor 1.6, which 4.1 and older refuse (`has on-disk format 1.6` where that build reads up to 1.5): to go back, `REINDEX` the index with the older release; there is no switch. (3) The aggregate scan is chosen more often for a grouped aggregate of one table with no lookup join, search or top-N whose normal plan is a sequential scan and a hash. (4) A `HAVING column IN (SELECT ..)` or `EXISTS (SELECT ..)` of a plain `GROUP BY` is one rule, a filter of the rows, and a sub-select of the groups is evaluated by the scan. Both are changes of plans, not of files, and `sift.enable_agg = off` is their only switch. (5) The log of a `track_changes` index whose triggers do not vouch for it (after `pg_upgrade`, a restore, `DISABLE TRIGGER ALL` then `ENABLE TRIGGER ALL`, a concurrent build) is repaired by itself: no format change, nothing to run. (6) what to change for each refusal slug is a new function, `sift.refusal_actions()`; `sift.refusal_slugs()` keeps its three columns, and the update changes no function that a view of yours may read. (7) `sift.eval_methods` no longer takes `latest_hint`, `snapshot_masks`, `unit_rewrite` and `skip_trigger_lookup` (nor the `-name` form): a `SET`, `ALTER ROLE`, `ALTER DATABASE` or `ALTER SYSTEM` that names one fails with `invalid value for parameter "sift.eval_methods"` and `Unknown evaluation method`, and a value already stored is ignored with a warning in every session; remove them from `postgresql.conf` and from the settings of roles and databases before updating (`docs/UPGRADING.md` has the queries; `skip_trigger_lookup` is now always on). (8) the setting `sift.enable_col` is removed (it never changed anything; `sift.enable_col_scan` and `sift.enable_agg` are the switches): with the library loaded a `SET`, `ALTER ROLE`, `ALTER DATABASE` or `ALTER SYSTEM` that names it fails with `invalid configuration parameter name "sift.enable_col"` (`"sift" is a reserved prefix.`), and a value already stored is dropped with a warning in every session (`invalid configuration parameter name "sift.enable_col", removing it`); remove it from `postgresql.conf` and from the settings of roles and databases before updating (`docs/UPGRADING.md` has the queries). (9) the setting `sift.table_auto_defrag_dead_pct` (no effect since 4.0) is removed and refused the same way, with the same advice. (10) the `sift_col` reloptions `rollup` and `rollup_dims` are `rollup_leaves` and `rollup_columns`: the old names stay accepted through 4.x with a NOTICE that names the new one, the new name wins when both are given, an index keeps the name it carries (`pg_dump` and `pg_upgrade` carry it over), and the messages, `sift.index_choices` and the errors say the new spelling. `scripts/upgrade-check.sh --to 4.1` lists (1) and (2) as blockers, and (7), (8) and (9) as blockers whatever the target
- Fixed: pg_sift: `scripts/upgrade-check.sh --to 4.1` (or any release before 4.2) listed a BM25 index whose write buffer is format 6 (index format 1.6) and a `sift_col` index with the presence leaf of a `jsonb` document as ok, though 4.1 and older refuse both (the leaf checked with 3.2.3, 4.0.5 and 4.1.4, the buffer with 4.1.4 and 4.0.5); they are blockers now, with `REINDEX INDEX x` with the older release as the advice for the leaf
- Improved: pg_sift: `NOT (doc @> '{"b": "x"}')`, `(doc @> ..) IS FALSE`, `NOT (doc ? 'k')`, `(a AND b) IS FALSE` and `FILTER (WHERE NOT (a OR b))` of containments, the same in a `CASE`, and `doc IS [NOT] NULL`, `(doc IS NULL) IS FALSE` of a `jsonb` column that may be NULL are answered by the index, where they ran on the normal plan (the index could not tell a NULL document from one without the declared paths). The index keeps a boolean column per `jsonb` column with paths that may be NULL (written by `CREATE INDEX` and `REINDEX` for the columns without a `NOT NULL` constraint, which need none; an index built before it has none), NULL where the document is SQL NULL. 150,000 documents, 1% to 50% of them without a declared member: `NOT` of a containment 85 M to 121 M cycles down to 7 M to 10 M, the count of `doc IS NULL` 35 M to 1.9 M. It costs 2.0% of the index of 1,000,000 rows after a merge and 3.2% of its build (insert +0.4%, promotion +4.6%, merge +5.0%); the column is no dimension of the pre-aggregates. An index built before keeps the normal plan for those forms until `REINDEX`; no column format minor changes, and a release before 4.2 reads an index with the column as of another layout and says `REINDEX`.
- Improved: pg_sift: a containment of a `jsonb` column whose index declares the paths (`doc @> '{"b": "x"}'`, `'{..}' <@ doc`, an array element) is answered by the index as `IS NOT TRUE`, in a `CASE` (a pivot of sums, whose `ELSE 0` adds `IS NOT TRUE`), in the `FILTER` of an aggregate and over an `AND` / `OR` with other conditions, where only a conjunct, an `OR` branch or a `FILTER` was: the rows the test does not keep (a NULL document among them) are the complement of the rows it keeps, a new node of the predicate that prunes no unit. `NOT` and `IS FALSE` of the test are served for a column with a `NOT NULL` constraint; for a column that may be NULL they keep the normal plan, because the index cannot tell a NULL document from a document without the declared paths. Measured on 150,000 documents, M cycles of one backend (normal plan, scan): `WHERE (doc @> ..) IS NOT TRUE` 69.9, 3.5; a `CASE` pivot `sum(CASE WHEN doc @> .. THEN 1 ELSE 0 END)` by group 138.3, 8.9; `FILTER (WHERE (doc @> ..) IS NOT TRUE)` 133.1, 8.0; `NOT (doc @> ..)` on a `NOT NULL` column 71.3, 3.5 (tests `agg_jsonb_wide_forms`, `agg_jsonb_not_exist`).
- Fixed: pg_sift: on PostgreSQL 19, `corr`, `covar_pop`, `covar_samp` and the `regr_*` aggregates of float8 columns failed in the aggregate scan with `float8_regr_accum: expected 8-element float8 array`: the state of the transition function has 8 elements from 19 (it keeps whether the inputs were constant), and the scan checked its own 6-element steps against the function on a 6-element state. The scan runs PostgreSQL's function for them on 19 (17 and 18 keep the kernel); `var_*` and `stddev_*` are unchanged.
- Improved: pg_sift: `count(e)` and `count(e.*)` of the whole row of the scanned table (a `HAVING count(e) > 100`, `count(e) FILTER (WHERE ..)`) are answered by the scan as `count(*)`, where they were refused with the slug `aggregate` and ran on the normal plan: the row datum of a tuple is never NULL, even when every field is. A table that an outer join nulls (the nullable side) or whose join preserves rows of another table keeps the normal plan, as do `count(DISTINCT e)`, the whole row of a joined table and `max(ctid)`.
- Improved: pg_sift: the value of a key read beside its test, `sum((doc ->> 'k')::float8) FILTER (WHERE doc ? 'k')`, `CASE WHEN doc ? 'k' THEN (doc ->> 'k')::float8 ELSE 0 END` or `WHERE doc ? 'k'` with the key as an aggregate argument, a group key or a comparison, is answered by the index of a `jsonb` column whose paths declare the key, where the test kept the normal plan when the key was read as well: a document that is no object reads NULL, and the documents that hold a value the path cannot store (a text that is no number, a container) are read again from the heap, raising the error of PostgreSQL where a row reaches the expression. A comparison of the read in an `AND` of the same `FILTER` (`FILTER (WHERE doc ? 'k' AND (doc ->> 'k')::float8 > 3)`) is answered too. The document itself, a path below the key and a `jsonb_typeof` guard around the read keep the normal plan.
- Improved: pg_sift: an average, a minimum or a count of a CASE with an ELSE that is a value (`avg(CASE WHEN status = 'paid' THEN qty ELSE 0 END)`, `min`, `max`, `count(CASE WHEN p THEN x ELSE 1 END)`) is an aggregate of the value with a filter on the condition, as the sum with ELSE 0 and the aggregates with ELSE NULL already were, so the scan computes it on the kernels of the value and counts the rows where the condition does not hold, where the CASE was evaluated for every row on the generic path and the scan was seldom worth it: a group where the condition never holds is 0, not NULL, and the zeros count in an average, as in the CASE. `min` and `max` for integers and floats, `avg` for integers and numerics (the same division of the same sum and count as the final function of `avg`) and for `float4` and `float8`, the ELSE a constant of the type (zero for the average of a float, whose sum of the constant would round differently from the CASE); the minimum and maximum of a numeric (equal values of a different scale tie in the order of the rows) stay as written. Tests `agg_case_filter_else`, `agg_case_filter`.
- Fixed: pg_sift: `sift.index_advice`, `sift.index_choices`, `sift.key_cut_decision`, `sift.col_verify`, `sift.col_clustering`, `sift.col_tracking`, `sift.col_stamps`, `sift.col_rollup_refresh`, `sift.col_refresh`, `sift.col_unit_agg` and `sift.col_unit_hist` lock the table before the index, as `DROP TABLE` and `ALTER TABLE` do: a function that held the index and then asked for the table waited for a `DROP TABLE` that waited for the index, and PostgreSQL aborted one of them with `deadlock detected` (the `DROP TABLE` when its `deadlock_timeout` ran out first); no result was wrong (`col_verify`, `col_rollup_refresh` and `col_unit_agg`/`col_unit_hist` from 3.2.0, `index_advice`, `col_clustering` and `col_refresh` from 3.3.0, `col_tracking`, `col_stamps` and `key_cut_decision` from 4.0.0, `index_choices` from 4.0.3; tested by `test/t/col_lock_order.sh`)
- Improved: pg_sift: `doc ? 'k'`, `doc ?| array[..]` and `doc ?& array[..]` of a `jsonb` column whose index declares the paths are answered by the index in a FILTER, an OR or a CASE (`sum(CASE WHEN doc ? 'k' THEN x ELSE 0 END)`) and `IS NOT TRUE`, and `NOT (doc ? 'k')` when the column is NOT NULL, where only a condition of the WHERE was: a count of the documents with a key beside a count of the rows reads one chunk of the leaf and no document, where the normal plan decodes every document. The answer is `?`'s for a document of every JSON type at the root, a JSON null and a NULL document (a query that also reads the key or the document, `NOT` of a column that may be NULL, and `avg`, `min`, `max` of `CASE .. ELSE 0` keep the normal plan). Tests `agg_jsonb_key_forms`, `agg_root_key_exists`.
- Changed: pg_sift: the planner chooses the aggregate scan more often for a grouped aggregate of one table (no lookup join, no search, no top-N) whose normal plan is a sequential scan and a hash of the groups, when the scan's own work per row is not small: an aggregate of an expression (`max(ends - starts)`, `sum(m01 * m02)`, `max((ends - starts) * 2 + 1)`), a `COALESCE`, a test of a range (`NOT lower_inc(r)`). The scan's cost counts its operators at 3 ns, 833 cost units a millisecond, and PostgreSQL's estimate of that normal plan runs at about 210 units a millisecond (measured on grouped aggregates over 150,000 rows: 3,500 to 4,630 units for 17 to 21 ms, whatever the shape, because its executor takes about 4 times what the plan prices), so a scan that was up to 4 times faster was refused as `cost` when its own rows were not few. The rows of such a scan are now counted in the units of the plan they are compared with; the start of the scan, the workers' start (scaled with it, so the choice of workers stays), the lookup joins, the reads of the lookup tables, the rows read again from the heap, an ungrouped aggregate, a top-N, a search, and a normal plan that reads an index keep their prices. Measured, normal plan then scan (milliseconds of the statement, minimum of five runs): `max(ends - starts)` by tenant over 150,000 rows 18.6 to 7.1 and `max((ends - starts) * 2 + 1)` 20.4 to 7.3 (13.3 to 7.0 and 13.0 to 7.8 with PostgreSQL's default parallel workers), over 18,750 rows 2.7 to 1.4 and 3.0 to 1.7, `COALESCE` measures over 20,000 events 3.4 to 2.1, sums of products 2.5 to 1.8, `NOT lower_inc(r)` 0.25 to 0.04, a text key of nearly as many values as rows under a filter (`agg_key_entries_cost`) 30 to 18. A statement the scan does not win (`avg(ends - starts), max(ends - starts)` over 18,750 rows: 9.0 and 9.2 M cycles) stays on the normal plan. The corpus pins the length statements (`c3_span_length_*`) and `test/plan/bands.tsv` the cost ratios of the plangate shapes.
- Changed: pg_sift: the planner chooses the aggregate scan more often for a grouped aggregate with lookup joins (`facts LEFT JOIN links ON links.parent_id = facts.dim_id GROUP BY ..`) when the normal plan reads more than 2,000 cost units from the scanned table, an index plan of the table included. The estimate of the scan, the rows of the lookup tables it loads included, is now counted in the units of the plan it is compared with (about 210 a millisecond, 833 for the scan's own operators), as the scan of one table already was; a normal plan that reads a few rows of an index (about 500 units, 0.4 ms) keeps the old price, and the other terms are unchanged. Measured on 250,000 rows (`sift_table`, a tuned profile: `seq_page_cost` 100, 8 workers, 4 GB `work_mem`), normal plan then scan: `linkleft_all_parents` 2,351.6 to 1,643.2 M instructions, 868.8 to 480.1 M cycles, 193.6 to 107.9 ms, 361,019 to 851 buffers; no statement of the 132 plan shapes in 8 settings gets slower, and `test/plan/bands.tsv` holds the cost ratios.
- Changed: pg_sift: the aggregate scan prices an expression that PostgreSQL evaluates (`max(ends - starts)`, `sum((a - b) * 2 + 1)`) at what it takes: 21 ns for each tuple read and 69 ns for each distinct combination of its columns (it counted 36 ns an entry and nothing for the tuple), so the estimate of such a scan is about 3 times higher on 150,000 rows of 10 distinct pairs (291 to 907 units for 3.1 ms) and 2.2 times on 150,000 distinct pairs (1,345 to 2,927 units for 13.5 ms); `max((ends - starts) * 2 + 1)` over 20,000 rows, a tie in time (0.59 ms normal, 0.65 ms scan), stays on the normal plan, and over 75,000 rows it still takes the scan (1.4 ms against 2.3 ms).
- Improved: pg_sift: `pg_squeeze` and a killed `pg_repack` are tested (`test/t/col_pg_squeeze.sh`, `test/t/col_pg_repack.sh`) and documented as they behave: the log of a `sift_col` index with `track_changes` is not complete after a squeeze, the reads check the heap and answer as it does, and the next `VACUUM` re-baselines it; what a killed `pg_repack` leaves does not touch the index of the table, and `DROP EXTENSION pg_repack CASCADE` removes it.
- Fixed: pg_sift: `pg_repack --only-indexes` of a `sift_col` index with `track_changes` failed with `DROP INDEX CONCURRENTLY must be first action in transaction` and left its `index_<oid>` copy behind (4.0.0 to 4.1.3): the copy got a pair of triggers of its own, which the drop of the copy took with it. The copy of a concurrent swap, `REINDEX CONCURRENTLY` (4.1.3) and now `pg_repack` (named `index_<oid>` in the schema of the table), creates none and the index that keeps its oid keeps its own. The log is not complete after the swap, and the next `VACUUM` re-baselines it.
- Changed: pg_sift: `DROP INDEX CONCURRENTLY` of a `sift_col` index that tracks its changes is refused before it changes anything, with `ALTER INDEX i SET (track_changes = 'off')` first, or `DROP INDEX` without `CONCURRENTLY`, in the hint, where PostgreSQL's `must be first action in transaction` named neither cause nor way out (the triggers of the option go with the index and need a transaction id that the command does not allow).
- Fixed: pg_sift: a `sift_col` index with `track_changes` whose log is not complete only because of what happened to its triggers or its build is made complete again by itself, where the reads checked the heap until the user ran `REINDEX` or `ALTER INDEX`: after `pg_upgrade` (every tracked index of the cluster, damaged or not: the triggers are made again by the restore and the log names the identities of the old ones), a restore, `ALTER TABLE .. DISABLE TRIGGER ALL` followed by `ENABLE TRIGGER ALL` (`pg_restore --disable-triggers`), `CREATE INDEX CONCURRENTLY`, `REINDEX CONCURRENTLY` and `DETACH PARTITION`. The re-baseline of 4.1.2 (the next `VACUUM`, the first read that checks the heap, the worker a write asks for; `ShareUpdateExclusiveLock` only) now runs for any log that the triggers, being the option's, do not vouch for. Triggers enabled for the origin only (what `ENABLE TRIGGER ALL` leaves) are first enabled for all sessions by the worker, which takes the lock of `ENABLE TRIGGER` with a conditional request (it never waits) for the time of a catalog change; triggers that the user disabled stay disabled and the reads check the heap. Real `pg_upgrade` of a cluster written by v4.1.0 (PostgreSQL 17 to 18): all four logs flags 1 and `triggers` false after the upgrade, heap-free again after one read with the worker on; the heap-free read, the checked read and the heap give the same answer at every step. The NOTICE of a concurrent build now says that the next `VACUUM` completes the log. Held by `col_track_revouch` (TAP) and `col_track_rebaseline`.
- Improved: pg_sift: `lower(r)` and `upper(r)` of a range column (`int4range`, `int8range`, `daterange`, `tsrange`, `tstzrange`) as an aggregate argument, a group key or a condition (`min(lower(r))`, `date_trunc('month', lower(r))`, `avg(upper(r) - lower(r))`, `WHERE lower(r) >= ..`) are read from the two lanes the index keeps for the containment operators, as the subtype's value (NULL for a NULL or empty range and for an infinite bound), where the image of each distinct range was decoded: on 75,000 spans `min(lower(r)), max(upper(r))` runs in 3.7 M cycles (was 31.3), a month of `lower(r)` 5.0 (was 97.6), `min(lower(r))` of a timestamp range 5.6 (was 33.8), a `WHERE` on `lower(r)` 1.2 (was 26.6), `avg(upper(r) - lower(r))` by tenant 11.8 (was 45.5). A column that holds a bound too large for a lane (an `int8` near its limits, a timestamp `infinity`) keeps the normal plan, which reads the image. No format change (test `agg_range_values`).
- Improved: pg_sift: the write buffer of a BM25 `sift` index is opened in buffer format 6, whose full pages have a bloom of term hashes of 8,192 bits (2 probes, 4% false positives at 900 terms) where format 0 has 256 bits for the page, which a page of 900 terms fills, so that every search read all the pages of the buffer for a term that none of its documents has: the bloom is filled when the page is full (the append of a document changes the head of the page, and the log record of the head grows with the bytes that change), a page the bloom rules out is not even copied, and the pages being appended to have the small bloom as before. 220,000 documents and a buffer of 2,000: a term in no document 1.96 to 1.23 M cycles and 2.17 to 0.95 M instructions (the index without a buffer 0.73 M and 0.56 M), a buffer of 5,000 2.60 to 1.24 M cycles and 2.03 to 1.01 M instructions; a term of 5% of the documents is read as before. Upgrading takes no step: the buffer open at the upgrade keeps format 0 and is read as it is, and the next one is format 6. **A BM25 index written by 4.2 (after its first insert) is not readable by 4.1 or older, which refuse it with a REINDEX hint: to go back to an older release, `REINDEX` the index with the older release** (the first buffer of format 6 raises the metapage of the index to 1.6 and nothing lowers it; `docs/FORMATS.md`, `docs/UPGRADING.md`)
- Improved: pg_sift: the documents that searches walk in the open write buffer of a BM25 `sift` index are added up for the buffer (shared memory, by generation, like the rows a `sift_col` write buffer decodes), and once they cost half of its promotion the worker seals and promotes the buffer: every search walks the buffer's pages, because the 256-bit bloom of a page is full at 900 terms, at about 250 cycles a document against 80 for a document in a segment, and the promotion costs 55,000 cycles a document once (295 M cycles and 95 ms for 5,000). 220,000 documents and a 5,000-document buffer: a count of a term of 5% of the documents 3.3 M cycles and 5.6 M instructions with the buffer, 1.2 M cycles and 1.2 M instructions after, a term in no document 1.6 M against 0.8 M; the buffer is promoted after 110 searches that walk all of it. A buffer of fewer than 2,000 documents, and a buffer that is only written, are left for `write_buffer_limit`, VACUUM and the seal as before
- Improved: pg_sift: the first statement of a backend that has not loaded the library (no `shared_preload_libraries`) is planned without the planner hook, and now gets the one step of the finished plan that needs none: the match of a BM25 index with a stemmer moves from the filter of a bitmap heap scan to its recheck condition (the executor start does it on the plan it was given), so `body @@> 'w' AND tenant_id = 7` no longer analyzes the document of each fetched row again in that statement
- Improved: pg_sift: `lower_inc(r)`, `upper_inc(r)` and their `NOT` of a `tsrange` or `tstzrange` column are exact on the lanes, where PostgreSQL's function decided every finite candidate: the lane of the bound already holds the form of the bound in the parity of its position (inclusive lower `2v`, exclusive `2v + 1`; inclusive upper `2v`, exclusive `2v - 1`), so the rows that pass the lanes are kept by that parity, with no change of the format (a bound too large for a lane is still PostgreSQL's). Measured on 18,750 spans, M cycles of one backend (scan off, PostgreSQL decides the candidates, parity): `upper_inc(seen)` 4.5, 2.1, 0.8; `NOT upper_inc(seen)` 9.8, 2.4, 1.4; `lower_inc(seen)` grouped 10.5, 2.0, 1.5; `NOT lower_inc(seen)` grouped 7.2, 2.4, 1.0 (test `agg_range_unary`, with ranges before 1970, at the ends of the lanes and infinite in the four forms of bounds).
- Improved: pg_sift: a test of a range, a network or a wide value (`numeric` of more than 18 digits, `money`, `interval`) that the source runs on the lanes of the index is run there as a branch of an `OR`, in the `FILTER` of an aggregate, in a `CASE` (a pivot of sums, whose `ELSE 0` adds `IS NOT TRUE`) and as `IS NOT TRUE` (and over an `AND` / `OR` of such tests), where only a conjunct of the `WHERE` was: the source gives the rows that pass as a bitmap of the unit, no value is read and no unit is skipped on the test of a branch (the rows of the other branches may be in it). `NOT lower_inc(r)` and `NOT upper_inc(r)` of a range column are tests of the lanes too: an empty range and an infinite bound by the lanes, the finite ranges of `tsrange` and `tstzrange` by the parity of the bound's position. A test with a NULL operand (a parameter of a generic plan) keeps no row, where the source took the operand as a value. Measured on 150,000 rows, M cycles of one backend (scan off, before, after): `count(*) FILTER (WHERE isempty(r))` 62.3, 66.4, 2.8; its `CASE` 63.6, 21.8, 5.3; its `OR` 53.5, 58.5, 12.8; `FILTER (WHERE ip << net)` 60.8, 26.3, 5.3; `FILTER (WHERE n > ..)` of a `numeric(30,4)` 72.5, 97.6, 3.8; its `CASE` 71.9, 125.6, 6.7; `FILTER (WHERE iv > interval '10 days')` 62.9, 10.7, 3.0 (tests `agg_wide_forms`, `agg_range_unary`, `agg_image_eq`).
- Improved: pg_sift: `isempty(r)`, `lower_inf(r)`, `upper_inf(r)`, `lower_inc(r)` and `upper_inc(r)` of a range column in a `WHERE` conjunct (and `NOT` of the first three, `IS TRUE`, `IS FALSE`) are tests on the two lanes the index keeps for the containment operators, where every row's image was decoded: an empty range, an infinite bound, the canonical form of `int4range`, `int8range` and `daterange` (and the infinite dates, which stay as written) are lane tests, the inclusive and exclusive bounds of `tsrange` and `tstzrange` are the parity of the position of the bound, and a row whose bound does not fit a lane is decided by PostgreSQL's function too (test `agg_range_unary`).
- Improved: pg_sift: the table of a lookup join that has its own bounds (`dim_id IN (SELECT id FROM dims WHERE id <= 2000)`, `day >= .. AND day < ..`, a bound that follows the equal columns of a btree such as `country = 3 AND tier < 2`) is read by an index scan with the bounds as its keys, where only `=` and `IN (list)` were: a lookup table of 200,000 rows read for `id <= 2000` loads in 0.24 ms for 2,000 rows read (was 10 ms for the whole table), and a grouped statement over 300,000 rows that keeps those ids runs in 16.8 M cycles (was 39.1 M; 50,000 groups 37.6 to 15.6 M). The two bounds of a range are judged together by the rows the statistics expect them to keep (a lower bound that keeps 99% of the table does not make an index scan of a range that keeps 1%); a bound that keeps a large share, `<>`, a bound of an expression and a collation other than the index's read the table by the heap scan as before (test `agg_lookup_range_bounds`).
- Improved: pg_sift: a bitmap heap scan over a `sift` (BM25) index with a stemmer (or any analyzer other than the default: a match that is the call `sift.match_idx(key, query, index)` in the plan) and another index (`body @@> 'qaaaf' AND category_id = 3` with a btree on `category_id`) checks the match again only on the rows of a lossy page: the index gives the rows of the bitmap exactly, but PostgreSQL does not see the call as the index condition it derived (an operator) and kept it as a filter of every fetched row, which tokenized and stemmed the document of each row again. 200,000 documents, the count of one word in one of 50 tenants: 28.1 to 6.8 M cycles and 60.7 to 11.8 M instructions (bench case `search_bitmap_recheck`: 11.8 M instructions and 1.3 ms, GIN on the `tsvector` 27.6 M and 2.2 ms), and the correlated sub-select that counts it for each of 50 tenants 123 to 69 M cycles. The rows and the scores are the same; a match in an OR of the bitmap (`BitmapOr`) keeps its filter
- Added: pg_sift: `sift.refusal_actions()` returns what to change for each slug of `sift.refusal_slugs()` (the statement, the index or a setting: the HINT of the notice of `sift.explain_refusals`), so that `sift.explain_why(q) w JOIN sift.refusal_actions() a USING (slug)` lists it. It is a function of its own and `sift.refusal_slugs()` keeps the three columns of 4.1: the result of a function cannot gain a column without dropping it, and `ALTER EXTENSION pg_sift UPDATE` would then fail for a database that has a view over it (`cannot drop function refusal_slugs() because other objects depend on it`); the update only adds a function and no view, default or function of yours is touched
- Improved: pg_sift: `doc ?& array['a', 'b']` of a `jsonb` column whose index declares every key as a path is answered by the index like `doc ? 'a' AND doc ? 'b'` (the documents that are an array or a string are read again from the heap). `?|` and `doc->'a' ? 'k'` keep the normal plan: an OR of the type tests of two leaves, and a string or an array at a nested path, which the entry flag `SIFT_SEG_ROOTS` does not say, are new mechanisms. Test `agg_root_key_exists`.
- Improved: pg_sift: `sift.explain_why` and the notice of `sift.explain_refusals` name the child of a `join_child` refusal that gave no detail (a join's probe form, a lateral function or sub-select the scan cannot run per input) and, for a child whose rows do not fit `sift.agg_lookup_max_kb`, the rows it is estimated at for each of how many distinct inputs and its width (test `agg_probe_rewrite`).
- Improved: pg_sift: `array_agg(DISTINCT x ORDER BY x)` and `string_agg(DISTINCT x, ',' ORDER BY x)` with the `ORDER BY` the `DISTINCT` argument in its own ascending order (NULLs last): the distinct values of a group ordered by the argument itself are a set that is sorted once, where every row of the group went into a tuplesort and the duplicates were dropped as they came out. `DESC`, `NULLS FIRST`, a `FILTER` and another order keep the sort. Test `agg_distinct_ordered`.
- Improved: pg_sift: the rows of a top-N per group (`(array_agg(id ORDER BY a DESC, id))[1:5]`, `row_number()`, `rank()` and `dense_rank() OVER (PARTITION BY g ORDER BY ..) <= N`) whose first sort key is a plain column: a tuple of a topn that comes after the last kept one of its group is dropped on the first sort key alone, read from the chunk with no allocation, where every tuple went through the generic row path. Tuples that tie with the last kept one on the first key, and NULL or NaN keys, go through that path as before. Test `agg_topn_chunk_edge`.
- Improved: pg_sift: `percentile_cont(array[0.25, 0.5, 0.75]) WITHIN GROUP (ORDER BY x)` and `percentile_disc(array[..])` of a constant array of fractions are one aggregate per fraction on the same values, so the percentiles of an array of fractions are picked from the one set of values the scan keeps, where PostgreSQL's transition function sorts every value of every group in a tuplesort. A NULL fraction is a NULL element, a group with no value a NULL, and the result has the fractions' dimension and lower bound; fractions outside [0, 1] and an array that does not start at 1 stay in the normal plan, which raises its errors. The planner plans both ways and keeps the cheaper. Test `agg_pct_array`.
- Improved: pg_sift: `doc ? 'k'` of a `jsonb` column whose index declares the path `k` (`paths = 'doc.k float8'`, any leaf type) is answered by the index: the rows whose `k` holds a value of any JSON type are the rows with the key, and the documents that are no object (an array holding the string, a string equal to it) are read again from the heap and tested as `?` tests them, so the answer is `?`'s for a document of every JSON type at the root. `SELECT count(*) FROM ke_fact WHERE attrs ? 'a' AND dim_a_id < 500` over 300,000 documents of 400 bytes takes 15.4 M instructions and 1.1 ms (788 blocks), where the normal plan takes 250 M instructions and 14,252 blocks (`bench/cases/jsonb_key_exists.sql`). A segment written before 4.2 does not say whether a document that is a string was a missing key (a new entry flag, `SIFT_SEG_ROOTS`, no format minor, `docs/FORMATS.md`): the plan reads all its NULL slots from the heap and is priced with them, so a table of old segments stays on the normal plan, and VACUUM flags the old segments in one pass over the heap (about 140 ms for 600,000 documents of 400 bytes) unless the table holds a document that is a string. A document that is a JSON string is now stored as the overflow of the columns below it, as an array always was, so a recheck-capable read of such a table reads those rows again (the rows are the same). `?|` and `?&` are not answered by the index. Test `agg_root_key_exists`.
- Improved: pg_sift: a sub-select in the select list or the HAVING of a grouped query that reads the group keys or the aggregates (`SELECT kind_id, count(*), (SELECT name FROM kinds k WHERE k.id = events.kind_id) FROM events GROUP BY kind_id`, `HAVING count(*) > (SELECT count(*) FROM members m WHERE m.kind_id = events.kind_id)`, `EXISTS (..)`, `count(*) NOT IN (SELECT ..)`, a ROLLUP's output) is evaluated by the scan node once for each group, as it is above PostgreSQL's aggregate: the query had stayed out of the scan for it (`output_expr`) and the grouping read every row. 300,000 rows and 1,000 groups: 36.8 to 1.85 ms and 350 to 23.4 M instructions; a sub-select that costs more than the grouping (a sequential scan of its table for each group) costs what it did. The statement plans without workers, as before
- Improved: pg_sift: a `HAVING` condition `column IN (SELECT ..)` or `EXISTS (SELECT ..)` of a plain `GROUP BY` is always answered as a filter of the rows, as a semi join the scan runs, where PostgreSQL keeps it above the aggregate (it moves a HAVING condition to WHERE only without a sub-select), so every row was grouped and the sub-select ran for each group, outside the scan (`GROUP BY kind_id HAVING kind_id IN (SELECT id FROM kinds WHERE enabled)`: 300,000 rows, 1,000 keys, 100 of them in the list, 40.1 to 0.58 ms and 342 to 5.7 M instructions). The planner no longer sets this plan against the one that tests the groups: the rows form was the cheaper in all 40 cells measured (300,000 rows, 100 to 200,000 groups, a list that keeps 1% or 50% of the keys, one table with the group columns in the index and one with four, with and without workers), at 0.09 to 0.78 of the cycles: 8.2 to 1.0 ms at 100 groups and 1% of the keys, 33.0 to 5.3 ms at 200,000 groups and 1%, 78 to 45 ms at 200,000 groups and 50%, 12.3 to 9.6 ms for two group columns with the list on one, 24 to 17 ms under a `WHERE`, and 20.8 to 13.2 ms for the top 10 groups. The condition is moved to `WHERE` after the other rewrites of the statement, so the sub-select is not turned into a probe for each group first. Only the shapes where the rows form wins move: an `IN` on the column as it is (or a relabel of it, `varchar` to `text`) whose sub-select does not read the query, whatever it holds (an aggregate, a `GROUP BY`, a `LIMIT`, a `UNION`: the planner makes a semi join of every uncorrelated `IN` in `WHERE`, where in `HAVING` it is a sub-plan of the groups that, when it does not fit `work_mem` to be hashed, runs for each group: 161 s against 28 ms for 20,000 keys of an aggregate at 64kB, and 0.1 to 0.9 of the cycles with a `LIMIT` or an aggregate in the sub-select at work_mem 256MB, in none of 40 cells slower: 33.3 to 4.8 ms for a `LIMIT` over 200,000 groups, 30.7 to 5.5 ms for an aggregate), and `EXISTS` of a sub-select without aggregate, grouping, limit, set operation or CTE; the condition reads no aggregate of the query and nothing volatile. `NOT IN` and `NOT EXISTS` (20% slower in the rows form in some cells, 2 to 3 times faster in others), an expression or a cast of the key (`amount::int IN (..)`, `ts::date IN (..)`: 49% slower in half the cells, and a cast that cuts the value is a function of the key that the groups may not agree on), a correlated `IN`, grouping sets and a `HAVING` without `GROUP BY` keep their `HAVING` and give the rows they gave (test `agg_having_where`, 46 statements, with workers and from prepared statements, rows equal to sift off).
- Improved: pg_sift: a table left-joined to a join of two small tables (`FROM kinds k CROSS JOIN regions r LEFT JOIN events e ON e.kind_id = k.id AND e.region = r.region GROUP BY k.name, r.region`: every kind and region with its count, zero for the pairs of no event) is not one preserved lookup, so the scan stayed out of it and the join read every event: the groups of the table by the keys of the join are made in the scan and joined to the pairs, a count is the group's (0 for a pair of none, and `count(*)` counts the row of a pair of none once), a sum, min or max the group's (NULL for none). `count`, `sum` of integers and numerics, `min` and `max` only: a float sum, `avg`, `count(DISTINCT)`, a column of the table outside the aggregates and a `WHERE` on it keep the join. The planner plans it both ways and keeps the cheaper. Test `agg_left_partial`.
- Improved: pg_sift: `FETCH FIRST n ROWS WITH TIES` by integer, date or timestamp columns of the index is a top-N of the scan, which gives the `n` best rows and every row that ties with the `n`-th (the heap has room for the rows the statistics say tie and starts again 16 times larger when its last row still ties) and the `Limit` ends the ties, where the statement was always sorted by the normal plan: `ORDER BY dim_a_id DESC FETCH FIRST 10 ROWS WITH TIES` over 300,000 rows takes 10.3 M instructions and 0.94 ms where the normal plan takes 387 M and 44.2 ms (case `topn_with_ties`, 4.3 M cycles where it takes 133 M); the rows are the normal plan's.
- Improved: pg_sift: `NOT EXISTS` (or `EXISTS`, or a scalar sub-select) over a table with a condition that its index does not take, `WHERE NOT EXISTS (SELECT 1 FROM order_items i WHERE i.order_id = o.id AND i.price > 150)`, is planned by what the child costs for every row the scan reads, not for the rows that are left after the anti join: the child's rows are loaded once into a lookup table, where a probe of the child's table for each row of the outer table (priced at a hundredth of what it takes) was chosen and ran several times longer than the normal plan. Anti and semi joins whose probes were already the cheaper plan keep them.
- Improved: pg_sift: an `IN` list tested against the key summaries of the units (the list, the runs, the bitmap of a `key_columns` leaf) is cut to the keys of each unit's range by a bisection of the list, and only the members within it are looked up in the unit, where each of the members was looked up in each unit: `SELECT kind_id, count(*), sum(amount) FROM t WHERE dim_a_id IN (75 keys) GROUP BY kind_id` over 300,000 rows laid out in 30 batches of a `key_cut` index takes 3.4 M instructions and 0.33 ms where it took 5.2 M and 0.50 ms (case `set_list_unit_span`, 1.9 M cycles where it took 2.7 M); the units read are the same.
- Improved: pg_sift: the values of an ordered set (`percentile_cont`, `percentile_disc`, `mode()`) that outgrow `work_mem` go to sorted runs of (group, value, count) pairs in a temporary file, merged when the scan has ended (at most 16 at a time), where every value went through a tuplesort at about 3,000 instructions: the median and the 95th percentile of 300,000 `amount::float8` in 40 groups take 188 M instructions and 32 ms where they took 1,205 M and 86 ms (case `percentile_select`, 125 M cycles where it took 384 M), `percentile_disc` of a decimal and a timestamp 348 M instructions and 56 ms where it took 2,303 M and 195 ms, slower than the normal plan's 172 ms (case `percentile_disc_int`), `mode()` 229 M instructions and 33 ms where it took 1,231 M and 106 ms (case `mode_histogram`).
- Improved: pg_sift: the values of a `percentile_cont` or `percentile_disc` that are nearly all distinct (four in five of the first 8,192 pairs are new) are appended to an array of pairs, where each value was looked up in a hash that held a quarter as many pairs of `work_mem` and moved to a sort past that: `SELECT grp, percentile_disc(0.5) WITHIN GROUP (ORDER BY amount), percentile_disc(0.95) WITHIN GROUP (ORDER BY amount) FROM t GROUP BY grp` over 200,000 distinct float8 values takes 49 M instructions and 5.1 ms where it took 805 M and 76.5 ms (case `percentile_distinct_values`, 25 M cycles where it took 306 M), and the array holds 262,144 pairs at the default `work_mem` before the sort takes over; the histogram of a `mode()` keeps the hash, and `EXPLAIN ANALYZE` shows `ordered sets of distinct values` for the sets that were arrays.
- Improved: pg_sift: `rank() OVER (PARTITION BY k ORDER BY o) <= N` and `dense_rank() OVER (PARTITION BY k ORDER BY o) <= N` (N up to 1,000, N = 1 included, the condition `rn = 1`, `rn <= N` or `rn < N + 1` over a sub-select or a CTE read once) keeps in the scan the rows of rank <= N of each group (the N best and every row that ties with the N-th) or of dense_rank <= N (the rows of the N best values of the sort keys), where the plan sorted every row. The window function and its condition stay in the query over the rows that are left, whose ranks are their own, so the rows are exactly the window's whatever the ties, the NULLs and the direction of the sort; a partition of several columns and a nullable partition column are the same form. Rows that tie with the N-th are all kept, as in the window: a group whose rows all tie keeps all of them, in memory, as `array_agg` does. `percent_rank`, `ntile`, another window function in the sub-select and N > 1,000 keep the window.
- Improved: pg_sift: `mode() WITHIN GROUP (ORDER BY x)` of a column of a type with a hash operator counts the values of each group in a hash, where every value went through two sorts (one for all the groups, one for the group) and `mode_final`: `SELECT grp, mode() WITHIN GROUP (ORDER BY status) FROM t GROUP BY grp` over 300,000 rows takes 163 M instructions and 13.6 ms where it took 903 M and 79.6 ms (case `agg_mode_text_groups`, 62 M cycles where it took 312 M); each group takes the value with the highest count, the first in the order of the call among equal counts, which is the value `mode_final` returns, and counts that outgrow `work_mem` go to the sort, each value put as many times as it was counted.
- Fixed: pg_sift: the table that counts the rows backends decoded from a write buffer (`col/wbuf/reads.c`, the promotion of the buffer once reads have paid for it) was attached in the first statement that decoded the buffer, which can be the supersede of an `UPDATE`; the registry attaches in a subtransaction, whose commit advances the command id, so a row updated by two statements of one transaction kept its middle version in the index and the heap-free read counted it (a `sift_col` index with `track_changes`: 20,003 rows where the heap has 20,000 after updates by a fresh backend). The registry attaches no table under a statement that writes now (`sift_dsm_attach` goes without it and tries again in the next statement; `sift._test_dsm_deferred` lists what a backend went without); the new tests `col_wbuf_reads_write_path` and `col_dsm_write_caches` compares the heap-free read, the checked read and the heap across such a transaction, a promotion between statements and workers.
- Improved: pg_sift: `NOT (tags && ARRAY[..])`, `NOT (tags @> ARRAY[..])` and `NOT (tags <@ ARRAY[..])` over an array column the index holds run on the index (they kept the normal plan with the slug `where`, because the test is NULL for a NULL array and NOT of NULL keeps no row): the rows are those whose array is not NULL and fails the element test, with the same rows as the normal plan in the six states of an index (built, write buffer, promoted, updated, merged, reindexed), in both layouts of an array column (lengths and element lists; image and element lists), over NULL and empty arrays, NULL elements, duplicates, multidimensional arrays and a lower bound other than 1. 21 of 30 statements of the matrix of negations ran on the index before, all 30 now. Held by `agg_array_not`.
- Improved: pg_sift: a test of the elements of an array column (`@>`, `&&`, `<@`, `= ANY`) that the index holds as whole arrays only (`paths = 'auto:'`, or an index whose element lists the build did not choose) is priced for each row it reads: the scan looks the image of every row up in its batch and resolves its entry, 170 instructions a row whatever the number of distinct arrays, which the model counted at nothing. A containment of one element over 300,000 rows of repeating pairs (441 distinct) took 24.2 M cycles on the scan against 13.6 M on the GIN bitmap of the normal plan and now keeps the normal plan (`sift.explain_why`: `cost`); an overlap, 25.5 M against 27.0 M, is a tie either way; `= ANY` with a date range (24.1 M against 70.4 M) and `unnest` (13.0 M against 126.1 M) stay on the index. With the element lists beside the arrays (`ALTER INDEX i SET (paths = '+tags[] int4'); REINDEX INDEX i`, 0.31 s for 300,000 rows) all four read the lists, 3.7 to 5.4 M cycles. Held by `agg_array_image_native`.
- Improved: pg_sift: a lookup of one row by its primary key, and every statement with no aggregate, group, window, `DISTINCT` or top-N shape, is planned as PostgreSQL plans it with less work of the planner hooks: the check of the column indexes of its tables reads a cache instead of opening the relation and each of its indexes, and the walk for a `CASE` an aggregate could take as a `FILTER` is made only for a statement that can hold an aggregate (the plan of such a statement is the same with `sift.enable_agg` on and off).
- Improved: pg_sift: a scan keeps what the tombstone log says of each heap block it asks about twice: the units of a key-cut segment list the same pages again and again, and every ask bisected the log four times and walked the entries of the page for their verdicts (about 2,600 instructions for each logged row of the statement over the whole table). Over a churned 600,000-row index with 59,768 entries in the log and no VACUUM since, the grouped sum of the table takes 142.2 M instructions on a backend's second statement where it took 207.7 M (-32 %, cycles 89 M to 58 M; the first statement of a new backend 220.7 M to 155.2 M), a set of 75 entities 21.0 M to 19.4 M, one entity 3.8 M to 3.9 M (a block asked about once is not kept: one bit). The grouped sum over 200,000 rows with 5 % updated and 5 % deleted and no VACUUM since takes 49.0 million instructions and 4.5 ms where it took 65.1 million instructions and 6.2 ms (case `col_log_block_cache`). A VACUUM that applies the log makes it 37.8 M (the same statement, 1.36 G instructions and 5.2 MB of WAL once).
- Improved: pg_sift: `CREATE INDEX` on a table with an array column (`text`, `varchar`, `int2`, `int4` or `int8`) whose arrays repeat keeps the elements beside the whole arrays: the build estimated the index growth at 49% where it is 10% of the leaf bytes (5% of the file) and the cost of an element test over the arrays at a price per different array, and so kept the arrays alone, where `@>`, `&&`, `= ANY` and `unnest` cost 150 to 250 instructions a row and `@>` and `&&` ran slower than the normal plan. 300,000 rows of two-element integer arrays (441 different, 60% of the rows with an array), one backend, warm, the minimum of 8 interleaved rounds, cycles (milliseconds, blocks) of the statement with the arrays alone, with the elements, and with the normal plan: `tags && ARRAY[3, 7]` 24.1 M (6.8 ms, 98) to 5.3 M (2.5 ms, 71), normal plan 19.4 M (9.8 ms, 2,897); `tags @> ARRAY[5]` 23.3 M (7.4 ms, 98) to 3.7 M (2.0 ms, 76), normal plan 11.6 M (6.0 ms, 2,886); `7 = ANY (tags)` 23.8 M (6.7 ms, 153) to 4.0 M (2.2 ms, 136), normal plan 67.3 M (18.6 ms, 2,944); `unnest(tags)` grouped 12.6 M (3.7 ms, 55) to 1.5 M (0.8 ms, 38), normal plan 128.5 M (49.9 ms, 2,944); cold, `@>` reads 49 blocks against 77 and 2,885. The index grows by 5.0% (5.57 to 5.85 MB, the build 15% longer). Arrays that are nearly all different keep their elements in place of the arrays, as before. An index built earlier keeps its layout: `ALTER INDEX i SET (paths = '+tags[] int4')` and `REINDEX INDEX i` add the elements (0.31 s for 300,000 rows). The hint of the `NOTICE` that says how to leave the elements out names `paths = 'auto:'` (an empty `paths` is auto, which chooses them again).
- Improved: pg_sift: the write buffer of a `sift_col` index is promoted once the backends that read it have paid for its promotion. A backend decodes the rows of the buffer in the first statement that reads it (5,900 instructions a row, kept for its next statements), and the server adds up the rows decoded for each buffer; when they cost half of what the promotion costs (18,600 instructions and 36 bytes of WAL a row, once), which two backends reading the whole buffer do, the worker seals and promotes it (`sift.col_auto_merge` and `sift.async_maintenance` on; a buffer of a quarter of a unit of rows at least, so that a small buffer does not become a segment that every later read opens). A buffer that is only written is not promoted before `write_buffer_limit`, and the write path has no new code. The first statement of a new backend over 200,000 rows with 25,000 of them in the buffer: one customer 96.4 million instructions and 10.3 ms before, 3.79 million instructions and 2.3 ms after; 75 customers 98.2 million instructions before, 4.47 million instructions after; the grouped sum of the table 120.3 million instructions before, 19.2 million instructions after (cases `col_wbuf_read_priced`, `col_wbuf_read_priced_set`, `col_wbuf_read_priced_all`). On a churned 600,000-row index (19,619 rows in the buffer, 59,768 entries in the log of deleted rows) the first statement over one entity went from 131.9 M instructions (51.3 M cycles, 14.8 ms) to 17.3 M (15.0 M cycles, 3.9 ms), a set of 75 entities from 149.8 M to 34.4 M, the grouped sum of the table from 335.9 M to 220.7 M, for 715 KB of WAL once; a second statement of the backend is the same (3.7 M and 3.8 M for one entity: one more segment to list). The 208 M the grouped sum still costs after the promotion is the log of deleted rows, which a VACUUM applies, not the buffer.
- Improved: pg_sift: `variance`, `var_pop`, `var_samp`, `stddev`, `stddev_pop` and `stddev_samp` of a float8, int4 or int2 column and `corr`, `covar_pop`, `covar_samp` and the `regr_*` functions of two such columns are added up for a chunk of rows at a time, in the state of PostgreSQL's own transition function (the same partial states, final functions and spill), where the transition function was called for each row: `SELECT grp, stddev(amount), variance(score) FROM t GROUP BY grp` over 300,000 rows takes 60 M instructions and 3.8 ms where it took 187 M and 11.2 ms (case `agg_stat_variance_groups`, 17 M cycles where it took 53 M), and `corr(amount, qty), regr_slope(amount, score)` 137 M instructions and 11.1 ms where it took 347 M and 20.4 ms (case `agg_stat_corr_groups`, 45 M cycles where it took 92 M); the steps are the ones the server's own function does (fused or not, found by running it once on a probe) so the rows are its own, and a row with a NaN, an infinity or a sum that overflows goes through the function. The scan is chosen by the cost model of the plan as before: these cases force it (`sift.agg_force`).
- Added: pg_sift: options of one `sift_table`, as functions, because a table access method has no reloptions before PostgreSQL 20: `sift.table_set_option(table, name, value)` sets the `sift.table_*` setting of that name for the table (the owner of the table; the value is checked as `SET` checks it), `sift.table_get_option` returns the value in force, `sift.table_reset_option` takes one option or all off the table, and `sift.table_options` lists every option of the tables the caller may see with its value, its source (`table`, `setting`, `default`) and its default. The option of a table beats the setting whatever its source. The rows live in `sift.table_option_values`, which `pg_dump` writes with the database: they follow the table through a rename, `SET SCHEMA`, `TRUNCATE`, `VACUUM FULL`, `CLUSTER`, a restore (also of a dump that leaves the table out, and `pg_restore -j`), `pg_upgrade`, a crash and a standby, and `DROP TABLE` deletes them. Settings read in a statement cost what they cost before (insert of one row 64,300 instructions, a grouped read of 20,000 rows 299,000, with and without options, with 1 and with 1,000 tables with options); a backend pays 328,000 instructions once, when it first opens the table of options, and 4,000 to 8,000 for each table whose entry it builds after that. This is the first SQL of 4.2: `ALTER EXTENSION pg_sift UPDATE` runs `pg_sift--4.1--4.2.sql` (a table of 4.1 reads as before and takes options). Held by `tbl_options`, `extension_lifecycle` and `tbl_options_cluster`. In [COL-TAM 8.1](docs/COL-TAM.md#81-options-of-one-table).
- Improved: pg_sift: `sift.explain_why` accepts the `EXECUTE` of a prepared `SELECT` and plans it for the values it is given (it refused it with "takes one SELECT statement"), and the notice of `sift.explain_refusals` carries a `HINT` for each of the 38 refusal slugs, one line on what to change in the statement, the index or a setting (or that the normal plan is the answer); the table of slugs in `docs/USAGE.md` and `docs/COL-PLAN.md` has the same line as a column
- Improved: pg_sift: the heap fetch of a top-N is priced at a page of the b-tree plan with a LIMIT (it was half a page, and with sorted runs or an order array a cpu price of a fifth of that), and, with sorted runs or an order array, no less than the pages the b-tree plan reads where a b-tree gives the first key of the order (the scan has no such floor against a sequential scan and a sort, which pays no random page), so a list of rows of one entity (`ORDER BY .. LIMIT n` over `WHERE entity = k`) keeps the b-tree plan on (entity, key) where that plan is faster. The top-N scan took such a list in 8 of 18 measured on 600,000 rows and in 11 of 12 on 200,000 (entities of 163 to 19,998 rows, 200 and 1,000 rows), at 1.7 to 2.4 times the instructions and 1.5 to 2.5 times the cycles of the b-tree plan; all 30 are the b-tree's now (the case `entity_list_btree_plan`: 1.09 million instructions and 0.09 ms against 1.88 million and 0.16 to 0.25 ms of the scan, 17 heap blocks hit and 185 read beside 218 hit and 184 read), and `pgbench -c 4` over a zipfian entity went from 6,800 to 7,300 statements a second to 9,300 to 11,600, where the b-tree plan makes 10,100 to 11,100. The lists the b-tree cannot give in order (another key, a filter on a column it does not hold, a page of the whole table) keep the scan, which takes 4 to 80 times fewer instructions; what is left of the difference in a statement that stays with the b-tree plan is the planning of the extension (about 130,000 instructions). Test `agg_cost_losers`.
- Improved: pg_sift: the merge of the small overlay segments of a `key_cut` index counts the rows of the units a request scans: the overlays of a promotion put their rows in units as wide as the index's, so a unit the merge spares a request is worth its rows (20 instructions each) and not the 46,000 instructions of a narrow one; the merge of two overlays of 30,000 rows is due after about 400 requests, not 4,400, and a churned 600,000-row index with 59,000 rows in two overlays merges after 1,300 requests instead of never (the statement over one entity: 1.30 -> 1.15 M instructions, a set of 75 entities 3.84 -> 3.22, WAL 2.0 MB for the merge; one customer over two overlays of 4,000 rows each: 0.63 million instructions and 65 buffers become 0.48 million instructions and 29 buffers)
- Improved: pg_sift: the merge of the small overlay segments also serves a `key_cut` index that is not cut into partitions (an index under the rewrite limit): the overlays are the segments under half the largest, and a step takes the smallest that come to one and a half times the largest of them, so a row is rewritten once for each growth by half and not at every promotion. One customer of 200,000 rows after 16 promotions over the first 200 customers: 0.93 million instructions and 110 buffers with 17 segments, 0.48 million instructions and 31 buffers after the merge of the overlays (case `col_units_per_key`). The automatic rewrite (VACUUM, the worker) takes one step of overlays at a time, an explicit `sift.col_maintain` goes on until they are one, and a segment that holds dead rows is no longer rewritten with them. A table that churns 15% of its rows and is vacuumed, read by 6,000 requests, reads 2 units for one customer where it read 3, and the maintenance says `overlay units` where it said nothing.
- Improved: pg_sift: `SELECT .. FROM (SELECT .., row_number() OVER (PARTITION BY k ORDER BY o) rn FROM t) s WHERE rn <= N` (N > 1, and up to 1,000 rows of each group in the scan) keeps only the first N rows of each group in the scan, as the rows pass, where the plan sorted every row (or held every row of a group in an array). The groups' arrays are cut inside the group (`(array_agg(c ORDER BY o))[1:N]`) and the scan, which reads an `array_agg(.. ORDER BY ..)` only through a slice `a[1:n]` or an element `a[n]` of a constant n (in the select list, the sort and the HAVING; read whole anywhere, it keeps every row), keeps the n best rows of each group in memory and gives them to the transition function in order when the scan has ended: a row that comes after the n-th is dropped after the compare of its sort keys. It is the form for any number of keys, a nullable partition column and a partition of several columns, and it is refused for groups whose first rows would not fit the memory of a hash aggregate (`first_memory`), which the window's sort spills. The rows are those of the window's first N, the rows of a tie in `o` in no order as in the window. `rank()` keeps the window.
- Improved: pg_sift: a containment `doc @> '{...}'` of an object document with a boolean, a JSON null or a string that reads as a boolean or a number as a member (`{"flag": true}`, `{"k": null}`, `{"flag": "true"}`) is answered by the leaves of an `astext` path: the text of the member and the JSON type the chunks keep (`jsonb_typeof`), which a string `"true"` does not pass. It is one of the top-level conditions of the WHERE, with the other members of the pattern as before. A containment, `?`, `?|` or `?&` that names a path the index does not declare in `paths` keeps the normal plan under the new reason `jsonb_path_not_declared`, which names the path.
- Improved: pg_sift: a condition that wraps a column in `coalesce(col, c)`, `nullif(col, c)`, `col IS [NOT] DISTINCT FROM k` or an integer widening cast (`int4_col::bigint = k`) is run as tests of the column by the comparison kernels, where it was an expression evaluated for every row. The rows are those of the expression for every value of the column, a NULL column and a NULL constant included, and a `NOT` around the condition keeps the expression as written. A path of a document and a cast that rounds (`float8::numeric`, `int4::float8`) keep the expression.
- Improved: pg_sift: `sum`, `avg`, `min`, `max` and `count` over `CASE WHEN p THEN x` (ELSE NULL or none), and `sum` over `CASE WHEN p THEN x ELSE 0 END`, run as the aggregate of x with a filter on p (`sift.agg_case_filter`, on): the scan computes x on its kernels under a test of p, not the CASE for every row, and the sum with ELSE 0 keeps what the CASE gave: 0 for a group where p never holds, NULL for an empty input and for values that are all NULL. A numeric zero with a scale, `avg`, `min` and `max` with ELSE 0, DISTINCT, more than one WHEN and a sub-select in the condition or the value stay as written, and a statement the scan does not run keeps the plan of the CASE as written.
- Improved: pg_sift: the direct probe of a correlated child compares an index tuple that has a NULL in an `INCLUDE` column or in a key column after the compared ones as fast as one that has none: the NULL sent the compare of every such tuple through `index_getattr`, one column at a time. Over a covering index whose included column is mostly NULL (the lookup child of an item rollup with an expression key, `INCLUDE (scope, ..)`) a list of 200 entities with 11,388 probes took 215.6 M instructions and takes 191.3 M (-11 %; cycles, the minimum of 8 interleaved rounds, 86.9 M to 80.5 M). The same statement over the covering index of a vacuumed lookup table is 10 % fewer cycles and 47 % fewer blocks hit than over the plain index (the recipe is in `docs/USAGE.md`: `VACUUM`, and `INCLUDE` every column the child reads, the column inside an indexed expression too).
- Improved: pg_sift: the price of the normal plan that probes the keys of a lookup PostgreSQL counts from statistics (a membership table joined to a fact table: `WHERE entity IN (SELECT ..)`, a dimension) is multiplied by the whole skew of the key column, where it was multiplied by its square root: the keys a join passes are the ones with rows, the hot ones of a skewed column, and PostgreSQL prices a probe at the mean's rows. A statement of 12 hot keys of a column of skew 7 was refused with `cost` (scan 2,240, normal plan 1,833, the dimension's statistics saying a table of 40 times its size) and took 282 million cycles, 42 ms, with the normal plan against 8.2 million, 3.4 ms, with the scan; a set of cold keys takes 2.5 times fewer cycles with the scan as well. The case `set_key_skew_dimension` multiplied by the skew of the column: 3.0 ms and 45 million instructions against 19.6 ms and 99 million of the normal plan; the statement `in_values_1k` of the application corpus (a VALUES list of 16 keys, 14.7 against 2.5 million cycles) takes the scan too. Test `agg_set_key_skew`; the plan gate and the corpus change no plan.
- Improved: pg_sift: a grouped statement numbers the groups of an integer key (an `int`, `bigint`, `date`, `timestamp` or `timestamptz` column or an expression of one, `GROUP BY id / 2`) from the value by pages of ids while the values lie in a dense range (a range of up to a million values, or of at most four times the groups), and by one hash table of the value and its id otherwise, where it used a hash table of the output value and another of the input value that each cost a call of the type's hash and equality: `GROUP BY id / 2` over 300,000 rows in no order of id takes 433 M instructions and 37 ms where it took 475 M instructions and 53 ms (case `group_int_expr_halves`, 173 M cycles where it took 231 M), and `GROUP BY k` of a bigint column whose values lie in a dense range 237 M instructions and 17 ms where it took 242 M and 21 ms (case `group_bigint_key_dense`, 87 M cycles where it took 110 M); the blocks read are the same. A grouped statement whose keys are sparse takes the hash table, which holds 16 bytes for each key and so a little more memory than the two tables did: a scan with a small `work_mem` may spill a stage earlier.
- Changed: bench: `bench/steps/fresh_backend_statements.sh` no longer counts the exit of the backend in the instructions and cycles of a statement run in a new backend (psql ended the session right after it, so the teardown of the process was in the difference between a new backend and a warm one): the backend now waits until perf has stopped. What `shared_preload_libraries` saves a new backend is in `docs/USAGE.md`.
- Improved: pg_sift: the `Lookup:` line of `EXPLAIN ANALYZE` says what the direct probes of a correlated child did and how well the lookup tables are vacuumed: `40000 child direct probes (11123 leaves, 70 upper pages pinned, 40000 heap fetches)` (`Child Probe Leaves`, `Child Probe Upper Pages Pinned`, `Child Probe Heap Fetches`, and for an index-only child `Child Probe Index Only Entries`, the entries the visibility map answered), and `99 % of the lookup tables' pages all-visible` (`Lookup Pages All Visible`); where a lookup table of 64 pages or more has under half of its pages all-visible the plan adds `Lookup Hint: VACUUM the lookup tables ..`. The tables a run loads whole are counted too.
- Improved: pg_sift: a backend that plans a statement over a `sift_col` index after a segment was added to it (a promotion of the write buffer, a table that takes inserts all day) no longer reads the statistics of every segment again: the record the first backend keeps in shared memory stands for the first segments of the directory, and only the segments it does not cover are opened. The statistics are the same bytes either way (test `col_leaf_fold` folds both ways and compares), and they no longer move when rows only enter the write buffer. The same record serves the planner when the sift planner features are off, which still costs the index. The pre-aggregate census counts up to 16 units of each segment (a share of 256 units for the whole index before), so that a segment's count does not depend on the others (`pre_census_units`); counts of blocks, not time (bench step `first_statement_planning`).
- Improved: pg_sift: the direct probe of a correlated child (`LEFT JOIN LATERAL (.. LIMIT 1)`) over a uuid, a date, a timestamp or a timestamptz key compares it in the probe itself, as it did a smallint, an integer or a bigint (the 16 bytes of a uuid, the integers of the others), and not through the type's order function: 40,000 probes into a lookup table of 300,000 rows take 585 M instructions for a uuid key where they took 756 M, 247 M for a date key where they took 347 M and 261 M for a timestamptz key where they took 360 M, and the best of the interleaved runs 338 M, 152 M and 235 M cycles where it was 412 M, 224 M and 289 M; the rows are the same (ascending, descending and NULLS FIRST indexes, negative values, the infinities of a date and a timestamp, NULL keys and entries, keys of another integer type). A column of another type keeps the order function, a key of one text column under the C collation or another and a key of several columns of arrays and texts are unchanged.
- Improved: pg_sift: `sift.json_build_object` writes the objects of a statement that builds one for every output row (the groups of a grouped statement: an object of totals, an object of changes, the list of the objects of a group) with less work for each: the keys that are constants or parameters of the query are written from the bytes the first call made of them (the string is no longer escaped again for every row), the text is built in a buffer of the call site and copied out once (it is no longer allocated and freed for every call), integers and float8 values are written without the output function's string and the text value `datum_to_json` makes of it, and a json value is copied as it is. The text is the core's, byte for byte (the test compares both for every type, at every `extra_float_digits`, and for keys that are parameters).
- Changed: pg_sift: a correlated lookup (a `LEFT JOIN LATERAL .. LIMIT 1` into a large mapping table) no longer asks the OS for the whole table and its index once its probes have read a sixteenth of it, which for a table larger than shared buffers asked for hundreds of megabytes the statement did not need and, for a statement with a few hundred keys, took longer than the probes. The direct probe asks for the heap blocks of the entries it has found, together, before it fetches them, and only while the statement is reading heap pages from the OS: at most one block for each entry of a key probed, none for a table in memory. The blocks the probes read are unchanged and `EXPLAIN ANALYZE` shows the blocks asked on the `Lookup:` line (test `agg_child_warm`, counts of blocks, not time).
- Improved: pg_sift: the direct probe of a correlated child (`LEFT JOIN LATERAL (.. LIMIT 1)`, the plans of `sift.agg_probe_rewrite`) over a key of several columns costs fewer instructions: the case `lateral_probe_direct` at 50,000 rows (44,000 keys of a constant array and three texts, probed in a lookup table of 300,000 rows) takes 452 M instructions where it took 549 M, and 671 M where it took 1,440 M in a database under an ICU collation; the rows are the same. The descent compares a key with an index tuple from the first column on which the bounds of the page's range do not both agree with it (a column that the separators on both sides of a page equal is the same in every tuple between them: the constant array and the leading texts are compared once, not at every step), keeps the pages above the leaves pinned for the batch (a pin of such a page blocks no VACUUM; the leaves are released at once as before), takes the entry the search ended at as the first compare of the walk, and sorts the keys of a batch with a comparison that the compiler inlines. Text under a collation that is not the C collation (`en_US`, ICU) is taken as equal where its bytes are equal and goes through the collation's order function only where they differ, and its bytes order the keys of the batch. No setting, no REINDEX.
- Improved: pg_sift: planning prices the list of keys of a statement once for all the plans it makes of it (the CTE shared and inlined, and each arm that reads the list): the selectivity of the whole list, 0.7 million instructions for a few hundred keys, was computed again for every copy of the list, so a statement of two arms over a list of keys takes 48.5 million instructions to plan where it took 50.8 (test `agg_key_list_once`).
- Changed: pg_sift: a run of a correlated `LIMIT 1` child that the scan probes in its btree directly is priced by what it takes, 55 operators (`AGG_PROBE_RUN`; it was 800), 1.9 times that for a key column of a collation other than C (`AGG_PROBE_COLL`), an entry of an Index Only Scan 3 and of an Index Scan 7 (`AGG_PROBE_SCAN`, `AGG_PROBE_FETCH`; 42 and 110), and the table of the first-row lookup read whole at the price of any other table read whole (`AGG_LOAD_ROW`; `AGG_FIRST_ROW` was added in cost units to a cost in operators): the probe costs 5,700 instructions for an integer or a date key, 6,900 for a text under the C collation, 7,700 for a uuid and 12,000 for an ICU text (100,000 rows, 1,250 to 20,000 keys), the table 820 to 980 instructions a row, so the probes win up to a fifth of the rows of the table as keys (a twelfth for ICU text) and were priced to win up to a twentieth: of 60 choices between them 21 were wrong by instructions (1,470 million lost) and none are now; the cost ratios of the link shapes of the plan gate fall (the band of `linkanti_2000_parents` 0.16 to 0.62 became 0.046 to 0.18), no plan of it changes (test `agg_probe_run_crossover`).

## 4.1.4 - 2026-11-06

- Fixed: pg_sift: an index scan of a `sift_col` index that PostgreSQL runs again for each row of an outer query (the inner side of a nested loop, a `LATERAL` join, a correlated subquery) returned some rows twice while the index held rows in its write buffer (after `INSERT` or `UPDATE`, before they are merged): the scan finds the buffered rows of an equality in their records, and when the first scan found that the equality kept more than a quarter of them it loaded the buffers instead and kept that source, so a later scan of an equality that kept few listed the same rows a second time. Present in every release since 4.0.0 (reproduced on 4.0.0, 4.0.4, 4.1.0 and 4.1.3); the rows were doubled, so the counts and sums computed from them were too. Until the update, `SET sift.enable_col_scan = off` for the statement avoids it. Held by `col_wbuf_lookup_rescan`.
- Fixed: pg_sift: `NOT (doc @> '{"b": "x"}')` (and `NOT` of a path test that the index runs as the equality of a text leaf) on a `jsonb` column with a `NOT NULL` constraint returned the rows that match instead of the rows that do not, when the aggregate scan answered it (`WHERE`, `FILTER`): the negation flipped a test it did not know. Such a `NOT` of anything but an `IN` list, an array element test or a NULL test keeps the normal plan. Held by `agg_jsonb_not_exist` (13 tests x 3 forms, on a `NOT NULL` and a nullable column)

## 4.1.3 - 2026-11-06

- Fixed: pg_sift: `REINDEX INDEX CONCURRENTLY` of a `sift_col` index with `track_changes` (the default for a table with a primary key) took ACCESS EXCLUSIVE on the table when it dropped the old index, since 4.0.0: the copy made two triggers of its own, PostgreSQL moves what depends on the old index to the new one at the swap, and dropping the old index dropped the copy's triggers (`DROP TRIGGER` needs the lock). The writers queued behind it, deadlocked with the swap (`deadlock detected`, the writer's transaction fails) and slowed down: 8 clients (an update, a delete and an insert-on-conflict in each transaction) while a loop reindexed, 40 s: 4 reindexes done, 270 deadlocks, 1.4% of the transactions failed, 400 transactions a second; a btree index in the same test: 650 reindexes, no deadlock, 32,900 a second. The copy now has no triggers of its own: the two of the old index move to it, and dropping the old index drops nothing of the table (it holds ShareUpdateExclusiveLock only): 587 to 655 reindexes, no deadlock, no failed transaction, 25,500 to 26,200 a second (a `sift_col` index with `track_changes = 'off'`: 634 to 663, 29,800 to 36,100). A reindex cancelled in the middle leaves an invalid copy that owns no trigger. The log of the copy is as incomplete as before (writers run beside the build; `REINDEX INDEX` without `CONCURRENTLY` completes it). Until the update: `ALTER INDEX i SET (track_changes = 'off')` before `REINDEX INDEX CONCURRENTLY`, and `ALTER INDEX i SET (track_changes = 'id')` after (the table's key column if it is not `id`). Held by `col_reindex_concurrent` (TAP) and `col_track_reindex_concurrently`.

## 4.1.2 - 2026-11-06

- Fixed: pg_sift: indexes written by 4.0.0 to 4.1.1 are re-checked on their own; results already wrong are repaired without `REINDEX`. The fixes of 4.1.1 stopped new wrong counts of `sift_col` indexes with `track_changes`, but the old versions an earlier release had left in the log (a statement whose command id moved ended row versions without a tombstone) stayed, and the index went on counting them after the update, `ALTER EXTENSION UPDATE`, a `sift.col_merge` and a `VACUUM` that skipped the indexes (PostgreSQL does when fewer than 2% of the pages hold a dead row: 30 stale entries of a 300,000-row table survived it); only `REINDEX`, `ALTER INDEX .. SET (track_changes = 'off')` and back, or a `VACUUM` that vacuumed the indexes took them away. The log of such an index (flags 1 in its header, `SIFT_TOMB_COMPLETE` alone) is not taken for complete any more, so its reads check the heap and answer as the heap at once, and it is re-baselined (a tombstone for every heap address no version is live for, as the install does, then the log is marked `SIFT_TOMB_COMPLETE | SIFT_TOMB_FIXED`, flags 3) by the first of: the next `VACUUM` of the table, also one that skips the indexes; the first read that checks the heap for it, which asks the worker (a table that nobody writes or vacuums is repaired too; `sift.col_auto_merge` on); the maintenance worker that a write of the table asks for. 600,000 rows with 30,000 updated, deleted and inserted since the last `VACUUM`: 111 to 119 ms and 7 to 14 MB of WAL; after a `VACUUM` 71 ms and 3 kB; `ShareUpdateExclusiveLock` only, so writers go on. A standby cannot do it and checks the heap until the primary's re-baseline is replayed. An older library reads flags 3 as an incomplete log and checks the heap, which is right. A re-baseline that aborted (the flags are set in place, the transaction is not) is done again. `sift.col_tracking` shows `complete` true and `heap_free` false until then. Held by `col_track_rebaseline` and, with the real library of 4.1.0 writing the wrong counts, `old_lib_refusal` (`scripts/old-lib-check.sh track`). Corrects the notes of 4.1.1, which gave the time of `ALTER INDEX .. SET (track_changes = 'off')` and of turning it back on a thousand times too long (a decimal comma of psql read as a thousands separator): 35 ms on 600,000 rows, and about 0.1 s to turn it back on. This repair covers a cluster upgraded by replacing the library (the usual package update), not `pg_upgrade`: after `pg_upgrade` (the restore recreates the triggers with a new xmin, so the witnesses of the log no longer match), a dump restore with triggers disabled, or `REINDEX CONCURRENTLY`, every tracked `sift_col` index has flags 1, `triggers` false and `heap_free` false, damaged or not (true since 4.0.0), the answers are right but the reads check the heap, and nothing but `REINDEX INDEX <index>` or `ALTER INDEX <index> SET (track_changes = '<value>')` (the value the index had) marks it, for every tracked index, one by one. An automatic repair for this case is planned for 4.2.
- Fixed: pg_sift: a transaction that took its snapshot before running `REINDEX`, `CLUSTER` or `ALTER INDEX ... SET (track_changes = ...)` on the same `sift_col` index with `track_changes`, and read after another session committed changes, missed the rows those changes ended (an `UPDATE`'s old versions, a `DELETE`): a wrong answer in every release since 4.0.0 (reproduced on 4.1.0 and 4.1.1: `REINDEX` and `CLUSTER` of an index with `track_changes` on a key, `ALTER INDEX` from a key to `'ctid'` and back, with an explicit `REPEATABLE READ` snapshot). Any other session was right, and so was `track_changes = 'off'` to a key and a key to `'off'` in the same transaction (a `READ COMMITTED` statement takes its snapshot after the command). The command writes a baseline of tombstones, and the transaction that wrote it took them for deletes its older snapshot sees through; its reads now check the heap (`isolation/col_track_own_baseline`, found by `test/fuzz/txn_fuzz.py`).
- Fixed: pg_sift: a `sift_col` index with `track_changes` counted the rows of an aborted or still-open `INSERT ... ON CONFLICT` in the heap-free read, until the write buffer was promoted (the fuzz repro: 1,052 rows against the heap's 1,041; `col_heapfree_xmin`: 40 more). A set of write buffer records is stamped "every inserter committed before this xid" so that a read takes its rows for visible without the heap; the record of an upsert has no inserter, and the first of them was taken for the record before it, which left the set stamped when a statement committed later. Present in 4.1.0 and 4.1.1 (since the heap-free reads of `track_changes`). The damage does not outlive the buffer: the promotion checks the rows against the heap (40 rows too many before `sift.col_promote`, none after, on the library of 4.1.0), so nothing needs repair after the update. Held by `col_heapfree_xmin`.
- Fixed: pg_sift: a backend could crash (SIGSEGV, the server restarts) on a BM25 query that runs the scan again for each row of an outer query (a correlated subquery, a lateral join) and rechecks phrases (`body @@> '"Hund und Katze"'`): the scan rescan deleted the context that the recheck of the next row worked in, and the context another part of the backend made next took its place, so the recheck reset it. A text search configuration's stem memo (`stemmer = 'german'`) was the context hit when the backend had not stemmed before; with any other context the recheck wrote into memory that was not its own. Present since 4.0.
- Fixed: pg_sift: a session that wrote a row version in a transaction that aborted and then saw a VACUUM truncate the heap could fail in its next statements on a `sift_col` index with `could not read blocks N..N in file ...: read only 0 of 8192 bytes`: the lookup of the write buffer returned the addresses it had cached for the rows of the aborted transaction, past the new end of the heap. A fresh session never saw it; it is an error, not a wrong answer. Present in 4.1.0 and 4.1.1 (found by `test/fuzz/txn_fuzz.py`). Held by `col_wbuf_lookup`.

## 4.1.1 - 2026-11-06

- Fixed: pg_sift: the results of aggregates over a `sift_col` index with `track_changes` (automatic for a table with a primary key) could count old versions of rows, in every release since 4.0.0 (reproduced on 4.0.0, 4.0.2, 4.0.3 and 4.1.0; the cases with an integer key did not show it): in a new session whose first statement that needed a table the backends share (the dictionary of a text or varchar key, the statistics of the index, the choices of the planner) was an `UPDATE`, `INSERT ... ON CONFLICT DO UPDATE`, `DELETE` or `COPY`, the registry of the dynamic shared memory attached it in a subtransaction, whose commit advanced the command id under the statement; a row hit by two statements of the same transaction was then counted twice after the commit. A table is attached by the next statement that reads now, and the statements that write go without it. Until you update: `ALTER INDEX <index> SET (track_changes = 'off')` on the `sift_col` indexes of those tables (no rewrite and no `REINDEX`; every row is then checked against the heap, which also repairs an index that was already wrong; after the update `SET (track_changes = 'id')` turns the tracking back on); for the first route only, `sift.col_dict_shared_cache_kb = 0` (with `sift.col_leaf_cache_kb = 0` and `sift.verdict_cache_kb = 0`: each backend then parses its own dictionaries; superuser, `postgresql.conf` and a reload, `ALTER SYSTEM` or `ALTER DATABASE .. SET`, no restart) also avoids it, and `shared_preload_libraries` does not. Held by `col_dsm_write_statements`.
- Fixed: pg_sift: aggregates over a `sift_col` index with `track_changes` (automatic for a table with a primary key) counted rows twice after an `UPDATE`, `MERGE`, `INSERT ... ON CONFLICT DO UPDATE` or a cursor `UPDATE ... WHERE CURRENT OF` that called a function that runs SQL inside the statement (a plpgsql function with `PERFORM` or `EXECUTE`, a trigger that updates the table): the function advances the command id inside the statement, and the entries of the versions the statement ended were looked for, and their tombstones written, with that command id instead of the statement's, so after the first call the probe of every row no longer saw the version the statement ended and its entry stayed. The command id of the statement is taken once and used for both. The same wrong count came from a bulk `UPDATE` of a table that also has a `sift` (BM25) index: the index seals its segments in a subtransaction of the statement, whose release advances the command id. Confirmed on 4.1.0 and expected on every release since 4.0.0, where `track_changes` began. Until you update: `ALTER INDEX <index> SET (track_changes = 'off')` on the `sift_col` indexes of those tables (no rewrite and no `REINDEX`; every row is then checked against the heap, which also repairs an index that was already wrong; after the update `SET (track_changes = 'id')` turns the tracking back on). Held by `col_stmt_cid` and `col_stmt_cid_bm25`.
- Improved: pg_sift: a `sift_col` index with `key_cut` that takes inserts all day and collects small segments over the keys of the new rows (overlays) is merged when the merge pays: the units and segments a request for a typical key of the overlays would no longer open, times the requests the index has had, against the rows the merge rewrites. `sift.col_maintenance_due` says `overlay units`, and VACUUM (alone, with no worker), the maintenance worker that the reads of a backend start (autovacuum of the table need not run) and `sift.col_maintain` merge the overlays into one segment and leave the partition-sized segments as they are. The merge is paced: a pass takes an eighth of `max_wal_size` of overlays at most, or less, the WAL budget of the churn (`sift.col_auto_merge_wal_pct`) does not hold it (a server that writes nothing else is when it costs least) but sets the size of a pass, that percent of `max_wal_size`, and the worker sleeps as `sift.maintenance_cost_delay` says. An index nobody has read, or with fewer than two overlays, is not merged, and a standby never is. Changed with it: an aggregate scan over a `sift_col` index now counts into `pg_stat_*_indexes.idx_scan` (it counted nothing before), which is what tells the rule that the index is read. The rows are the same (test `col_overlay_merge`, counts of units and segments, not time).
- Fixed: pg_sift: `DROP TABLE` (and the other commands that lock a table and then its indexes) could deadlock with the maintenance worker of a BM25 index, in every release since 3.7.1 (4.0.x and 4.1.0 included; BM25 indexes only): the worker held the index and, in the middle of a merge, asked for the table, which the command held while it waited for the index, and one of them failed with `deadlock detected`. Nothing was lost: the statement or the worker was aborted, and running the statement again worked. The workers of the BM25 and `sift_col` indexes and the SQL functions that maintain or report on an index now lock the table first and then the index, as a statement does; a worker whose table or index is dropped meanwhile ends without a message.
- Fixed: pg_sift: a backend could end with a segmentation fault when a statement ended whose scan probed the btree of a lookup table directly through an Index Only Scan child (the plans of `sift.agg_probe_rewrite`): the heap slot of that probe was allocated in the scan node's memory context but listed in the executor state's slot table, which is reset after the node's context is deleted. Which backend crashed depended on the plan the shared verdict cache gave it. It was found with the cheaper price of that probe below, which makes the plan win more often; 4.1.0 contains the same allocation (the plan seldom wins there, and the test of the shape, `agg_probe_direct_slot`, runs without a crash on it); updating is recommended. The slot is now allocated in the executor state's context, and a testing build errors when a scan node ends with a slot of the table in its own context.
- Fixed: pg_sift: a statement that names its entities with `IN (SELECT ..)` over a table of 10,000 rows or more (a group's membership table) in a CTE ran the normal plan: the semi join was made a probe of that table's btree inside the CTE that only names the keys, and flattened into the statement it left the CTE with no join condition of its own (`join_condition`), as the dimension re-joined to the keys did before. A query level that reads no table with a column index and no column of the query around it (the CTE, VALUES list or sub-select that names the keys) is no longer rewritten to probes, and a left join of the dimension to those keys is planned as written. The tables of the statement that the scan reads (the events and what they join) are probed as before.
- Changed: pg_sift: `sift.explain_why` and `sift.explain_refusals` name the tables that wait when no order of the joins of a statement looks each table up after the tables its keys read (`join_condition`), where the refusal gave no detail.
- Fixed: pg_sift: a statement that names its entities with `IN (SELECT ..)` in a CTE and joins them to a table of 10,000 rows or more by the key (`FROM entity_ids AS pids JOIN entities ON entities.id = pids.id`, then the events of those entities, the measures of an event through `LATERAL jsonb_each` and a `LEFT JOIN` of a table keyed by two columns) ran the normal plan: the join to the large table was made a probe of its btree for each key the keys CTE gives, which left the CTE with no join condition of its own once the statement was flattened, and the scan found no order to look the joins up in (`join_condition`, with no detail). A table that an inner join ties only to a CTE, a `VALUES` list or a sub-select that reads no table with a column index is no longer made a probe: the planner's own nested loop probes it by those keys and the scan looks its rows up by the keys of the events.
- Improved: pg_sift: `track_changes = 'ctid'` keeps the log of ended rows for a table that has no key the index holds (no primary key, or an index without it): two AFTER ROW triggers write the heap address of each version an UPDATE or DELETE ends (for a version made by a HOT update, the root of its chain), so grouped aggregates over unvacuumed rows read no heap page there either. It is never chosen by `auto`: name it. 200,000 rows: an UPDATE of 5% in an indexed column +16.8% time and +22.1% WAL, a DELETE of 2% +2.8% time and +37.6% WAL, INSERT +0.8% time and +2.2% WAL (the transaction id of each row in the index), one-row transactions +5.7% to +9.3%; on a table with a key the same index by address costs +26.1% and -5.1% where the key's triggers cost +52.6% and +20.9%. PostgreSQL queues an event for each row a statement updates or deletes, about 42 bytes a row that it does not spill: one statement over 10,000,000 rows raises the peak memory of the backend by about 420 MB (the key's transition table spills to disk); run such a statement with `track_changes = 'off'`. One NOTICE at `CREATE INDEX`, `REINDEX` and `ALTER INDEX .. SET` says so. Held by `col_track_ctid`, `col_track_ctid_chains` and the TAP `col_track_ctid`; the budget is `bench/steps/track_gate.sh`.
- Fixed: pg_sift: the maintenance worker that vacuums for `sift.col_auto_merge` takes a table for clean only when the visibility map also holds its pages (a VACUUM that began before a transaction committed wiped that transaction's rows from the statistics, so the pages it wrote stayed out of the map until the next change: 31 of 195 pages all-visible after 6,000 rows updated, instead of all), and a worker ended in the middle of a VACUUM frees its memory context at exit.
- Improved: pg_sift: a `sift_col` index asks for the VACUUM of its table when the transaction that builds it (`CREATE INDEX`, `REINDEX`) leaves 16 pages or more out of the visibility map, or when a transaction writes it a thousandth of the table (1,000 rows at least) by INSERT, COPY or UPDATE, and starts the worker at the commit: a table that was just loaded answers from the pre-aggregates of its units at once, where its first reads checked every row in the heap until a scan, autovacuum or the next insert asked (200,000 rows loaded, then indexed, a grouped sum 8.4 ms and 2,900 blocks before, 0.03 ms and 10 blocks after; `sift.col_auto_merge`, on by default, stops it).
- Fixed: pg_sift: a table that was loaded and not yet vacuumed or analyzed (`pg_class.reltuples` is -1 until then, and `relallvisible` 0) is also planned with the probe rewrite: the rewrite of an `EXISTS`, `NOT EXISTS`, `NOT IN` or join to a large btree table asked `pg_class` for the rows and refused a table without them, so the scan read the whole 1,000,000-row table or left the statement to PostgreSQL's plan. It takes the planner's own estimate of the rows (the pages and the row width) now. On 200,000 events and a links table of 1,000,000 rows, never analyzed: a NOT IN 48.5 ms and 618.9 million instructions before, 3.5 ms and 13.4 million after (vacuumed: 1.4 ms, 8.3 million); a join 41.5 ms, 496.7 M before and 7.3 ms, 21.4 M after; the IN list of a keyed table 157.3 ms and 889.4 M (PostgreSQL's plan) before, 14.5 ms and 67.8 M after; the entries of an index 34.7 ms before, 10.9 after (test `agg_probe_unvacuumed`; analyzed but not vacuumed, the plans were the probes already, at 4 to 12 ms, where the heap is asked for the rows of the entries until the first `VACUUM`).
- Changed: pg_sift: the price of a run of a correlated `LIMIT 1` child that the scan probes in its btree directly (`AGG_PROBE_RUN`) is 800 operators, where it was 1,800: 12,000 to 15,800 instructions are measured for a key, and the statement of 3,000 keys over 193,653 events and a table of 600,000 rows was priced at 14,965 for the scan against 11,454 for the normal plan and took the normal plan in a vacuumed table (75.6 ms, 523.1 million instructions), where the scan takes 4.9 ms and 38.3 million (7,475 against 11,454); the choices of the plan gate do not move (test `agg_probe_unvacuumed`).
- Fixed: pg_sift: the keyed load of a lookup table (one btree scan with the keys of the scan's IN list, instead of the table read whole) is priced by what it takes: 18 operators an entry and 4 a key (the entry 9 less from a covering btree on an all-visible page), where it was 150 and 30 against 9 for a row of the table read whole, so the btree lost to the whole read at a tenth of the rows where the two take the same at a half; a table of another access method (`sift_table`) keeps 150 an entry. Measured on 48 reads (tables of 50,000, 150,000 and 500,000 rows, a unique key and 20 rows a key, 1%, 10% and 50% of the keys, never analyzed, analyzed and vacuumed): 2,100 instructions and 178 ns an entry, 1,000 and 93 ns a row read whole. The planner's choice was wrong by more than 5% of the instructions in 16 of 48 and is in 0 (2,702 M instructions and 357 ms lost against the best read before, 117 M and 118 ms after; for example 500,000 rows, 5,000 keys, 20 rows a key, vacuumed: the table read whole 479 M instructions and 34 ms, the btree 32 M and 7 ms). The size of the table the load prices is the planner's estimate (it was `reltuples`, -1 before a VACUUM or ANALYZE, which made a unique key of a table never analyzed win the btree by accident: 150,000 rows, 20,000 keys, 8.6 ms), and a table never analyzed that has no statistics for the rows a key has gets them counted in the btree for up to four keys of the list (20 rows a key, 150,000 rows, 750 keys: the table was read whole, 309 M instructions, and the btree takes 181 M). Test `agg_lookup_keyed_crossover`, calibration table in docs/AGG.md.
- Improved: pg_sift: the direct probe of a correlated child takes an integer column compared with the key of another integer type (`status = 0` on a `smallint` column, an `int` column against a `bigint` key; a key beyond the range of the column matches no entry), which left the child to run its own Index Only Scan for each key, and an Index Only child's filter on an index column (`INCLUDE (label)`) rejects an entry before the visibility map or the heap is asked about it, as the entry's columns are those of its own version: where the pages of a table not yet vacuumed are not all-visible, each entry cost a heap fetch to be rejected after it. 200,000 events and a links table of 1,000,000 rows, `NOT EXISTS (.. AND status = 0 AND label = 'lbl_7')` for 200 parents, never analyzed: 12.3 ms, 14,206 blocks and 44.5 million instructions before, 4.0 ms, 698 blocks and 24.6 M after (analyzed only 11.7 ms, 14,194 blocks before, 4.7 ms, 699 after; vacuumed 4.6 ms, 839 blocks, unchanged); a join to the same table 4.7 ms and 1,241 blocks, a NOT IN 1.5 ms and 239 blocks (tests `agg_probe_direct`, `agg_probe_unvacuumed`).

## 4.1.0 - 2026-11-05

- Fixed: pg_sift: the direct probe of a correlated child (`LEFT JOIN LATERAL (.. LIMIT 1)`, the plans of `sift.agg_probe_rewrite`) kept its heap fetch open after each batch of keys, and PostgreSQL 19 keeps the pin of the last heap page (and of its visibility map page) in the fetch until it ends: a build with assertions printed `WARNING: resource was not closed` at the end of every statement that ran the probe. The fetch is begun for each batch and ended with it (test `agg_child_warm` on PostgreSQL 19).
- Improved: pg_sift: the sum of a float column that is not grouped (`SELECT sum(value_a) FROM t`, `avg`, a filtered sum) keeps the running sum in a register while a chunk is added up, where it updated the one group's slot in memory and each row waited for the store of the one before it: the sum takes the rows in the same order, so the result has the same bits, and a grouped sum is unchanged
- Improved: pg_sift: an `ORDER BY .. LIMIT n` over a table joined by a text key (`JOIN keys ON keys.k = t.title`, `IN (SELECT ..)`, `EXISTS`; an inner join needs a unique index on the key) is answered by the Top-N scan, as the joins by an integer or date key are: the keys of the joined table are the exact condition `title = ANY(keys)` of the scan, so no join is left for the rows of the heap. 600,000 rows and 200 keys, `ORDER BY dim_a_id DESC, id LIMIT 10`: 6,461 M instructions, 1.8 M buffer accesses and 3.6 s become 118 M, 188 and 0.14 s (30,000 rows and 20 keys: 23.6 M instructions become 2.0 M); the rows are those of the plan without the extension. Held by `agg_topn_join_text`. No REINDEX.
- Improved: pg_sift: an aggregate of a column of the scanned table under a `RIGHT` or `FULL JOIN` to a lookup (`count(f.id)`, `sum(f.v)`, `max(f.t)` .. `FROM f RIGHT JOIN dim ON dim.id = f.k GROUP BY dim.g`) runs on the kernels of the column, as under an inner join, and no longer builds an id for each distinct value of the column in a hash table: the scan costs much less and is chosen over the normal plan.
- Improved: pg_sift: the probes of a join or sub-select over a large btree table (`sift.agg_probe_rewrite`) take the forms written by hand as well: a comma join (`FROM x, big WHERE big.k = x.k`), `NOT IN` where neither side can be NULL (a declared NOT NULL column, or a condition of the WHERE that a NULL fails; otherwise the statement keeps PostgreSQL's plan), a statement with sub-selects, lateral entries, whole-row Vars or a named join (`j.col`) that read the table, and a `NOT EXISTS` over several tables of the FROM list. The table the scan reads (a `sift_col` index, a `sift_table`) is never the one probed. The child of a probe has no bitmap scan, which the scan cannot walk in the btree. The plan with the probes wins by a tenth, not by half, of the other plans' price.
- Improved: pg_sift: a float column grouped by the buckets of its values is evaluated for each bucket: a float8 column (a float4 one through its conversion) that every expression of a key, an aggregate argument or a filter reads only through `floor`, `ceil`, `trunc`, `round` or an integer cast of itself, of itself divided by a constant or of itself times a constant (`floor(x / 5)::int`, `round(x / 10)`, `trunc(x * 0.25)`) takes the key of a row from the bucket number of its value, which the vector kernels of the scan compute (AVX2, AVX-512 and NEON, bit for bit the scalar function's: x / c or x * c rounded to nearest even and moved to the floor, the ceiling or the truncation, within int32), and PostgreSQL's expression runs for the first value of each bucket where it ran for every distinct value (a measurement column has a value a row). A value with no bucket number (NULL, NaN, an infinity, a quotient or product that overflows or underflows to 0, which PostgreSQL raises for, a result outside int32, a negative zero) keeps the id of its value, so the errors, the signed zeros and the key of a NULL are PostgreSQL's. The case `float_step_buckets` (1,000,000 rows with a distinct value in almost every row, 200 buckets) takes 107.5 M instructions where `sift.agg_float_step = off` takes 624 M instructions, and reads the same blocks. A statement that reads the column in another way as well (the column itself, two different steps) keeps the ids of its values. New setting: `sift.agg_float_step` (on).
- Improved: pg_sift: the key test an inner or semi join on a text column gives the scan now also covers a text field of the elements of an array (`FROM t, jsonb_to_recordset(t.tags->'items') AS o(sub text) JOIN keys ON keys.k = o.sub`): each unit keeps the rows with an element that holds a key before the elements are resolved one by one, and a list of keys is tried up to a sixteenth of the table's rows (it was 256 keys), dropping itself after its first units when it keeps most of the rows. `sift.agg_join_filter = off` turns it off. No REINDEX.
- Improved: pg_sift: an inner or semi join on a text column (`WHERE body IN (SELECT k FROM keys)`, `JOIN keys ON keys.k = t.body`) with up to 256 distinct keys tests the column against the keys on the encoded form of each unit, as the integer joins do: units whose dictionary holds none of the keys are not read, and the long strings of an inline unit that match no key are no longer copied into a dictionary and hashed one by one. `EXPLAIN ANALYZE` shows `Join Text Filters`; `sift.agg_join_filter = off` restores the plain join. No REINDEX.
- Improved: pg_sift: the in-memory dictionary table of the strings of a unit hashes long strings in wide steps, where it hashed them with PostgreSQL's `hash_bytes`: a scan that reads the long strings of an inline text column into a dictionary (a join or `GROUP BY` on it) hashes them faster. Nothing stored changes.
- Improved: pg_sift: a condition against a scalar sub-select that gives one row whatever the data (`WHERE amount > (SELECT avg(amount) FROM t)`, `kind_id = (SELECT id FROM kinds WHERE name = 'x')` on a unique name) and the same sub-select in an aggregate's argument are tested by the scan, which asks the sub-select once when it starts, as it asks a parameter of the statement; they kept the whole query out of the scan. A sub-select that may give a second row (it raises when a row reaches it) keeps the normal plan, and so does a query the parallel plan runs: the scan with such a parameter runs in one process.
- Improved: pg_sift: a sum of a numeric column times a value the statement asks once (`SELECT grp, sum(amount * (SELECT count(*) FROM kinds)) FROM t GROUP BY grp`, `sum(amount * $1)` in a generic plan) is the sum of the column times that value: the scan sums the column and the product is made once for each group above it, with the same rows, display scale, NULL and NaN (test `agg_factor_constant`, case `agg_sum_scalar_factor`)
- Improved: pg_sift: a scan that reads a scalar sub-select as a constant (`WHERE amount > (SELECT avg(amount) FROM t)`, `sum(amount + (SELECT max(w) FROM kinds))`) runs in the workers the plan asks for, where it ran in one process: the leader runs the sub-select when the scan starts and each worker gets its value, with the same rows (test `agg_scalar_initplan`, case `filter_scalar_subselect_bound`)
- Improved: pg_sift: a sum of a numeric column times a value of a table joined by its unique key (`SELECT t.region, sum(e.amount * k.rate) FROM events e JOIN tenants t .. JOIN kinds k ON k.id = e.kind_id GROUP BY t.region`) adds up the amounts for each key first: the scan groups by the keys of the query and the kind and sums the amounts, and the query above joins the groups to the lookup and adds up the products, with the same rows, display scale, NULL and NaN (test `agg_eager_lookup`, case `agg_sum_lookup_product`)
- Improved: pg_sift: a group table that outgrows hash memory costs less for each group it writes to the partitions and merges back: the partition of a group is hashed a key at a time over the groups with a multiplicative mix of the key values (the hash of the bytes of each value was most of the write), and the ids of an integer key whose values are far apart (the table of open addressing) are probed in line, a value that has an id costing a load and a compare. A spilled scan hashes the groups into partitions as before; the rows are those of the ordinary plan.
- Improved: pg_sift: a scan with two or more integer keys whose direct arrays (the capacities of the keys multiply) are more than hash memory puts its groups in the hash table at once, and is read in a few passes of the groups that the memory holds, where it started the scan again for every share on a table that was over the memory before a row was read; the table of ids of a single key is no longer sized past hash memory for the groups the planner expects, so the first chunk does not put it over.
- Improved: pg_sift: a scan whose group table is several times hash memory chooses the number of passes from the planner's estimate of the groups and reads the table once for each pass, where it read it once to find out, threw that scan away and read it again for every share; a pass takes the share of the groups that the memory measured in the last one holds, the groups of a pass are returned as they are asked for, with no store of the result, and the shares of a pass are one range of hashes that the scan tests in a vector. A statement with float sums, `count(DISTINCT)`, or a parallel scan is read as before.
- Fixed: pg_sift: the passes of a scan (and the shares of a parallel scan) split the groups by a multiple of the golden ratio, as the table of open addressing of the ids of an integer key hashes, so the values of one share crowded one part of that table and a probe walked long runs of slots: the split multiplies by another constant.
- Improved: pg_sift: a join to a large table by a btree column (`x JOIN big ON big.k = x.k AND ..`, `LEFT JOIN`) and an `EXISTS`, `NOT EXISTS` or `IN` over one are also planned as the lateral child that probes the btree for each distinct key the scan meets (`sift.agg_probe_rewrite`), where the keys are not a list the scan names: the table is not read whole, only the entries of the keys the scan has. The plan with the probes is kept where its price (the entries each probe examines, in an Index Only Scan or from the heap) is lower than the table read whole, the keyed load and the ordinary plan; a join of most keys, or of rows that are fetched from the heap, keeps the other plans. The rows are the same, with NULL keys, keys twice and keys the table has no row for, in a serial and a parallel scan.
- Improved: pg_sift: a table read by a btree for the keys of the scan (the scan's `IN` list, a dependent lookup) takes its rows from the entries of the index when the index holds the columns the join reads (key columns or `INCLUDE`) and the heap page is all-visible, as an Index Only Scan does: no heap fetch, no page visit; a page that is not all-visible is fetched from the heap, row by row, and `EXPLAIN` shows the count (`heap fetches`). Of two indexes that start with the key, the planner and the loader take the one whose entries cost least for a key (the rows its next columns leave, a cheaper row where it holds the columns), and the price of the keyed load drops with the share of all-visible pages, so an `EXISTS`, `NOT EXISTS` or join to a large table with a covering index is taken by the scan for more keys than a heap read was; `NOT EXISTS` and `LEFT JOIN` to a table keyed by the scan's `IN` list are priced like the inner and semi joins.
- Improved: pg_sift: a row of the table looked up by its unique key and an `EXISTS` in the select list, the `GROUP BY`, an aggregate's argument or its `FILTER` of a grouped query (`SELECT sum((SELECT d.w FROM dim d WHERE d.id = f.dim_id)) FROM f GROUP BY ..`, `count(*) FILTER (WHERE EXISTS (SELECT 1 FROM notes n WHERE n.dim_id = f.dim_id))`) are one left join, which the scan reads in its pass, in place of one probe for each row; the plan is taken where its cost is the lower. A row with no match is NULL, as the sub-select is (an expression of the looked up columns, `coalesce(t.col, 0)` among them, is asked only for a row that was found), and the rows are those of the ordinary plan.
- Improved: pg_sift: the keys of a correlated `LIMIT 1` child are probed in its btree a batch at a time: the aggregate scan sorts the keys of a chunk, descends the child's btree once and walks from the leaf of one key to the next while the next key is on that leaf (or the pages under the same parent), comparing text and array keys as bytes, where it ran the child's plan once for each key. The rows are those of the child's index scan (visibility, dead entries, HOT chains, duplicate keys, NULL keys, a split during the descent); a child with a filter or projection that reads a parameter of the outer row or runs a sub-plan, a key that is not an equality of the index column's own type, an index that is not a btree (or has no heap-TID keyspace), a `LIMIT` other than 1, a parallel or backward scan and a serializable transaction (the probe takes no predicate locks) keep the child's plan for every key. The case `lateral_probe_direct` (120,000 events, 1,000 entities) takes 606 M instructions where `sift.agg_probe_direct = off` takes 1,657 M instructions, and reads the same blocks cold (one btree block more in 7,429 in the run). `sift.agg_probe_direct = off` runs the child's plan for every key; held by the tests `agg_probe_direct` and `agg_probe_direct_race`.
- Improved: pg_sift: a table joined by the column that the scan's `IN` list names (`JOIN links ON links.parent_id = e.foreign_id .. WHERE e.foreign_id IN (..)`, or `EXISTS`) is read by one scan of its btree with the list as the scan keys, and the join's `column = value` conditions on the next columns of the index as further keys, where the table was read whole: only the entries that can match are fetched from the heap. The plan prices the scan against the table read whole (a descent for each key, the heap rows of the entries they find and the pages those visit, at what a page costs where the table's pages are, against the rows and pages of the table), so a list of most of the keys keeps the whole load, and an inner or semi join to a large table that the normal plan did better on is now taken by the scan. The rows are those of the whole load (NULL and repeated elements, keys the table has no row for, serial and parallel scans).
- Improved: pg_sift: the direct probe of a correlated `LIMIT 1` child takes an Index Only Scan child as well (an `EXISTS` over the columns of the index, a lookup of an `INCLUDE` column): the entries of the key are walked on the leaf, the visibility map decides which of them ask the heap, and the leaf stays pinned until it has been asked (a VACUUM could otherwise remove the entry of a dead tuple and mark the page all-visible between the read and the check). The case `lateral_probe_index_only` (3,000 keys, 900,000 lookup rows) takes 45.2 M instructions where `sift.agg_probe_direct = off` takes 57.3 M instructions, and reads the same blocks. The probe starts from the index AM's own root lookup (the relation cache's copy of the btree metapage), so it reads no block the child's plan does not; held by the tests `agg_probe_direct` and `agg_probe_direct_race`.
- Improved: pg_sift: the keys of a filtered table joined to a grouped sub-select are looked up in the filtered table before the groups are made (`SELECT .. FROM (SELECT k, count(*) FROM t GROUP BY k) a JOIN dim ON dim.id = a.k WHERE dim.name = 'x'`): the sub-select asks for the keys the table keeps, which the planner reads as a few probes of an index of `t` where the groups of the whole table cost a scan or a pass over an index. The plan is taken where its cost is clearly the lower, with the scan or without it; the join above is unchanged, and so are the rows. Only a table with a unique key and a condition that picks rows by constants asks for it: a statement that joins a grouped sub-select to a table that repeats its key is planned once. 300,000 orders and 1,800,000 fact rows with a b-tree on the key, warm medians of 5 on one backend (case `group_join_dim_filter_keys`, run e8cd177d0886-20261105T194357Z-7e16): the normal plan (`sift.enable_agg = off`) takes 9.2 ms, 112.8 M instructions and 182 buffers, the keys looked up 0.34 ms (0.24 ms of it planning), 1.27 M instructions and 32 buffers.
- Improved: pg_sift: the keys of a correlated `LIMIT 1` child are probed in its btree a batch at a time: the aggregate scan sorts the keys of a chunk, descends the child's btree once and walks from the leaf of one key to the next while the next key is on that leaf (or the pages under the same parent), comparing text and array keys as bytes, where it ran the child's plan once for each key. The rows are those of the child's index scan (visibility, dead entries, HOT chains, duplicate keys, NULL keys, a split during the descent); a child with a filter or projection that reads a parameter of the outer row or runs a sub-plan, a key that is not an equality of the index column's own type, an index that is not a btree (or has no heap-TID keyspace), a `LIMIT` other than 1 and a parallel or backward scan keep the child's plan for every key. The case `lateral_probe_direct` (120,000 events, 1,000 entities) takes 606 M instructions where `sift.agg_probe_direct = off` takes 1,657 M instructions, and reads the same blocks cold (one btree block more in 7,429 in the run). `sift.agg_probe_direct = off` runs the child's plan for every key; held by the tests `agg_probe_direct` and `agg_probe_direct_race`.
- Changed: pg_sift: the planner counts the pages of a table's visibility map with the CPU's popcount instruction where the CPU has one (x86-64 builds the baseline code, which counted the bits in C), and `sift.simd = scalar` keeps the C count: the pass over the map of a table of 20.7 million pages takes 6.8 million instructions fewer (a map of 634 pages, measured with the sparse-file lab; the counts are the same, test `agg_vis_counts`).
- Improved: pg_sift: a scan whose pages are not in shared buffers reads each unit of a `sift_col` index as one plan: once a unit's head had to come from storage, the columns that every unit before it read, and the head of the next unit, are asked for together, sorted, so the blocks that follow each other are one read, where each column was a read of a block or two in the order the code asked for it. The same rows, fewer read requests and fewer blocks (the sequential read-ahead of single misses no longer reads pages nobody asked for); a scan on cached pages does nothing different, and a column that only some units read (the columns of the rows a filter kept) is not read ahead. Counts of requests and blocks, not time. Held by `col_first_statement_reads`.
- Improved: pg_sift: a list of keys is no longer looked up in the point directories when it has more keys than 16 for each unit a segment has left: a scope list of thousands of ids joined to a table, next to a search that matches ten rows, probed the directory for every id (a block or more each, 17,000 blocks of the column index at a million rows) and cut no unit; the units answer it now and the same statement reads about 700 blocks of the index. A short list is still looked up.
- Fixed: pg_sift: a `LIMIT`, a merge join or a window above a grouped sub-select (`SELECT .. FROM (SELECT k, count(*) FROM t GROUP BY k) a JOIN dim ON dim.id = a.k ORDER BY a.k LIMIT 10`) took the scan whenever it cost less in total than the normal plan, and PostgreSQL dropped the normal plan that gives its first group at once, the one these queries want: the sub-select is planned for all its rows and ignores the startup cost. The normal plan stays a candidate next to the scan and the plan with the lower cost for the rows the query asks for is the one, except a normal plan that reads through a condition on the table (its first group is as far as that condition lets it be, which the statistics do not know for columns that move together) (test `agg_group_startup`, cases `group_join_limit_startup` and `group_union_limit_correlated`)
- Fixed: pg_sift: `GROUP BY GROUPING SETS`, `ROLLUP` and `CUBE` over keys that are expressions of a sub-select (`SELECT k, count(*) FROM (SELECT upper(kind) AS k FROM t) p GROUP BY ROLLUP (k)`, a `LATERAL` sub-select with `split_part(key, '_', 1)`, a `coalesce` of a lookup) kept the normal plan with the reason `group_key`, while the same keys under a plain `GROUP BY` were answered by the scan: the planner hands such a key to the grouping step as a placeholder that nothing else nulls, and the scan now reads it as the expression it stands for. The rows, `GROUPING()` and the NULL rows of the sets are those of the normal plan (test `agg_grouping_set_keys`, case `grouping_sets_subselect_keys`)
- Fixed: pg_sift: `SET ..; EXPLAIN ANALYZE ..` sent in one query string (`psql -c`, `\;`) as the first statement of a connection that has not preloaded the library ran the ordinary plan and did not read the index: the first statement of such a connection is planned again at the executor start, and the plan is now made from the statement the string holds for the EXPLAIN, not from the first statement of the string. Held by `agg_first_statement`.
- Improved: pg_sift: the warm-up of a correlated lookup (a `LEFT JOIN LATERAL .. LIMIT 1` into a large mapping table, which asks the OS for the rest of a relation once the probes have read a sixteenth of it) advises the OS in ranges of 1 MB, one call for each 128 blocks, where it made one call for each 8 kB block: the same blocks are asked in 128 times fewer calls, and the blocks the probes read from storage are unchanged (counts of calls and bytes, not time).
- Changed: pg_sift: a statement with lateral sub-selects that can be pulled up into the join, and no aggregate, is planned once after its first run in a backend instead of twice: a page of events with a lateral over the elements of a document and a lateral last row takes 2.4 million instructions to plan against 3.6 million, and the plan is the same (case plan_cost_lateral_pull_up; a statement a shared CTE needs already kept its choice). The statements of one shape that differ in their constants keep the choice of the form that won, rechecked every eighth time, as for a shared CTE.
- Changed: pg_sift: the estimate of a join of a column with a list of constant ranges (`event_date <@ periods.range`) no longer asks for the table's conditions together with each range when those conditions already keep the rows of the ranges (the statement says the same periods as a filter and as the join): the statistics of several columns were read for each range. A request of the events of 1, 20 and 200 parents with two periods takes 2.24, 2.37 and 2.88 million instructions to plan instead of 2.44, 2.57 and 3.09 (cases plan_cost_keys_1, plan_cost_keys_20 and plan_cost_keys_200; the estimates are the same, test `agg_range_join_rows`).
- Changed: pg_sift: the price of the scan for the rows a join adds beyond the fact table (a table with many entries for each parent joined by the column the scan groups by) follows what the planner knows: the rows the join adds, the bound columns (a column with many distinct values by its statistics), the aggregate's argument; and the normal plan of such a join is priced for the rows it hashes and aggregates. A join, `EXISTS`, `NOT EXISTS` or `LEFT JOIN` of a table of four times the rows of the fact table to 2,000 parents or to all of them, and the join over all parents of tables of 1,000,000 rows with 333 entries for each parent, were the normal plan, which takes 2.3 to 10 times the instructions (733 M against 326 M for the join at 2,000 parents, 2,196 M against 217 M for the `LEFT JOIN` over all parents), and are the scan now; no choice that was right moved (`plangate --calibrate` finds none of the 23 shapes of links and lookups wrong; figures in docs/AGG.md). The normal plan is charged for the rows that joins of tables add, not for those of a lateral function (`jsonb_each_text`, a PostgreSQL estimate of 100 rows for each call), a sub-select or a CTE.
- Changed: pg_sift: a lookup table read whole from its heap is priced at what the load takes, 7 operators for a row kept as keys and 9 for a whole row, where it was 10 and 20 (410 to 520 instructions a row were measured with the lookup cache off, whatever the row keeps): an `EXISTS` over all the parents of a table of four times the rows of the fact table was priced at the normal plan and is the scan now (3.6 million instructions against 131.6 million in the warm run, 52 against 126 with the lookup cache off; 3.7 ms against 12.2 ms), and a `LEFT JOIN` under the lane with `seq_page_cost` 100 takes the scan (22.8 million against 287.4); the 16 shapes of a 1,000,000-row validation set keep the scan they had (docs/AGG.md).
- Improved: pg_sift: an aggregate with an `ORDER BY` or `DISTINCT` in its call (`array_agg(id ORDER BY amount DESC, id)`, `array_agg(memo ORDER BY ..)`) sorts the values of a group in tuples that hold the fixed-width columns first: the sort and the transition read the column after a text one by walking the tuple (`nocachegetattr` was 31% of the instructions of a window-function rewrite over 250,000 rows with a text column). A top-3 of each kind with a text column takes 2,740 million instructions against 5,071 million and 205 ms against 282 ms (the normal plan: 3,840 million, 245 ms).
- Changed: pg_sift: the planner counts the all-visible pages and the clean superblocks of a table in one pass over its visibility map, once for each planner run, where it read the map three times (two for the statistics and one to tell whether the facts it kept are stale): a table of 20.7 million pages is 74 million instructions to plan instead of 383 million.
- Changed: pg_sift: the planner computes the share of the table that the keys of a CTE list own once for each planner run and arm, where every join and cost of the arm computed it again: a request of several arms asked for it 14 times and now asks for it at most once for each arm, 9.8 million instructions less to plan.
- Improved: pg_sift: a `key_cut` index whose runs are as large as partitions is rewritten by key range: the index of a table larger than `sift.vacuum_rewrite_limit` that a build wrote as several segments (the rows outgrew the memory of its segment writer) kept them for good, each sorted by the key and holding rows of every key, so a lookup of a few keys read a unit in each segment and the rewrite waited for churn and for the WAL budget; the maintenance rewrite now finds that the key ranges of the segments overlap, cuts them by key (not held by `sift.col_auto_merge_wal_pct`, as the rewrite of the peers of a build is not) and the same keys read one unit. The rewrite is paced: a pass rewrites about `sift.col_auto_merge_pass_mb` (1024 by default; 0 does all of it at once) of the runs, a window of those that meet in the key at a time, and the next VACUUM, read tick or `sift.col_maintain` goes on from the directory it left, so the first maintenance after an upgrade does not write an index at once; the index is valid and equal to the heap's between passes. 33 runs of 5 MB: 170 MB of WAL in one pass, passes of at most 45 MB (8 runs) and 293 MB in all (1.7x) at the default's share.
- Improved: pg_sift: `CREATE INDEX` and `REINDEX` of a `key_cut` index on a table of more than a few runs write runs that do not overlap in the key (`sift.col_build_key_ranges`, on by default; off writes the runs in the order of the table): the build reads a sample of the table for the quantiles of the key, sends the rows to a temporary file for each stretch of the key while it scans, and writes the stretches in the order of the key as runs numbered one after the other, as the rewrite of a build does. A key is in one unit where it was in one of every run, nothing is due after the build, and the WAL and the time of the rewrite that cut the runs are not spent: a build and its rewrite took twice the time and the WAL of the build. The cost is a sample of the table, temporary files of about the size of the index (no WAL; a `temp_file_limit` below that, or a disk of the temporary tablespaces with less than twice the room, leaves the runs in the order of the table), and a build about a quarter longer than the runs alone. For a key of integers, dates, timestamps, numerics or floats (the columns `key_cut` takes) and a serial build (a `key_cut` index is built by one process) (case `col_key_cut_ranges`).
- Improved: pg_sift: a scan with a long list of numbers (a semi-join table, a long `IN`) no longer compares the list with the one of the last begin at every begin, once in the zone tests and once in the keep cache: the begins of a `key_cut` index whose units follow each other (a rewritten index, a build by key ranges) are many, and each cost a pass over the list. The lookup of a couple of thousand keys costs about a seventh less CPU on such an index.
- Improved: pg_sift: a long list of numbers (a semi-join table, a long `IN`) is one test for every segment of a `sift_col` index and every begin of the scan, where each segment made, sorted and chained it again: the lookup of a few thousand keys in an index of dozens of segments (a rewritten `key_cut` index, the runs of a build) costs about a tenth less CPU, and the rows are the same (case `col_list_test_once`).
- Changed: pg_sift: the extension is version 4.1: `ALTER EXTENSION pg_sift UPDATE` from 4.0.3 runs `pg_sift--4.0.3--4.1.sql`, which changes nothing in the SQL (the two functions `sift.index_choices` and `sift.help` came with 4.0.3), and from 4.0 it goes through 4.0.3. `scripts/gen-reference.sh` counts a patch release that has update scripts but no install script as a release of its own, so those two functions are `Since` 4.0.3.
- Improved: pg_sift: a group key that is itself a `COALESCE` of two columns (`GROUP BY coalesce(status, country), customer_id`) is ordered as a key for the `ORDER BY` or window above it, and the order of an integer, date or timestamp key with many distinct values is priced by what it costs (the values are sorted by the value, with no call for a compare), where the plan kept a Sort above the scan because the key looked more expensive to order than to sort. The rows and their order are the normal plan's. Bench case `group_order_coalesce_key`.
- Improved: pg_sift: a query that groups the groups of a sub-select or an inlined CTE again with an aggregate that sorts (`array_agg(n ORDER BY customer_id) GROUP BY country`, `json_agg(.. ORDER BY ..)`, `count(DISTINCT ..)`) gets them from the scan in the order it sorts by, and no Sort is put between them (a parent that hashes keeps the groups unordered). Bench case `group_order_sorted_agg`.
- Improved: pg_sift: a `Sort` of many rows above the groups of a scan, with two or more keys or a key that is not an integer (`ORDER BY count(*) % 3, country DESC, customer_id`, `ORDER BY lower(name), id DESC`, the output of a full join of two aggregates sorted by `COALESCE` of its keys), is done by a `Custom Scan (SiftSort)` node: it ranks the distinct values of each key once with the comparator of the key (its sort operator, collation and nulls order) and sorts the rows by the ranks, so the rows come out in the order of the `Sort`, ties in the order they came in, and the comparator runs once per distinct value and not once per comparison. A scroll cursor, a merge join and a rescan read the node as they read a `Sort`; a `Sort` under a `LIMIT` and a `Sort` of a single integer key stay, and so does one that does not fit `work_mem`.
- Improved: pg_sift: the scan that gives its groups in the order of the `ORDER BY` or the window above it takes less work for each group: one pass over the groups for each sort key finds its ids and distinct values, and the ranks are packed into the sort key as they are made, so the extra arrays and the passes over them are gone; keys whose ranks do not fit one packed integer together are sorted as before.
- Improved: pg_sift: the radix sort that orders the entries of a `CREATE INDEX` (the order arrays, the point directory, the runs, the sizes of a rewrite) and the groups of an ordered scan takes fewer passes over its entries: it cuts the range of the keys into as few digits as the range needs (up to 12 bits a digit once there are tens of thousands of entries), leaves a digit that every entry shares where it is, and counts the entries of a digit just before it moves them.
- Improved: pg_sift: an aggregate with two to four integer group keys whose groups are kept in a hash table (the product of the key ranges is too large for the direct arrays) probes the table with the comparison unrolled (the ids of the keys are compared one by one, not by `memcmp`) and, for a table that fits the cache of a core, hashes each tuple as it probes, with no prefetch and no pass over the chunk for the hashes.

## 4.0.5 - 2026-11-06

- Fixed: pg_sift: an index scan of a `sift_col` index that PostgreSQL runs again for each row of an outer query (the inner side of a nested loop, a `LATERAL` join, a correlated subquery) returned some rows twice while the index held rows in its write buffer (after `INSERT` or `UPDATE`, before they are merged): the scan finds the buffered rows of an equality in their records, and when the first scan found that the equality kept more than a quarter of them it loaded the buffers instead and kept that source, so a later scan of an equality that kept few listed the same rows a second time. Present in every release since 4.0.0 (reproduced on 4.0.0, 4.0.4, 4.1.0 and 4.1.3); the rows were doubled, so the counts and sums computed from them were too. Until the update, `SET sift.enable_col_scan = off` for the statement avoids it. Held by `col_wbuf_lookup_rescan`.
- Fixed: pg_sift: `NOT (doc @> '{"b": "x"}')` (and `NOT` of a path test that the index runs as the equality of a text leaf) on a `jsonb` column with a `NOT NULL` constraint returned the rows that match instead of the rows that do not, when the aggregate scan answered it (`WHERE`, `FILTER`): the negation flipped a test it did not know. Such a `NOT` of anything but an `IN` list, an array element test or a NULL test keeps the normal plan. Held by `agg_jsonb_not_exist` (13 tests x 3 forms, on a `NOT NULL` and a nullable column)
- Known issue: pg_sift: The SQL administration functions (`index_advice`, `col_verify`, `col_clustering`, `col_refresh`, `col_rollup_refresh`, `col_unit_agg`, `col_unit_hist`, `col_tracking`, `col_stamps`, `key_cut_decision`, and `index_choices` from 4.0.3) lock the index before its table, so a `DROP TABLE` or `ALTER TABLE` that runs at the same moment can be aborted with `deadlock detected`. It is an error, never a wrong answer, and it has no effect on the data: run the statement again. Present in every release up to 4.1.4; fixed in 4.2, not in a patch release.

## 4.0.4 - 2026-11-06

- Fixed: pg_sift: indexes written by 4.0.0 to 4.0.3 are re-checked on their own; results already wrong are repaired without `REINDEX`. The fixes of 4.0.4 stop new wrong counts of `sift_col` indexes with `track_changes`, but the old versions an earlier release had left in the log (a statement whose command id moved ended row versions without a tombstone) would stay, and the index would go on counting them after the update, `ALTER EXTENSION UPDATE`, a `sift.col_merge` and a `VACUUM` that skipped the indexes (PostgreSQL does when fewer than 2% of the pages hold a dead row); only `REINDEX`, `ALTER INDEX .. SET (track_changes = 'off')` and back, or a `VACUUM` that vacuumed the indexes took them away. The log of such an index (flags 1 in its header, `SIFT_TOMB_COMPLETE` alone) is not taken for complete any more, so its reads check the heap and answer as the heap at once, and it is re-baselined (a tombstone for every heap address no version is live for, as the install does, then the log is marked `SIFT_TOMB_COMPLETE | SIFT_TOMB_FIXED`, flags 3) by the first of: the next `VACUUM` of the table, also one that skips the indexes; the first read that checks the heap for it, which asks the worker (a table that nobody writes or vacuums is repaired too; `sift.col_auto_merge` on); the maintenance worker that a write of the table asks for. `ShareUpdateExclusiveLock` only, so writers go on. A standby cannot do it and checks the heap until the primary's re-baseline is replayed. An older library (4.0.3 and before) reads flags 3 as an incomplete log and checks the heap, which is right. A re-baseline that aborted (the flags are set in place, the transaction is not) is done again. `sift.col_tracking` shows `complete` true and `heap_free` false until then. Held by `col_track_rebaseline`.
- Fixed: pg_sift: a session that wrote a row version in a transaction that aborted and then saw a VACUUM truncate the heap could fail in its next statements on a `sift_col` index with `could not read blocks N..N in file ...: read only 0 of 8192 bytes`: the lookup of the write buffer returned the addresses it had cached for the rows of the aborted transaction, past the new end of the heap. A fresh session never saw it; it is an error, not a wrong answer. Present in 4.0.0 to 4.0.3 (reproduced on 4.0.3). Held by `col_wbuf_lookup`.
- Fixed: pg_sift: a `sift_col` index with `track_changes` counted the rows of an aborted or still-open `INSERT ... ON CONFLICT` in the heap-free read, until the write buffer was promoted (`col_heapfree_xmin`: 40 more rows than the heap). A set of write buffer records is stamped "every inserter committed before this xid" so that a read takes its rows for visible without the heap; the record of an upsert has no inserter, and the first of them was taken for the record before it, which left the set stamped when a statement committed later. Present in 4.0.0 to 4.0.3 (reproduced on 4.0.3). The damage does not outlive the buffer: the promotion checks the rows against the heap. Held by `col_heapfree_xmin`.
- Fixed: pg_sift: a transaction that took its snapshot before running `REINDEX` or `CLUSTER` on the same `sift_col` index with `track_changes`, and read after another session committed changes, missed the rows those changes ended (an `UPDATE`'s old versions, a `DELETE`): a wrong answer in every release since 4.0.0 (reproduced on 4.0.3 with an explicit `REPEATABLE READ` snapshot). Any other session was right. The command writes a baseline of tombstones, and the transaction that wrote it took them for deletes its older snapshot sees through; its reads now check the heap (`isolation/col_track_own_baseline`).
- Fixed: pg_sift: a `sift_col` index with `track_changes` gave a wrong answer (rows counted twice) after an `UPDATE`, `MERGE`, `INSERT ... ON CONFLICT DO UPDATE` or a cursor `UPDATE ... WHERE CURRENT OF` that called a function running SQL (any plpgsql function with `PERFORM` or `EXECUTE`, a trigger that updates the table): the function advances the command id inside the statement, and the entries of the versions the statement ended were looked for, and their tombstones written, with that command id instead of the statement's, so the probe of every row after the first call no longer saw the version the statement ended. The command id of the statement is taken once (`sift_col_stmt_cid`) and used for both. The function can also be the maintenance of a `sift` (BM25) index on the same table: its steps run in a subtransaction, and when its write buffer seals inside the `UPDATE` (a bulk update of text) they advanced the command id the same way, with no function of the user in the statement. Present in 4.0.0 to 4.0.3.
- Fixed: pg_sift: a `sift_col` index with `track_changes` could give a wrong answer (rows counted twice: the old versions of rows updated by two statements of one transaction) when the first statement of a backend that needed one of the tables the backends share (the dictionaries of the index, its statistics, the choices of the planner) was an `UPDATE`, `INSERT ... ON CONFLICT DO UPDATE`, `DELETE` or `COPY`: the registry of the dynamic shared memory attaches in a subtransaction, whose commit advances the command id under the statement. A table is attached by the next statement that reads now (the statements that write go without it). Present in 4.0.0 to 4.0.3.
- Fixed: pg_sift: a BM25 query that runs the scan again for each row of an outer query (a correlated subquery, a lateral join) and rechecks phrases (`body @@> '"dog and cat"'`) could write into memory that was not its own: the scan rescan deleted the context that the recheck of the next row worked in, and the context the backend made next took its place, so the recheck reset it. The recheck context is now a child of the query's context and ends with the scan node. Present in 4.0.0 to 4.0.3 (the code is the same in all four); 4.1.0 and 4.1.1 crashed in the first statement of a backend through the stem memo of a text search configuration, which 4.0.x does not have.
## 4.0.3 - 2026-11-04

- Fixed: sift_table: a `COPY` into a `sift_table` with parallel workers (`max_parallel_maintenance_workers` above 0, the PostgreSQL default is 2) could fail with `ERROR: invalid memory alloc request size 1879048192` after about a second, when its workers had written every group the backend handed over before the backend looked at them (the backend is descheduled for a moment: a loaded host, a busy runner; rare on an idle machine; since 4.0.0). The statement was aborted and nothing of it committed (the blocks it had written are freed by the next `VACUUM`), and the same load run again usually worked. The backend takes the groups it handed over and no more.
- Fixed: pg_sift: the flush of the WAL that the maintenance worker of a `sift_col` index makes before the VACUUM it runs asked for the position after the header of the next page when the last record ended with a page, and the server answered `xlog flush request .. is not satisfied` (the worker's pass failed and was tried again): the end of what is inserted is flushed.
- Fixed: pg_sift: `pg_upgrade` of a cluster of a release before 4.0 failed with `sift: function sift.col_track_old_rows() does not exist` whenever the dump made a `sift_col` index after the primary key of its table (the order depends on the PostgreSQL major and on the names): the restore runs with the extension still at the old version, which has no function for the statement triggers of `track_changes`, and the default `auto` chose the key and tried to install them. The index is now built without the log, with the advice `the extension is not updated yet`, and `REINDEX INDEX` after `ALTER EXTENSION pg_sift UPDATE` tracks it; `track_changes = '<column>'` named in that state is refused with the way out. Spec: `col_track_old_extension`.
- Changed: pg_sift: `CREATE INDEX`, `REINDEX` and `REINDEX CONCURRENTLY` of a `sift_col` index print one short NOTICE (the count of the automatic choices, advice and limits, and their values on two lines, with the command that lists the rest) where each choice printed a NOTICE with a DETAIL and a HINT of its own, and one more line when the segments are of a column format that an older library cannot read. `SELECT * FROM sift.index_choices('index')` gives, for each choice, the value, the cost measured on the index, the reason and the command that turns it off, and `sift.help('slug')` returns a row of docs/messages.tsv, the one source of docs/MESSAGES.md and of those texts. A concurrent build says in one line that the `track_changes` log is not complete and which `REINDEX` makes it so. The choices are the same; `client_min_messages = warning` hides the summary. The `compression = 'auto'` figures of each column go to the server log at DEBUG1, and the two NOTICEs of the strings and of the lookup rows that were not shared are shorter. The extension is version 4.0.3: `ALTER EXTENSION pg_sift UPDATE` from 4.0 creates the two functions and nothing else, and no column format changes.
- Improved: pg_sift: the number of participants of a parallel `CREATE INDEX` of a `sift_col` index is chosen by the weighted formula from the rows, the columns and whether the index has a point directory, and no longer by one participant for each further stretch of rows: the build time another participant saves is weighed against what its segments cost every statement, the index and, with a point directory, the lookups of scattered keys, which probe each segment, so a table of a few hundred thousand narrow rows is built alone and has one segment (a lookup of 500 scattered keys reads about 40% fewer blocks than in the two segments the stretch rule gave), and larger or wider tables keep their participants. `parallel_workers` of the table and `min_parallel_table_scan_size` still ask for participants as before.
- Fixed: pg_sift: the defaults that `CREATE INDEX` chooses from the statistics of the table (`point_tid_columns`, `order_array_columns`, `order_array_group`, `key_order`, the analyze of a table that has none) are the same on PostgreSQL 17 and 18: the build counted the rows of a table from `pg_class`, which a b-tree built before it had left holding the real count on 17 and nothing on 18, where it took the planner's guess from the widths of the columns (many times the rows for a table with a long text column), so the same table was analyzed by one version and not by the other and got a different index. The rows are now those of the pages the table has times the density of the rows `pg_class` counted, or of a sample of its pages when none was counted.
- Improved: pg_sift: an `ORDER BY .. LIMIT n OFFSET m` that the top-N scan answers with its own columns as the output no longer makes the `Limit` node pull `m` rows to throw them away: the plan gives the `Limit` no `OFFSET`, and the scan counts the entries of the `OFFSET` without storing a row of NULLs and returning it for each (a negative `OFFSET` still raises its error, from the scan). `SELECT d FROM olt ORDER BY d OFFSET 100000 LIMIT 10` over a date of 730 values in 1,000,000 rows (warm, whole statements with the planner's own choice, 7 repetitions) takes 1.82 ms and 20.9M instructions against 2.86 ms and 40.9M before, and a b-tree on the date takes 2.51 and 3.00 ms (two runs) and 35.2M; the blocks are the same, 82 read cold (0.6 MB) against 90 for the b-tree (runs ff931c08bebf-20261103T180430Z-0702 and fb9b4acd6f58-20261103T181009Z-92a0). At 300,000 rows (an OFFSET of 30,000) it takes 0.52 ms and 6.8M instructions against 0.98 ms and 12.5M, and at 100,000 rows 0.29 ms and 2.8M against 0.45 ms and 4.9M: the time and the instructions still grow with the OFFSET, a tenth of the rows, at half what they did (runs ff931c08bebf-20261103T180407Z-9c89, fb9b4acd6f58-20261103T180945Z-c558, ff931c08bebf-20261103T180346Z-9649, fb9b4acd6f58-20261103T180924Z-59a1). An OFFSET of 10,000 over a text key of 1,000,000 rows takes 0.30 ms and 2.7M instructions against 0.45 ms and 4.7M, and a b-tree 0.45 ms and 5.0M (runs ff931c08bebf-20261103T180542Z-a145 and fb9b4acd6f58-20261103T181122Z-8df3). Held by `agg_topn_offset_skip`.
- Improved: pg_sift: the statement triggers of the `track_changes` indexes of one table probe the unique index once for all of them, not once for each index: with 3 indexes a DELETE of 2% of 100,000 rows costs 73.4 M instructions instead of 104.9 M and a key change of 1% 92.3 M instead of 111.9 M, each index keeping its own log.

## 4.0.2 - 2026-11-04

- Fixed: pg_sift: `pg_squeeze` of a table with a `sift_col` index lost rows of the index when writers inserted during the squeeze (about one run in three, `sift.col_verify` said `Heap tuple (..) is not in the index`; the index answered with fewer rows until `REINDEX`), with `track_changes` on or off: the squeeze applies the changes it decodes while the decoding session reads the catalogs as of the past, where the copy it buffered rows for does not exist, and the buffer of those rows was taken for the one of a dropped index and cleared. The buffer is flushed under its own OID there.
- Fixed: pg_sift: `pg_squeeze` of a table with a `sift_col` index with `track_changes` (the default since 4.0.0) failed with `track_changes: column "id" can be NULL`: its copy (`tmp_<oid>`) has no NOT NULL and no key while the index is built. The index is made and the log of the table is not complete after the squeeze (`REINDEX INDEX`), as the changes it applied are heard by no trigger.
- Fixed: pg_sift: a `sift_col` index with `track_changes` of a table that became a partition (`ATTACH PARTITION`) or an inheritance child (`INHERIT`) keeps a log that is not complete after `DETACH PARTITION` or `NO INHERIT`: the event trigger of ALTER TABLE ends the completeness when the table becomes a child, so the statements of the parent that it did not hear no longer make a read after the detach leave rows out. A table attached before this release still needs `REINDEX INDEX` at the detach.

## 4.0.1 - 2026-11-04

- Fixed: pg_sift: a `sift_col` table with a primary key could not be restored by `pg_restore -j` or `pg_upgrade`: `track_changes` is stored as the key's name (`track_changes=id`, which pg_dump writes) and `CREATE INDEX` refused it with `column "id" has no unique index of its own` whenever the index was made before the key, which a parallel restore does for most tables and the restore of `pg_upgrade` did for all of them. `CREATE INDEX` and `REINDEX` now accept the column with a NOTICE (`ALTER INDEX` keeps the error): reads check the heap until the key exists. A statement that finds no unique index to look its rows up by (not made yet, dropped) now ends the completeness of the log; before, a key dropped, rows deleted and the key made again left a log that reads took for complete, and a grouped read counted the deleted rows.
- Fixed: pg_sift: a read took the log of a `sift_col` index with `track_changes` for complete after its table had become a partition (`ATTACH PARTITION`) or an inheritance child (`INHERIT`), where the statements on the parent fire none of the statement triggers of the table, so the deletes and key changes they made were not in the log and a grouped read answered with rows that were gone: a read now leaves the log (and the heap answers) while the table is a child, as it already did for a subscription target. After `DETACH PARTITION` or `NO INHERIT`, `REINDEX INDEX` gives the index a new baseline.
- Fixed: pg_sift: after `pg_repack` of a table `sift.col_tracking` of a `sift_col` index with `track_changes` said its triggers and heap-free reads were off until the next `REINDEX`: the log of the index built on the copy of pg_repack held the identities of the triggers of the copy, which go with it. The index now takes those of the table's own triggers, which pg_repack leaves in place and the swap does not touch (the copy is a full rewrite under a snapshot, and what the writers do meanwhile is applied to it by statements the copy's triggers hear), so the log is complete at the swap and a read of rows changed and not yet vacuumed stays out of the heap.
- Fixed: pg_sift: `REINDEX INDEX CONCURRENTLY` and `REINDEX TABLE CONCURRENTLY` of a `sift_col` index with `track_changes` (the default since 4.0.0) left the two statement triggers with the oid of the index they were made for, which the swap drops, and every UPDATE and DELETE of the table failed with `could not open relation with OID n` (INSERT and reads went on): a trigger now serves the index it depends on, and the swap moves that dependency to the new index. A table that ran it is repaired by `REINDEX INDEX` of the index, or by `ALTER INDEX x SET (track_changes = 'off')` and then `SET (track_changes = 'auto')`; the log of the new index is not complete after a concurrent build, as for `CREATE INDEX CONCURRENTLY` (`sift.col_tracking`).

## 4.0.0 - 2026-11-03

- Improved: sift_table: a COPY, CREATE TABLE AS or REFRESH MATERIALIZED VIEW without parallel workers encodes each unit into one open segment as it fills, instead of buffering eight units of rows as heap records and writing a segment for each run, and the segments of a load are the few large ones a merge used to make afterwards: `bulk_copy_wal` of 10,000,000 rows, the COPY and the `sift.table_maintain` that followed, 24,534 to 13,836 ms and 255,160 to 167,908 M instructions (a heap table 8,050 ms and 106,902 M), WAL 47.7 to 14.2 bytes a row, 28 segments to 2 and the table 196 to 137 MB; of 1,000,000 rows 18,772 to 17,068 M instructions, the time unchanged (1,375 and 1,440 ms, the runs overlap), WAL 19.4 to 16.4 bytes a row. A load with workers is not changed.
- Added: pg_sift: `track_changes` (on by default as `'auto'`, the primary key of the table when the index holds it and the table is one a statement trigger hears, else off with a NOTICE; `WITH (track_changes = 'off')` turns it off; indexes of earlier releases are not changed by `ALTER EXTENSION UPDATE`) of a `sift_col` index on a heap table keeps a log of the entries that stopped being live: it installs two internal statement triggers on the table (AFTER UPDATE and AFTER DELETE, dropped with the index, one NOTICE at `CREATE INDEX`, `REINDEX` and `ALTER INDEX .. SET`) and writes a baseline of the dead versions in a pass over the heap, and a read of an MVCC snapshot takes the log for the whole truth when it is complete: a grouped count or a top-N over rows updated and not yet vacuumed reads 2 heap blocks instead of the pages of every changed row (300,000 rows, the recent 20% updated: the grouped count 21.77 to 10.66 M instructions, 1,241 to 2 heap blocks and 10.09 to 0.41 MB read cold, the top 20 by a key 2.28 to 0.80 M, the count of one tenant 10.09 to 0.41 MB cold; a point read and an index-only scan 2.48 to 2.59 M and 0.36 to 0.37 M). It costs writes on every table it tracks (100,000 rows, instructions of the statement over a plain index: an UPDATE of 5% of the rows 144.6 to 207.7 M, a key change of 1% 66.4 to 86.9 M, a DELETE of 2% 53.6 to 72.3 M, an INSERT +0.6%, `CREATE INDEX` and `REINDEX` +187 M for the baseline), and the gain is gone after a VACUUM and absent for point reads and index-only scans, so the option pays when a table is read many times for each write while rows are unvacuumed ([docs/USAGE.md](docs/USAGE.md#reading-without-the-heap-track_changes)).
- Improved: pg_sift: an UPDATE under `track_changes` holds the tombstones of a statement and appends them to the log by the page, and the statement triggers of `track_changes` look up the keys of 8,192 old rows at a time with one array key: the extra cost of an UPDATE of many rows and of a DELETE over a plain index is far lower; the tombstones of a transaction or savepoint that aborts are never written.

- Added: pg_sift: the developer `unit_rewrite` evaluation method gives each unit an independent segment generation within the directory capacity, then rewrites only a unit whose exact owned tombstones pass its price after VACUUM confirms removal. Old generations retain the snapshot and standby reuse horizons. The method stays off by default; establishing the layout needs CREATE INDEX or REINDEX. Held by `col_unit_generations`.

- Improved: pg_sift: `CREATE INDEX` of a `sift_col` index with an order array on a text column ranks the strings of the dictionary once for the array of the strings and the array of the codes (it ranked them twice) and decodes each string once, not at each comparison: 100,000 rows of `text COLLATE "und-x-icu"` take 1,542 M instructions where they took 2,641 (strings that share a prefix), 1,477 where 2,431 (random hex), 1,261 where 1,943 (words with numbers), 2,149 where 3,776 under a libc locale and 873 where 1,223 under the C collation; the index is the same bytes
- Added: pg_sift: `sift.stats()` counts what the reads of a backend paid for what maintenance would spare them, whether or not maintenance may run: `rows_dirty_units` (rows of units with a dead row read as units), `read_penalty_kinstr` (what the rows checked in the heap and the rows of dirty units cost the reads, in thousands of CPU steps) and `maintenance_asks` (VACUUMs and rewrites the backend asked a worker for). `sift.eval_methods = 'penalty_maintenance'` (a test build) prices the VACUUM request by that sum against the cost of the VACUUM; it is off by default.
- Improved: pg_sift: a bitmap heap scan of a `sift_table` hands its slot the rows of the column groups undecoded, as the index fetch does, and fetches by TID, through the units the statements before kept decoded, the rows of a superblock that selects fewer than 50 of them: it built the whole tuple of each fetched row, decoding the string chunks of every column. A bitmap of 25 rows of 300,000 takes 0.39 M instructions and 6 blocks (4.9 M and 38 before) and of 1,000,000 rows 0.63 M instructions and 6 blocks (18.3 M and 240 before); 4,110 rows of 1,000,000 take 36 M instructions and 36 blocks (95 M and 988 before) and 4,110 rows of 300,000 take 12.8 M instructions (18.5 M before). Of eight bitmaps of 25 to 36,000 rows none is dearer than before (at worst 0.93 of the instructions), and the geometric mean is 0.30 of them at 300,000 rows and 0.14 at 1,000,000. The plan the planner picks for `one_day` and `one_parent_narrow` (30,000 rows: 26 M instructions, the index scan 0.23 M, estimates 2% apart) now costs 0.7 M and 0.25 M.
- Improved: pg_sift: the ordered index-only scan of a `sift_col` index (the first column of the index read alone, in order) is offered to a statement that has a window function or an aggregate as well, where it was offered to a plain `ORDER BY` only, and a merge join no longer puts a Materialize over it: the key in order, a window over the key and a merge join of two tables on the key read the order array of each index in place of a table scan and a Sort. At 1,000,000 rows of a random integer key (warm, whole statements with the planner's own choice, 7 repetitions): the key in order (`SELECT count(*) FROM (SELECT k FROM t ORDER BY k OFFSET 0) s`) takes 51 ms and 785M instructions against 147 ms and 1,741M for the heap plan, a b-tree on (k, id) 42 ms and 708M (run 07c83e7cf116-20261102T212146Z-94df); a window over the key (`SELECT max(r) FROM (SELECT rank() OVER (ORDER BY k) r FROM t) s`) 253 ms and 3,256M against 335 ms and 4,338M, a b-tree 219 ms and 3,180M (run 07c83e7cf116-20261102T212224Z-7161); a merge join of two tables on the key (`SELECT count(*) FROM (SELECT a.k FROM t a JOIN u b ON a.k = b.k ORDER BY a.k OFFSET 0) s`) 151 ms and 2,165M against 504 ms and 4,467M, a b-tree 114 ms and 1,957M (run 07c83e7cf116-20261102T212346Z-ba5a). The b-tree stays ahead on all three
- Changed: pg_sift: a `sift_col` index with an entity column (a foreign key or a column named `*_id`, 10 values or more) and a date or timestamp column gets `key_order = 'auto: entity, ts'` by default where the layout pays: it replaces the second order of the dates that the plain default builds, or the units are cut by the entity, or the heap is sorted by it; a default `key_cut` by a timestamp of distinct values alone gets `key_order = 'auto: ts'`. One decision at `CREATE INDEX` and `REINDEX`, a NOTICE says it; `key_order = 'off'` or any `key_order`, `order_columns`, `order_array_columns` or `order_array_group` of your own leaves the layout as it was. 300,000 rows, rows sorted by the entity (100 entities): the latest row of each of 100 entities reads 4.3 M instructions instead of 72.7 M and `ORDER BY entity, ts OFFSET 10,000` 14.7 M instead of 51.0 M, with an index 40% smaller; 40 tenants of skewed sizes in a random order, the latest 20 of one tenant 0.3 M instead of 0.8 M (a b-tree 0.2 M); a timestamp key_cut alone, `OFFSET 10,000` 6.2 M instead of 51.9 M and the top 20 0.3 M instead of 6.3 M. The plain order arrays of the other columns are not built then: with 10 entities in a random order `ORDER BY entity, ts OFFSET 10,000` takes 30.6 ms instead of 15.6 ms (a b-tree 0.3 ms) and the index is 7% larger, and the join of 100 entities on rows sorted by the entity reads 4.8 M instead of 2.0 M instructions (0.48 against 0.28 ms); 100 to 500 entities in a heap that is not sorted by the entity, 20 and 40 entities in a random order under a date cut, and partitioned tables keep the plain default.
- Improved: pg_sift: `ORDER BY g, k` (both ascending or both descending) and `WHERE g = x ORDER BY k` over a `sift_col` index on `(g, k)` whose first column is `NOT NULL` is an index-only scan that reads the second order of `k` grouped by `g` one group after the other, where a table scan and a Sort were; the build writes that order whole (exact keys, not compact chunks) for the second key column of an index whose group is its first; the index of the 1,000-group table takes 19.9 MB against 31.5 for the b-tree. At 1,000,000 rows of a random key in 1,000 groups (warm, whole statements with the planner's own choice, 7 repetitions; `SELECT count(*) FROM (SELECT g, k FROM ow ORDER BY g, k OFFSET 0) s`) it takes 51 ms, 893M instructions and 698 buffers against 272 ms, 3,811M and 5,407 for the heap plan; a b-tree on (g, k, id) takes 49 ms, 784M and 3,835 (run efd3b66c2e19-20261103T103832Z-5148). In 100,000 groups of 10 rows the scan takes 78 ms and 1,290M against 164 ms and 2,473M for the heap plan, and the b-tree stays ahead at 47 ms and 784M: a group costs the scan about 4,600 instructions to open (run efd3b66c2e19-20261103T103922Z-8707). With 10 groups the default build writes no order grouped by `g` (fewer than 100 values), so the sort stays at 318 ms (run efd3b66c2e19-20261103T103741Z-6d39); with `order_array_group = 'g'` the scan takes 51 ms and 891M against 324 ms and 4,442M, a b-tree 48 ms and 784M (run efd3b66c2e19-20261103T104011Z-9c20). A table of about 4 rows a group or fewer keeps the sort, a nullable `g` or an order grouped by another column keeps it, and a partitioned table gets a Merge Append of the scans. Regression `col_ordered_pair`.
- Improved: pg_sift: the top-N of a scan that reads units one after the other (`WHERE tenant_id = 5 ORDER BY event_date DESC LIMIT 20` on a `key_cut` index, whose units hold the rows of several tenants) sizes the candidate buffers of a unit that starts afresh without copying the ones of the unit before: the buffers grew from 29,119 rows to 58,238 for the second unit and `repalloc` copied 2.6 MB of a unit that had ended, at each execution. A walk that fills its buffers as it goes still keeps what it has. At 300,000 rows (40 skewed tenants, warm, the planner's own choice, run 0e767cbf3944-20261102T175823Z-1481 after and 0e767cbf3944-20261102T180342Z-cb9f before): execution 0.18 ms to 0.11 ms and 1.3 to 1.0 million instructions, a b-tree on (tenant_id, event_date) 0.2 million. The remaining cost is reading the two units (about 58,000 rows for the 12,039 of the tenant), which this does not change. The rows are those of PostgreSQL's plan (2,214 statements over the 41 tenants, both directions, limits 1, 20 and 500, offsets 0, 7 and 3,000 and three filters). Regressions `agg_topn_*` and `col_topn_*`.
- Improved: pg_sift: `order_array_group = 'auto'` weighs each array of the index alone (B = log2(3.5) / log2(1.05) against C = G / 3 + W / 5, the date and timestamp arrays first), so the default index of a table with an entity and a timestamp gets a second order for the timestamp and keeps the integer `counter_a` plain: the latest row of each of 100 entities (`LATERAL .. ORDER BY created_at DESC LIMIT 1`) over 300,000 rows reads 5.0 M instructions and 670 blocks where it read 171.2 M and 31,838, the latest 5 of each 8.3 M against 176.0 M, the top 20 of one entity 0.4 M against 2.0 M; the index takes 16.17 bytes a row against 14.80 (+9.2%; 16.18 against 14.79 at 1,000,000 rows) and stays 49% under the b-tree on the two key columns (31.6 bytes a row); the date key (`created_at::date`) grows 8.96 to 10.65 (+18.9%), its chunks stay whole; a `CREATE INDEX` runs +1.3% instructions at 300,000 rows and +6.8% at 1,000,000. The same compact second order shrinks an index whose default second order was already written on a low-cardinality integer column: a timestamp key with `counter_a` 17.50 to 12.12 bytes a row (-31%), a timestamp-range key -25%, an integer key -11%, a date-range key -6.5%, and `WHERE counter_a = 7 ORDER BY key LIMIT 20` reads 0.4 M instructions where it read 0.3 M (0.6 M against 0.4 M at `OFFSET 100`, 158 blocks against 43)
- Improved: pg_sift: the second order of an index with `order_array_group` is laid out by `order_layout`: a chunk of it that holds the keys of several groups is compact (the heap address of each row and its key rounded to a bucket, the first key of every group but the first in the chunk kept exact) where that is at least a fifth smaller and has no run of 8 equal keys in a group, so the default index of a timestamp and an entity holds much less than with whole chunks of it, and so does any other index whose default second order is written (a timestamp, date or integer key with a column such as a counter of many values as its group); the reads that use the second order take more work than with whole chunks (the heap probes of the exact keys) and keep their wins over an index without it, measured for the timestamp-and-entity key and the date key; `order_layout = 'whole'` and a `REINDEX` write the whole chunks as before and a segment written before is read as before; a segment with a compact chunk of the second order is column format 41 (3.7 and older refuse it)
- Improved: pg_sift: a VACUUM of a `sift_col` index seals an open write buffer that holds a quarter of a unit of rows as well as one that fills a quarter of `write_buffer_limit` in pages: an index on a point column, whose row is a few bytes, kept tens of thousands of rows in the buffer, and every probe of a join read them one by one. Held by `col_wbuf_seal_rows`.
- Improved: pg_sift: an index-only scan of a `sift_col` index asks for the VACUUM of its table (`sift.col_auto_merge`) when the rows it returns lie on pages the visibility map does not hold (a sample of its rows is looked up in the map; the check costs a clock read once per scan and a lookup for one row in 64), where only an insert or an aggregate scan asked: a table that only updates in place (HOT) leaves no dead tuple for the statistics once a read prunes it, and reads alone left the map unset for ever. Held by `col_auto_merge_read`.
- Improved: pg_sift: a backend whose reads of a `sift_col` index find dead rows in units (a unit with a dead row is read as a unit, not served from the point directory or its pre-aggregates) asks for the rewrite of the index when those reads have cost half of it, where the rewrite waited for 30% of the units to hold a dead row and for the WAL budget of the changes: rows deleted from one segment left every read of its keys at the price of the unit until a later change reached the 30%. The reads are counted by the backend (a connection that reads once never asks), and `sift.col_auto_merge = off` or a table that opted out of autovacuum asks for nothing. Held by `col_auto_merge_dirty_reads`.
- Changed: pg_sift: the background pass of a `sift_table` is asked for when the reads of the table since the last pass have cost as much as the pass, where it was asked for when 5% of the rows were tombstones: a table deleted from by less than that, and read, stayed dirty for ever (every unit with a dead row is read from its rows). A read pays for the rows of each superblock that holds a tombstone, a pass about 1,500 instructions a row (a maintenance step counted at half): 38 reads of a table dirty throughout, whatever the share of deleted rows; a table nobody reads is left to autovacuum. The aggregate and sequential scans of a `sift_table` now count in `pg_stat_user_tables.seq_scan`. `sift.table_auto_defrag_dead_pct` has no effect and is kept so a configuration that sets it loads (`tbl_defrag_dead_pct_deprecated`). Held by `017_tbl_autodefrag`.
- Improved: pg_sift: a VACUUM of a `sift_table` writes the pre-aggregates of the units that kept a dead row net of those rows, and the aggregate scan of the table answers such a unit from them (`Units Net Of Dead Rows` in `EXPLAIN ANALYZE`) where it read the unit's rows and tombstones: a `sift_col` index already did. The same answers as the normal plan after the VACUUM and after later deletes (1,000,000 rows, 4.5% deleted, `GROUP BY` a 30-value column with a count and a sum: 153 M instructions before the VACUUM and 1.7 M after it, the heap table 2,731 M and 2,418 M; `table_read_churned`, `table_read_settled`). Held by `tbl_net_sidecar`.
- Changed: pg_sift: a parallel `CREATE INDEX` of a `sift_col` index launches a worker for each further 150,000 rows of the table (at the default `min_parallel_table_scan_size`; the limit goes down with it and a table that sets `parallel_workers` has its participants), so a small table is built by fewer participants, in fewer segments.
- Improved: pg_sift: a scan of a `sift_table` with deleted or updated rows not yet vacuumed reads the tombstones of a page of a column group once for the page, not twice for each row (a grouped count and sum over 300,000 rows with 10% updated, 5% inserted and 5% deleted takes about a quarter of the instructions it took); a page asked for fewer than 16 rows keeps the old path. Held by `tbl_page_tombstones`.
- Fixed: pg_sift: `count(*) WHERE g = x` of the biggest group of a `key_order = 'g, ts'` index (a third of the rows) after updates and a `VACUUM` is answered by the pre-aggregates of the units, net of their dead rows, at 4.0 M instructions; it took the entries of the group's order array, a liveness test each, whenever every unit held a dead row, and read more of them than the units cost. An index whose count is cheaper by entries than by units still counts by them.
- Removed: pg_sift: the NOTICE of `CREATE INDEX` on a `sift_col` index that the primary key of a table referenced by a foreign key is not a column of the index. It claimed that a join of two sift tables on the key keeps the normal plan; in three join shapes (the join grouped by a column of the referenced table, a `cust_id IN (SELECT id ..)` filter and the join from the referenced table, 100,000 orders over 5,000 customers) the plan is the aggregate scan with or without the key in the index and the instructions differ by 3.1% at most (1,549,691 without the key, 1,597,549 with it), while the key adds 19% to 131% to the index. The NOTICE for the columns of a foreign key stays: without one the join reads 181,543,272 instructions and 564 blocks against 17,854,263 and 72.
- Improved: pg_sift: a join that looks up a `sift_col` index with one equality for each row of the outer side (`JOIN t ON t.c = o.k` on a wide index, 1,000 outer keys) probes it with the plain Index Scan, where the planner offered only the index-only scan, whose probe costs 17 to 26 µs against 1.4 to 9.5 µs: the join takes 1.26 ms, 1.2 ms and 1.32 ms at 100,000, 300,000 and 1,000,000 rows (hash join over a sequential scan 7.5 to 15, 23.7 and 86.9 ms; the nested loop with index-only probes 17, 20.8 and 20.6 ms; a b-tree about 1.0 ms), and the index-only scan stays for the whole-index reads; held by `col_join_is_probe`.
- Improved: pg_sift: a `LATERAL` sub-select that looks up a `sift_col` index with an equality on a column of the outer row (`JOIN LATERAL (SELECT n FROM t WHERE t.id = o.id OFFSET 0) x ON true`, 1,000 outer keys, 300,000 rows, `point_columns` and `point_tid_columns` on `id`) was priced as a correlated sub-select, at 0.9 of the sequential scan of the table for each probe, which sent the join to the index-only scan: the probe is priced as the loops of the join and the plain Index Scan runs it, 1.6 ms and 19 M instructions (a b-tree 1.2 ms and 11.6 M; planning 0.1 ms in each); held by `col_join_is_probe`.
- Improved: pg_sift: the planner charges the start of an aggregate scan of a partitioned table 40,000 operators, not 100,000 (the 0.25 ms a scan with lookup tables pays): the top-N of one tenant of a table partitioned by hash into 64 (`WHERE tenant = 7 ORDER BY ts DESC LIMIT 20`, 300,000 rows, 1,000 tenants) runs in the scan, which reads the one partition's top-N in 0.07 ms and 0.5 M instructions, where the plan kept the sequential scan of the partition and the sort; a b-tree on the same rows takes 0.02 ms and 0.2 M instructions.
- Improved: pg_sift: the sum, minimum and maximum of the rows of a group of an integer measure (the statistics a VACUUM, a build and a rewrite compute for each group of each dimension) are taken by an AVX-512 kernel, where the AVX2 form compared and blended for each 64-bit extreme: the second VACUUM of a 13-column index of 300,000 rows (1.5% dead) takes 575 Minstr, not 645 (-11%), with the same bytes; held by `col_kernels_simd`.
- Improved: pg_sift: a `LATERAL` top-N (or a correlated sub-select) over a table partitioned by range reads the top-N of each partition and merges them, where PostgreSQL scanned every partition and sorted for each outer row; `LEFT JOIN LATERAL (SELECT created_at FROM lp_ev WHERE dim_a_id = d.id ORDER BY created_at DESC LIMIT 1)` for 100 entities over 100,000 rows in 12 monthly partitions (`order_array_group = dim_a_id`) takes 0.33 ms, 5.9 million instructions and 160 buffers (4,311 million instructions and 154,304 buffers, 357 ms, as PostgreSQL planned it; a b-tree on the same rows 0.20 to 0.28 ms, 3.6 million instructions, 202 buffers). When the first `ORDER BY` column is the key of the partitioning the partitions come in the order of the sort and the later ones are never executed once the first answers; for another order a Merge Append of the 12 (`ORDER BY v DESC, id LIMIT 1`: 47 ms against 1,776 ms). The query is planned both ways and the cheaper kept: with a b-tree on the partitions that plan stays. Held by `agg_topn_part_arms`; the case `lateral_top1_partitions`.
- Improved: pg_sift: a `LATERAL` top-N (or a correlated sub-select) over a table partitioned by hash, whose key the query pins to a column of the outer row (`WHERE e.tenant = d.id ORDER BY ts DESC LIMIT 1`), is one scan of the table that drops all the partitions but the one that holds the value of each row (the remainder of its hash: `63 partitions removed`) and walks that partition's order array, fetching the rows from its heap, where it was one scan of each partition under a test of the value. 100 entities over 300,000 rows in 64 hash partitions (`order_array_group = dim_a_id`) take 0.85 ms to plan and 1.4 ms to execute, 11.7 million instructions and 1,516 buffers as statements, where PostgreSQL took 63 ms to execute and 441 million instructions (a b-tree on the partitions 0.40 ms to plan, 0.73 ms to execute, 9.6 million instructions and 265 buffers); in 16 hash partitions they take 0.27 ms to plan, 1.1 ms to execute and 8.2 million instructions, where PostgreSQL took 102 ms and 1,038 million (a b-tree 0.16 ms, 0.52 ms and 5.6 million). The walk reads more buffers than the b-tree's probe, 15 a probe against 2.6, fewer instructions than it used to take (30.8 and 24.8 million at 64 and 16 partitions). A partition whose columns have other attribute numbers than the first partition's is read through the units. Held by `agg_topn_part_leaf` and `agg_topn_part_arms`; the case `lateral_top1_hash_partitions`.
- Improved: pg_sift: `count(*)`, `GROUP BY` and `IN` on a `key_columns` leaf answered by the key table of a segment (1,000,000 rows, 3,000 keys, a count-only table) read 23 blocks and 450 k instructions, not 53 and 797 k: the table is decoded with copies of the fields it reads (62% of the statement before), and the dirty bits of the units of a segment are read in one pass over the LIVE page, not one visit a unit.
- Improved: pg_sift: `order_array_group = 'auto'` builds the grouped order array for an integer key too, and for a date the arrays leave out and a timestamp the units are cut by, by one weighted formula (B = log2(3.5) / log2(1.05) against C = G / 3 + W / 5, the real bytes of the second order: 5.6 B a row and the 24 B of each group's entry): the latest row of each of 200 groups (`LATERAL .. ORDER BY created_at DESC LIMIT 1`, 5 columns, an entity of 3,000) takes 0.98 ms / 11.8 M instructions where it took 28.1 ms / 379 M at 300,000 rows, and 1.06 ms where it took 4.70 at 1,000,000 (a b-tree on (entity, created_at) 0.57 and 0.68 ms); `WHERE grp = 17 ORDER BY k LIMIT 20` 2.951 to 0.043 ms at 1,000,000 rows (a b-tree 0.014); the index grows +86% at 300,000 rows (4.55 against 9.49 MB of b-trees) and +72% at 1,000,000 (15.07 against 31.56 MB).
- Improved: pg_sift: `WHERE tenant_id = x ORDER BY event_date DESC LIMIT 20` reads the units a filter leaves when the walk of the order array would fetch more rows than they hold: 300,000 rows, 40 tenants, `key_cut` by the tenant and the group order on, 0.107 to 0.071 ms and 154 to 42 blocks (without the group order 0.070 ms).
- Improved: pg_sift: `count(*) WHERE entity = x` over an index with the grouped order array counts the group from the entries of its second order, and the group-only array keeps no plain order beside it: 1,000,000 rows, 5 columns, an entity of 3,000, 0.265 to 0.031 ms, 3.84 to 0.34 M instructions and 359 to 67 blocks (a b-tree 0.021 ms, 0.22 M, 4 blocks), 300,000 rows 0.209 to 0.028 ms; the index without the plain order beside the group order is 14.6% smaller at 300,000 rows and 1.5% at 1,000,000 with `key_cut`.
- Improved: pg_sift: `count(*) WHERE g = x AND k BETWEEN a AND b` (the events of one device or tenant in a week) over a `sift_col` index whose order array of `k` is grouped by `g` (`order_array_group`) counts the entries of the group between the two keys, where it read the units that hold the group, and the sums, minima and rows of the same condition read only the units that hold those entries, narrowed to their rows: 300,000 rows 0.57 to 0.40 M instructions for the count, 0.86 to 0.61 for `count(*), sum(x)` (1,000,000 rows, b-tree 0.19), no bytes, no option. A condition on `g` with a range, a segment with dead rows (the units that are not exact are read after the exact ones are counted) and an index with no grouped array read the units as before.
- Improved: pg_sift: an index-only scan of a `sift_col` index whose returned columns are integers, booleans, dates, timestamps or floats writes the values of each row into one index tuple it keeps, where it formed a tuple for every row (`index_form_tuple`, about 40% of the cycles of the scan) and kept the values of a unit in two arrays: a whole-index scan of 1,000,000 rows takes 781 Minstr and 76 ms for `count(*), sum(k)` over it (1,209 Minstr and 99 ms before; the b-tree 665 Minstr and 47 to 59 ms) and 1,190 Minstr for all its rows (1,619 before; b-tree 1,075), 100,000 rows 78 and 119 Minstr (121 and 162 before). A row with a NULL in a returned column, a string column or a numeric one keeps the old path. The same rows as the heap's (`col_ioss_fixed`).
- Improved: pg_sift: a page under a condition on another column (`WHERE deleted_at IS NULL ORDER BY k, id LIMIT 20 OFFSET 10000`, `ORDER BY d DESC, id`) over the order array tests the condition of the entries before the OFFSET on the values of the units, in batches of 2,048 to 16,384 entries (`IS NULL`, `IS NOT NULL`, integer and date comparisons, `AND`, `OR`), and fetches from the heap only the entries the units cannot say (dirty pages, the write buffer). 1,000,000 rows, churned: heap fetches 10,245 become 40, buffer accesses 10,558 become 1,418 (`k`) and 9,796 become 1,563 (`d`); cold, 47.2 ms and 10,489 reads become 16.7 ms and 1,002 (`k`; the b-tree 44 to 58 ms, 7,900 to 9,600 reads), 36.7 ms and 6,335 reads become 20.5 ms and 1,109 (`d`); 300,000 rows: 10,641 accesses become 1,045 and cold 20.2 ms 16.0 ms (`k`). Instructions: 26.9 M to 26.7 M (`k`), 29.8 M to 39.5 M (`d`, the batches read the date column of the units).
- Improved: pg_sift: the overflow guard of `sum()` and `avg()` of a `float8` column reads the selected rows through the vector kernel of the largest absolute value (the dense path already did), where a loop tested each row's byte mask: a scan of 1,000,000 rows with `count(*)`, `sum(numeric)` and `avg(float8)` takes 67.9 Minstr (76.0 before), a grouped scan with five aggregates 124.9 Minstr (132.3). The same answers, and the same error, as the normal plan (`agg_float_guard`).
- Improved: pg_sift: the strings of a text order array whose dictionary is in compressed blocks (a block of 64 strings to decompress for each string read: 14,600 instructions) keep the last 32 blocks read, where each read of a bisection or of the walk decompressed its block again (110 of the 261 blocks a text cut of `OFFSET 10000` read were blocks read a few reads before). 1,000,000 rows of random 10-character strings, churned, `ORDER BY t, id LIMIT 20 OFFSET 10000`: 11.6 M instructions and 968 buffer accesses become 10.0 M and 730 (the b-tree: 6.6 M, 42); on a backend that has not read the dictionary before, 1,067 become 829 buffer accesses. A dictionary in LZ4 blocks would decode in 1,000 instructions against 15,900 for these strings at 77% more bytes: not changed.
- Improved: pg_sift: `count(DISTINCT x)` of an integer or date column with no `GROUP BY` sets the bits of its one bitmap with the directory of the group read once for the chunk, not for each value: `SELECT count(DISTINCT dim_a_id) FROM events WHERE counter_a > 30` over 1,000,000 rows takes 5.1 ms and 42.9 Minstr (6.3 ms and 50.5 Minstr before; the loop is bound by the loads of the bit's word, 24 of the 43 Minstr). The same counts as the normal plan (`agg_distinct_one_group`).
- Improved: pg_sift: `min()` and `max()` of an integer, date, timestamp or float column with no `GROUP BY` take the extreme of a chunk of rows from the vector kernel (AVX2, AVX-512, NEON, scalar) where a loop went through the group ids of each row: `SELECT min(counter_a) FROM events WHERE value_num > 1` over 1,000,000 rows takes 2.4 ms and 20.7 Minstr (8.6 ms and 50.3 Minstr before), `max(event_date)` 3.8 ms and 19.1 Minstr (8.3 ms and 47.7 Minstr). A float whose extreme is a zero stays on the loop, since the kernel returns +0 for either sign. The same rows as with `sift.simd = scalar` and as the normal plan (`agg_minmax_one_group`).
- Improved: pg_sift: `point_tid_columns` by default also for an integer key of nearly unique values that the order array already answers, and for one that follows the order of the table (an id from a sequence without a primary key), when the directory's gain pays against the growth of the index counted with the b-tree it spares. Random heap, an index of `id, v, w` over 1,000,000 rows grows 11.4 to 13.4 MB (three b-trees 36.9 MB) and a count and sum over 1 / 10 / 60 / 500 keys of `id` takes 0.020 / 0.029 / 0.101 / 0.50 ms where it took 0.025 / 0.050 / 0.205 / 1.08 (b-tree 0.010 / 0.033 / 0.070 / 0.72); an index of 8 columns grows +16.8%, builds +4% and writes 100,000 rows +8% longer. Heap in the order of `id`, 1,000,000 rows: 2.65 to 5.68 MB (a b-tree on `id` 21.4 MB), 1 / 10 / 60 / 500 keys 0.216 / 0.173 / 0.721 / 1.665 ms to 0.021 / 0.033 / 0.058 / 0.591 (b-tree 0.013 / 0.029 / 0.120 / 0.865); a table of fewer than about 42,000 rows keeps no directory (a unit costs less than 500,000 instructions over it).
- Improved: pg_sift: `ORDER BY k LIMIT 20 OFFSET 10000` over a table stored in the order of `k` (no order array on `k`, 100,000 to 1,000,000 rows, 12 columns) reads the 10,020 rows from the end of the one unit its key follows, and the heap takes the rows as the run they already are: it is not built, filled one by one or sorted at the end (a presorted run is the heap reversed). 3.0 to 3.7 ms and 28.9 Minstr before; 0.9 to 1.3 ms and 6.0 Minstr after (the b-tree on the same table 0.6 to 0.7 ms, 7.0 Minstr; 22 blocks cold against 222), the same 20 rows as before, in every state of the table after writes and a VACUUM (`col_topn_unit_ends`). A unit is read from its end when the rows that can win are a quarter more than the rows asked for, not twice as many.
- Added: pg_sift: `key_order = 'ts'` (or `'tenant, ts'`) on a `sift_col` index, off by default, stands for `key_cut`, `order_columns`, `order_array_columns` and `point_tid_columns` on that column (for `'tenant, ts'`: `order_columns` and `order_array_columns` on the key and `order_array_group` on the tenant, `key_cut` left to the build) while you have not set them: on a table sorted by the key `ORDER BY ts LIMIT 20 OFFSET 10000` takes 5.5M instead of 22.2M instructions (0.56 against 1.77 ms) and 500 points 7.9M instead of 54M (0.85 against 18.3 ms), over 100,000 rows of 11 columns; on a table in arrival order a range of the key takes 0.53M instead of 2.09M and the ordered shapes what the default takes, where `key_cut` alone takes 2x (top-20) to 6.7x (OFFSET 10,000) more; with a tenant the latest row of each of 100 tenants takes 4.1M instead of 13.9M instructions (300,000 rows, 1,000 tenants), the index is 21% smaller and the build 52% cheaper than the default's; 12 rounds of 1% inserts, updates and deletes cost 5,711M instructions against 6,957M by default (-18%; -17% for tail keys, -41% for monotone keys) and, against `key_cut` alone, 10 to 24% more WAL and an index 18 to 57% larger for the order arrays; the index of a sorted table is 30% larger than the default's
- Improved: pg_sift: an aggregate over an equality or an IN list of up to 512 constants on a column the index looks up by itself (`point_tid_columns` or an order array) keeps the index and bitmap scans of the `sift_col` index beside the aggregate scan, and the costs choose (a group by, a window, a sort or any other condition keep the paths they had): `SELECT count(*), sum(v) FROM t WHERE id IN (60 keys)` over 1,000,000 rows with `point_tid_columns = 'id'` runs in 0.09 ms, 2.0 M instructions and 202 blocks where the aggregate scan took 0.25 ms, 2.9 M and 272 (a b-tree on `id`: 0.11 ms, 1.9 M, 240); 10 keys 0.03 ms against 0.10 (b-tree 0.03), 500 keys 0.72 ms against 1.35 (b-tree 1.13); the times are the execution of EXPLAIN ANALYZE with warm buffers, the instructions those of the statement with its planning, and planning adds 0.11 to 0.16 ms, so a client that times the statement reads more: `pgbench` (one client, 60 keys) 0.29 ms against 0.23 on the b-tree, `bench/bench cases` (psql, one CPU) 0.43 ms against 0.57.
- Added: pg_sift: `docs/MESSAGES.md` lists every NOTICE, WARNING and `VACUUM VERBOSE` INFO line of the library in one table, grouped by operation: the exact text, where and when it appears, the decision pg_sift took, its benefit and trade-off with the numbers of the option cases, and how to switch it off, with a table of the decisions at a glance. The table is generated from `docs/messages.tsv`, and the lint check `scripts/gen-messages.sh --check` fails when a message of `src/` or `sql/` has no row, a row has no message, a text, DETAIL or HINT differs, or a number is in no cited measurement.
- Improved: pg_sift: an aggregate over an equality or an IN list of up to 512 constants on a column the index looks up by itself (`point_tid_columns` or an order array) keeps the index and bitmap scans of the `sift_col` index beside the aggregate scan, and the costs choose (a group by, a window, a sort or any other condition keep the paths they had): `SELECT count(*), sum(v) FROM t WHERE id IN (60 keys)` over 1,000,000 rows with `point_tid_columns = 'id'` takes 0.10 ms where the aggregate scan took 0.51 (10 keys 0.03 against 0.28, 500 keys 0.48 against 1.23; a b-tree 0.07, 0.03 and 0.58 ms).
- Improved: pg_sift: the group of ties that holds the first row to keep of an `ORDER BY date DESC, id LIMIT 20 OFFSET 10000` page has the keys after the first read from the units by heap address (`sift_col_gorder_values`, integer, date and timestamp keys of the entries a scan counts as exact), not fetched from 2,780 heap rows: 1,000,000 rows of 365 dates, the first run after the buffers are evicted takes 5.6 ms and 240 reads where it took 19.4 ms and 2,471 (the b-tree: 3.2 ms, 35); 300,000 rows 3.1 ms and 106 reads where it took 8.5 ms and 1,290. Warm, 14.5 M instructions become 20.2 M (the b-tree 19.4 M) and the time is the same. Delta-coded chunks read the rows of one block of 1,024 with one decode.
- Improved: pg_sift: the rows of the group of ties that holds the first row to keep, which the `OFFSET` throws away, are not fetched from the heap a second time to go out (`ORDER BY d DESC, id LIMIT 20 OFFSET 10000` over 1,000,000 rows of 365 dates: 4,453 heap fetches become 2,780 and 16.3 M instructions 14.5 M).
- Improved: pg_sift: `ORDER BY text_key, id LIMIT 20 OFFSET 10000` over the sorted array of a text column (1,000,000 rows, churned): the OFFSET is cut off the front of the sources in one step before the walk of the tie groups, as for one key, the cut of a text key finds its string in three rounds of one candidate per source (it bisected every source before), and after a churn it is taken once more for the entries that dead rows and old versions took (the walk that follows read a string, a block of the dictionary to decompress, for each of them). 1,000,000 rows of random 10-character strings, the default write buffer, an `UPDATE` of 7%, a `DELETE` of 1% and a `VACUUM`: 20.0 M instructions, 196 buffer accesses and 6.9 ms become 11.6 M, 968 and 2.0 ms (the b-tree: 6.6 M, 42, 1.0 ms); strings that run in sequence (`t00000001`, ...) 17.6 M become 7.6 M instructions and 1,620 become 948 buffer accesses (the b-tree: 6.6 M, 42); the cut takes `ORDER BY k, id` with an integer `k` from 9.7 M to 7.0 M (b-tree 6.5 M).
- Improved: pg_sift: the `OFFSET` cut of a text key continues through the entries whose heap page is not all-visible (the string of the heap row is compared with the string of the entry, and the entry counts if the snapshot sees the row), where it stopped at the first unit that could not say whether an entry is live and fell back to the strings of the heap or to reading the units, which after churn a quarter of the units are: `ORDER BY t, id LIMIT 20 OFFSET 10000` over 1,000,000 rows after an `UPDATE` of 7%, a `DELETE` of 1% and a `VACUUM` takes 3.9 ms and 27.0 million instructions (sift off 116.9 ms and 1,262 million; a b-tree 0.65 ms and 6.6 million).
- Improved: pg_sift: a unit with dead rows asks the heap for the pages it checks through a read stream, from its eighth page on (combined reads ahead of the check, where each page was read as it came), and a unit cut by a key checks every row of a page at its first visit (the other units of the scan meet the page again and take the cached mask, where the page was read a second time from disk once the scan had read more than the buffers hold). A grouped sum over 1,000,000 rows with 9% dead and no `VACUUM`, cold, 64 MB of buffers: 25,957 read requests and 209.8 MB where it made 1,446 and 110.5 MB (the sequential scan: 905 requests, 100.3 MB), 890 million instructions where it took 801, 75,060 heap page visits where it made 38,145; 300,000 rows: 4,035 requests where it made 611 (the scan: 350), 186 million instructions where it took 187.
- Fixed: pg_sift: the price of a lookup join whose condition is no key (`k IN (SELECT n + 1 FROM keys)`: the addition can raise, so `n + 1` is a condition of the rows and not a hash key) counted its test for the rows the join lets through (500), not for the tuples the scan keeps, which it tests against every row of the joined table (one test per distinct value of the column, 1,000,000 of them): the scan was chosen against a hash join that runs in 63 ms and took 9.0 s at 1,000,000 rows (2.6 s at 300,000 with the plan forced). The tests are counted for the tuples the scan keeps, so the hash join is the plan (7.9, 21 and 63 ms at 100,000, 300,000 and 1,000,000 rows); a column or a cast that cannot raise as the key is the lookup it was (0.5, 0.7 and 1.9 ms).
- Improved: pg_sift: the lists of dead rows that an `OFFSET` over the sorted array asks for (the rows `VACUUM` cleared from a segment, by unit) are kept by the backend for its next scan (up to 16 MB, 8 bytes a dead row: 642 KB for 78,000 of 1,000,000), and a dead entry of a group the offset throws away is no longer fetched from the heap unless another source repeats its address. `ORDER BY k, id LIMIT 20 OFFSET 10000` over 1,000,000 rows after churn and autovacuum (7.8% of the rows dead): 15.3 million instructions and 1,142 blocks became 9.7 million and 241 (the same table after a rewrite 8.4 and 190, the b-tree 6.5 and 31); the text key 23.1 million and 2,537 blocks became 17.6 million and 1,622; the date key 23.6 and 4,874 became 18.1 and 4,001 (the b-tree 19.4 and 35). A list is used while the segment has cleared no more rows than when it was made.
- Improved: pg_sift: a `LATERAL` top-1 or correlated sub-select that reads the global order of a `sift_col` index (`ORDER BY key LIMIT n` with the group or a bound taken from the outer row) keeps its source of the index open from one outer row to the next, where each probe opened the segments, the visibility state and the layout again: a probe that read nothing but the filters hands the source on, any other scan opens it again as before. 200 probes over 300,000 rows 12.3 M to 8.0 M instructions and 1.66 to 1.09 ms, over 1,000,000 rows 14.6 M to 9.0 M and 1.99 to 1.08 ms (a b-tree 1.0 to 1.2 ms); the rows are those of a scan without the engine (a matrix of 18 shapes through a write buffer, deleted rows, a promotion, a cursor, a prepared statement and workers, and 3,000 random statements), and the memory of the source does not grow with the probes.
- Improved: pg_sift: a `LATERAL` top-1 that keeps its source of the index open from one outer row to the next reads the group table of a segment (the list of the groups of a grouped order array, 24 bytes a group, up to 1 MB) once for the backend from its second probe, where every probe searched it on the pages (12.6 reads of the segment for each probe): `LEFT JOIN LATERAL (SELECT created_at FROM events e WHERE e.dim_a_id = d.id ORDER BY created_at DESC LIMIT 1) x ON true` for 200 entities over 300,000 rows takes 6.94 million instructions and 743 blocks where it took 8.30 million and 1,572 (34,200 instructions a probe become 27,300 by callgrind; a b-tree 5.46 million and 604)
- Improved: pg_sift: a `LATERAL` top-1 that keeps its source of the index open from one outer row to the next also keeps what the global order learned of the units of a segment (the rows `VACUUM` cleared from them and which heap pages are all-visible), where every probe asked the segment for its dirty units again to know whether an entry may go out without a visit to the heap (3,300 instructions and two page reads of each probe): `LEFT JOIN LATERAL (SELECT created_at FROM events e WHERE e.dim_a_id = d.id ORDER BY created_at DESC LIMIT 1) x ON true` for 200 entities over 300,000 rows takes 6.27 million instructions and 345 blocks where it took 6.94 million and 743 (a probe pins 1.6 buffers where it pinned 3.5; the b-tree 5.46 million and 604)
- Improved: pg_sift: a `LATERAL` top-1 that keeps its source of the index open from one outer row to the next, and every `ORDER BY key LIMIT n` that the walk of the global order answers, no longer builds what only the units read (the combination states, the zone tests of the conditions, the vectors of a unit, the group table) and makes the state of the expression of an operand once for the scan, where the three places that evaluate `e.dim_a_id = d.id` each made their own (4,900 instructions of each probe): `LEFT JOIN LATERAL (SELECT created_at FROM events e WHERE e.dim_a_id = d.id ORDER BY created_at DESC LIMIT 1) x ON true` for 200 entities over 300,000 rows takes 5.26 million instructions where it took 6.27 million (24,400 instructions a probe become 19,500; the b-tree 5.46 million)
- Improved: pg_sift: a `LATERAL` top-N or correlated sub-select over a table whose write buffer holds rows puts the rows of the write buffer in groups once for the statement (and keeps them for the next while the buffer is the same), where each entity read all of them to find its own: a top-1 for 100 entities with 5,000 rows in the buffer takes 1.0 ms and 4.4 Minstr, was 3.3 ms and 16.6 Minstr (300,000 rows; with an empty buffer 1.0 ms and 4.1 Minstr, the b-tree twin 0.8 ms and 4.2 Minstr); the top-20 of 100 entities 2.9 ms, was 7.1.
- Improved: pg_sift: `DISTINCT ON` (and `row_number() = 1`) over a column of a by-value type, for every group of a table, is chosen among the rows of a chunk before the values of the row are copied, and a better row overwrites the one a group holds when its values are by-value: the latest row of each of 3,000 entities over 300,000 rows takes 12.8 ms and 88 Minstr, was 33 ms and 320 Minstr (the b-tree twin 241 ms and 1,328 Minstr).
- Improved: pg_sift: an index scan of a `sift_col` index that starts over for each row of an outer query (the inner side of a nested loop on an equality, `JOIN t e ON e.k = p.k`) keeps the segments it opened for the last key while the snapshot and the segments stay the same, where every key read the directory of the index and the header of each segment again (6,800 instructions and three page reads of each probe): 1,000 keys against 100,000 rows of a unique integer key take 14.8 million instructions, 5,028 blocks and 2.6 ms through an Index Scan, where they took 24.8 million, 8,039 and 6.3 ms (an Index Only Scan 69.0 million and 9,818 blocks where it took 82.2 million and 15,832; the b-tree 10.1 million and 3,008)
- Improved: pg_sift: `order_array_columns = 'auto'` weighs a date or timestamp column of fewer than 1,000 distinct values by the default formula (benefit log2(S) / log2(1.05) against a third of the share of the index it adds and a fifth of the share of the build it adds, the NOTICE says both) instead of leaving it out: `ORDER BY d DESC, id LIMIT 20 OFFSET 10000` over 300,000 rows of 365 dates took 14.3 ms, 109 million instructions (the b-tree on (d, id): 2.4 ms, 20.2 million) and takes 2.0 ms, 10.6 million, 854 blocks (S 10.7; 100 values 7.6, 30 values 4.1, 3 values 0.96, so a column of 3 values is not chosen), for about 3 bytes a row (+12% of the index) and 11% more build of 7 columns. The group the offset ends in is the largest of the NULL rows and one value's, so a column that is mostly NULL is not chosen.
- Improved: pg_sift: the rewrite of a segment sorts its rows by heap address in a dense array of (address, row) pairs, reads the range of an integer column from the zone map instead of scanning its rows again, and compares the second key without a lookup (a forced rewrite of a 5-column index cut by two keys, 3 runs each: 300,000 rows 1,735 million instructions where it took 1,569, 1,000,000 rows 6,363 where it took 5,691, 3,000,000 rows 17,918 where it took 15,794; 10, 11 and 12% fewer).
- Improved: pg_sift: a row written to the write buffer of a `sift_col` index (`INSERT`, `COPY`, `UPDATE`) is encoded and queued without a call of the StringInfo functions for each value: `COPY` of 100,000 rows of 13 columns into a table of 1,000,000 rows takes 16,538 instructions a row where it took 17,325 (no index 11,176; a b-tree on the same columns 26,505), so the index adds 5.4k instead of 6.2k.
- Improved: pg_sift: VACUUM counts the dead postings of each term of a segment again only when the documents cleared since the last count are 5% of the segment's documents or more, and below that the table it left stands and the document frequencies it gives are scaled to the live documents: after one round of 1% of the rows inserted, updated and deleted (2% dead, so the count is skipped; `search_vacuum_dead_postings`, every process of the VACUUM counted) VACUUM takes 88 M instructions, 37 ms and 10.4 MB of WAL at 100,000 documents and 637 M, 179 ms and 34.2 MB at 300,000, against a GIN index of a tsvector on the same rows 1,293 M, 173 ms and 39.9 MB, and 2,850 M, 389 ms and 99.5 MB. Averaged over 12 such rounds (the dead share grows to 24%, the count is repeated every third round and the large segment is rewritten once, in round 10 at 300,000 documents and round 12 at 1,000,000; WAL with `full_page_writes = off`) it takes 803 M instructions, 92 CPU ms and 3.7 MB at 100,000 documents (1,058 M and 3.8 MB before; GIN 1,474 M, 177 ms and 19.5 MB), 2,888 M, 356 ms and 12.8 MB at 300,000 (3,576 M and 12.7 MB before; GIN 3,005 M, 409 ms and 43.1 MB) and 10,875 M, 1,246 ms and 44.4 MB at 1,000,000 (13,059 M and 44.4 MB before; GIN 9,655 M, 1,380 ms and 109.1 MB): over the 12 rounds the GIN index takes fewer instructions at 1,000,000 documents (0.89 times), sift less WAL and time. A round that skips the count takes 608 M instructions against 1,638 M at 300,000. Scores differ from those of a fresh `REINDEX` by 0.1 to 0.2% at 1 to 3% dead (0.3 to 0.5% at 10%) and documents tied near the cut may reorder; a merge and a `REINDEX` are exact, and counts stay exact (`dead_stale`)
- Fixed: pg_sift: an insert that finds the directory of a BM25 index nearly full and the smallest segments claimed by the maintenance worker's merge waited for that merge (20 s in `maint_lock`, with the worker in a 20 s merge) instead of merging the other segments itself: each pick that met claimed segments used one of the eight tries of the step, and a worker holding eight of the segments the policy offers first used them all; a pick that learns a claimed segment is no longer a try, so the step goes on to the segments nobody holds (`maint_lock`: the insert took 20 s and waited 19.6 s, now 0 s)
- Improved: pg_sift: the VACUUM rewrites a lone segment whose dead rows cost more in the next 100 scans than the rewrite costs (the rule of an index with `dead_rewrite_pct` 0), with prices measured on the index and not guessed: a rewrite takes 800 instructions and 230 for each byte of a row (it took 900 and 463, which stopped the rewrite of an index with order arrays, 15 bytes a row, at a fifth of its rows dead), a scan 700 for each dead row of a unit or 58 for each of its rows when more than a tenth is dead; with the sidecar off a 1,000,000-row index with a tenth or a fifth of its rows dead is rewritten by the first VACUUM (100,000 and 200,000 dead rows left before, none now).
- Improved: pg_sift: the dead rows of a unit are turned into heap addresses without decoding the offsets of every row of its pages: a page that holds all its offsets gives the address of a row by arithmetic and a page with gaps selects the set bit of its bitmap, where each page was expanded to all its offsets. An `OFFSET` 10,000 over 1,000,000 rows after churn and `VACUUM` (`ORDER BY k, id LIMIT 20 OFFSET 10000`, 33 units, 78,000 dead rows) takes 16.6 million instructions where it took 27.2 (the b-tree: 7.8); the text key 24.4 where it took 35.0.
- Improved: pg_sift: the sorted array of a key written in the compact layout (`order_layout`, 41) is read after deletes and updates: a segment with rows that `VACUUM` cleared was refused the compact blocks and the scan read its units (`OFFSET` 30,000 `LIMIT` 10 over 300,000 rows after a `DELETE` of 3%, an `UPDATE` of 3% and a `VACUUM`: 33.4 ms, 306.8 million instructions, 1,945 blocks, where the whole layout took 3.8 ms); the addresses of the rows the segment lost are listed once, an entry at one is a dead row without asking the heap (the new tenant of the address is never read), and the same query takes 4.4 ms, 29.8 million instructions and 2,940 blocks (the b-tree: 5.2 ms, 42.6 million, 30,080).
- Fixed: pg_sift: wrong rows in 3.4.0 through 3.7.1: a `LATERAL` join or a correlated sub-select whose inner query reads the global order of a `sift_col` index (`order_array_columns` on the `ORDER BY` column, with or without `order_array_group`) took the group it pins (`WHERE e.g = d.id`) and the bounds of the key (`AND e.ts > d.t`) from the outer query as 0 instead of the value of each outer row, so the inner query found no row: `SELECT .. FROM d, LATERAL (SELECT ts FROM e WHERE e.g = d.id ORDER BY ts DESC LIMIT 1) x` returned no row for the outer rows (a `LEFT JOIN LATERAL` NULLs, a scalar sub-select NULL), with no error. The same statement with `sift.enable_agg = off` shows the right rows. The values of each probe are used.
- Improved: pg_sift: a `count(*)` over a range of a column with an order array (`order_array_columns`), on a table with rows inserted since the last VACUUM, counts the entries of the range for a segment however many there are (a chunk at a time, the memory of a chunk and of the addresses on pages that are not all-visible), where a range of more than 16,384 entries in a segment made the scan read the whole segment: `count(*) WHERE txt >= 'a' AND txt < 'b'` over 1,000,000 rows of random strings with 5,000 inserted since the VACUUM 18.3 ms and 292 M instructions to 0.93 ms and 9.9 M (3,000,000 rows 0.49 ms; a b-tree 3.98 and 11.0 ms), the counts unchanged.
- Fixed: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index with `key_cut` (the default `key_cut` is chosen from the data) took longer with workers than alone: the segments of the participants overlap, and the merge that follows rewrites every row, in slices of the key range that each read every input again. A `key_cut` index is built by one process; a build without `key_cut` and `CREATE INDEX CONCURRENTLY` are shared as before. 13 columns (`col_build_cube_13cols`), 64 MB of `maintenance_work_mem`, 8 cores for the run, wall ms with 0 / 4 / 8 workers before: 1,000,000 rows 3,223 / 3,769 / 3,998 (CPU of the backend alone 3,201 / 3,736 / 3,962; WAL 35 / 77 / 77 MB; the file 33.7 / 35.7 / 35.7 MB); 4,000,000 rows on one core 13,718 / 28,474 / 29,075. The merge outside a build keeps its slices (`sift.col_merge`, the rewrite): see the line on the rewrite by workers.
- Improved: pg_sift: a scan that tests the small-set summaries of the units of a leaf at every call (a `LATERAL` top-1 for each row of the outer query, a point probe in a loop) reads and checks the summaries of a segment once for the backend (up to 16 leaves, 16 MB), tests each range of one probe once for a unit, and does not read the summary of a unit whose smallest or largest key lies in the range, where every call read, checked and tested the summary of every unit again.
- Improved: pg_sift: VACUUM counts the dead postings of each term of a segment (the table that keeps the document frequencies of the live documents exact) by reading the postings of a LIST term one by one against a flat table of the dead rows of each superblock, where it built the group of every superblock of every term and intersected them: a VACUUM of a 300,000-document index with 3,000 rows inserted, 3,000 updated and 3,000 deleted takes 1,637 M instructions and 158 ms where it took 2,243 M and 218 ms (same WAL, 3.17 MB); the scores after VACUUM equal those of a fresh index (`dead_counts`). The count still reads every posting of the segment: 1,000,000 documents, 12 rounds of 1% inserted, updated and deleted, VACUUM 13,059 M instructions, 1,325 CPU ms and 44.4 MB of WAL on average (every process of it counted, `full_page_writes = off`) against a GIN index of a tsvector on the same rows 9,655 M, 1,380 ms and 109.1 MB
- Improved: pg_sift: `ORDER BY sift.score(ctid) DESC, key LIMIT k` (an Incremental Sort over the ranked scan) is served by one search of the index that keeps the `k` best hits and every hit that ties with the `k`-th, where the scan asked for 16 times the limit and one row (`Limit Hint` 336 for `LIMIT 20`) and read 16 times as many again when a tie group ran past them: the finished plan tells the scan (`Includes Ties: true` in EXPLAIN, `Limit Hint` the limit), the candidates that tie with a worst score that may still rise wait, unverified, for the final worst score, and the participants of a parallel search publish their ties for the merge; a scan that is not under such a sort, or whose quals drop hits, searches as before. M instr of the sift lane, tip -> change at 100,000 / 300,000 / 1,000,000 documents: `search_top_20_ties` 11.2 -> 7.0 / 31.8 -> 20.1 / 37.6 -> 37.0 (GIN with `ts_rank_cd`: 6.0 / 17.6 / 58.8), top_20 10.0 -> 6.8 / 21.6 -> 16.4 / 61.4 -> 51.8, filter_include 15.6 -> 6.9 / 29.3 -> 17.4 / 65.1 -> 51.0, top_20_or_three 12.4 -> 9.0 / 31.0 -> 18.8 / 77.3 -> 43.0, top_20_and 4.8 -> 4.9 / 13.7 -> 13.5 / 42.0 -> 12.5, filter_recent 2.4 -> 1.3 / 4.5 -> 2.1 / 9.4 -> 4.9, join_region 13.1 / 33.3 / 63.8 -> 13.1 / 33.3 / 63.9 and list_terms 0.6 / 1.2 / 3.3 -> 0.6 / 1.3 / 3.4 (the tie arrays); tests `rank_ties_group`, `rank_ties_parallel` and `rank_ties_sort`
- Fixed: pg_sift: the stem memo of a language read the header of an entry through a pointer that is not aligned for its two-byte fields (a load that UBSan reports in `scripts/fuzz.sh analyze`, harmless on x86-64 and AArch64, undefined in C): the headers are copied out and the bytes are read in place.
- Improved: pg_sift: a `count(*)` whose only conditions bound one text column (a lower and an upper comparison, or an equality) is counted for the segments of a `sift_col` index with an order array of the column (`order_array_columns`) and the collation of the comparison from the ranks of the two bounds among its strings, where every unit was read (no zone prunes a range of random strings); the units that are not exact are read, as for an integer column.
- Fixed: pg_sift: the batches of a write buffer that an index allows to grow big (`write_buffer_limit` raised) are kept for the next scan of the backend whatever their size, within six times the bytes of the records they were made from, where those over 16 MB were decoded again by every scan and the order kept with them could not be found: `SELECT count(*) .. WHERE k BETWEEN ..` over 1,000,000 rows (13 columns, 64 MB of write buffer, 200,000 buffered rows, no VACUUM) takes 68.7 M instructions and 19.0 ms where it took 840 M and 95.5 ms, and its batches (45 MB) stay in the backend until the buffer is promoted or the 4 sets are used; the memory of a buffer at the default limit (4 MB, 10 to 17 MB of batches) is as before.
- Improved: pg_sift: the merge of the segments of a `sift_col` index with `key_cut` outside a build (`sift.col_merge`, and the rewrite that `VACUUM`, the maintenance worker and `sift.col_maintain` start, which is the same step) is made, from 12 MB of segments (about 600,000 rows of a 7-column table, the size from which the gain in time outweighs the extra bytes and blocks), by the backend and 2 of the `max_parallel_maintenance_workers` workers, a slice of the key range each, and leaves 3 segments disjoint in the key, where one process wrote one segment; a rewrite no longer takes segments that no unit is in two of again. A narrower input, or `max_parallel_maintenance_workers = 0`, is merged by one process as before. `sift.col_merge(index, true)` of 1,000,000 rows (20 MB of segments) takes 623 ms with 2 workers where one process took 965 ms (4 workers are slower than 2 at every size); 3,000,000 rows (62 MB) 1,219 ms where it took 2,985 ms; the index is 3.1% and 2.7% longer, the WAL 10.4 to 11.0 MB (1M), the instructions +4.2% and the blocks +7.7%. Five read queries (point, range, group by, top-n) over the 3 segments take the same or less time and blocks within +14% on a few dozen (1,000,000 rows: 0.16 to 4.6 ms; 3,000,000: 0.2 to 13.8 ms); the next merge of the 3 segments with the buffer takes 1,074 ms at 1M where it took 1,302 ms after a one-segment rewrite (3M: 2,711 and 2,704 ms), 2 workers 648 ms.
- Improved: pg_sift: a group table of many times hash memory is read in passes of the groups, not spilled: when the table of a serial `GROUP BY` (with `count(DISTINCT)` or other aggregates) first passes `work_mem` early in the scan, the scan starts again as passes over the units, each for the groups whose integer, date, timestamp or text key hashes to some of 32 shares, and the groups of a pass go to the result at once, with no partial state or set written and merged (`Group Passes` and `Group Pass Restarts` in `EXPLAIN ANALYZE`; no setting). At `work_mem = 4MB`, `count(DISTINCT uid)` and `count(*)` by a text key of 110,000 groups over 1,000,000 rows take 76 ms where the spill took 344 ms (1,283 to 416 M instructions, 4,465 temporary blocks to none; PostgreSQL 755 ms), by an integer key 61 ms where it took 143 ms (809 to 372 M instructions); 3,000,000 rows of 110,000 groups 356 ms and 266 ms where it took 1,065 and 684 ms; 3,000,000 rows of 1,500,000 groups at 64 MB 426 and 580 ms where it took 776 and 1,330 ms (PostgreSQL 1,743 and 1,512 ms). `orders` of 1,000,000 rows, `count(DISTINCT category)` by `customer_id` (200,000 groups) at `work_mem = 8MB`: 65 ms where PostgreSQL takes 533 ms (572 M instructions, 933 blocks read cold). A table that the estimate puts above about 25 times hash memory is spilled as before, with no pass; one share that does not fit anyway leaves the rest of the scan to the spill, read together: the same temporary blocks and up to 6% more instructions (192 to 204 M at 300,000 rows of 107,000 groups at 1MB; 2,462 to 2,498 M at 1,000,000 rows), the time not distinguishable from the spill's.
- Improved: pg_sift: the merge of the segments of a `sift_col` index with `key_cut` outside a build whose output is cut into partitions (the index is over `sift.vacuum_rewrite_limit` or a third of `maintenance_work_mem`, as `sift.col_merge(index, true)` at 64 MB of `maintenance_work_mem` over 1,000,000 rows of 13 columns, 28 MB of segments cut into 2) is made in a slice for each partition by the backend and one worker per slice less the backend, where one process merged them one after the other: 1.71 s becomes 1.11 s (median of 3, 1.68 / 2.49 / 1.71 s against 1.20 / 1.11 / 1.06 s), the instructions of all the processes +6.5% (9,660 M to 10,290 M), the WAL the same (59.6 MB) and the segments 0.1% longer (61.86 MB to 61.90 MB); `max_parallel_maintenance_workers = 0` merges by one process as before.
- Improved: pg_sift: a scan with `ORDER BY key LIMIT n` over a `sift_col` index with a non-empty write buffer keeps the buffer's order ranges with its batches for the next scan of the backend and extends them when records are appended, where every scan rebuilt them, and reads no header page of the buffer when the batches are kept: `top_asc20` with 5,000 buffered rows over 100,000, 300,000 and 1,000,000 rows takes 126 k, 142 k and 143 k instructions where it took 437 k, 454 k and 453 k (1.2x of the same query with no buffer; with 20,000 buffered rows 129 k to 145 k where it took 1.35 M), and 13 to 22 blocks where it took 14 to 23. The ranges are rebuilt when a record is deleted or rewritten, when the rows appended lie far outside the keys already held, or when the key changes; a failed extension leaves the old ranges in place (`sift.test_col_gkept_fail`, `sift._test_col_gkept_stats()` for tests).
- Improved: pg_sift: a grouping-sets query (`ROLLUP`, `CUBE`, `GROUPING SETS`) whose `ORDER BY` or window sorts by its group columns gets its groups from the scan in that order, as a plain `GROUP BY` does, where PostgreSQL sorted every row of the result above it; a rollup row ties with the real NULL group of its key with the set of more keys first, as PostgreSQL's own rollup returns them.
- Improved: pg_sift: a scan that groups by two to four keys, whose rows mostly open a group of their own, starts its group table as a hash table sized for the groups the planner expects, where the arrays indexed by the keys doubled a key at a time and copied every accumulator at each doubling.
- Improved: pg_sift: a lookup join loads the dimension table it reads whole into a map sized for its rows, hashes consecutive integer keys into consecutive slots (a cache line for sixteen of them) and looks a by-reference column value up without copying it; an integer key whose range holds more than 4,096 values and 16 values for each selected row goes to the hash table at once.
- Improved: pg_sift: a scan whose groups the `ORDER BY` wants in order prefetches the key ids and the accumulators of the groups sixteen ahead of the one it builds the tuple of.
- Improved: pg_sift: the check against the heap of the rows a scan selects on a page that is not all-visible (a table no VACUUM has covered) no longer decodes the offsets of every row of the page, and visits only the selected rows: a count of 1% of the rows of a 1M-row table after update and delete churn without VACUUM takes 25.5 M instructions and 5.0 ms instead of 56.4 M and 9.8 ms (everything in shared buffers; the b-tree: 18.5 M, 2.4 ms), with the same blocks.
- Improved: pg_sift: a query that reads a unit with dead rows reads its liveness bits 64 at a time instead of a byte at a time: a grouped sum over a 13-column index of 300k rows with 9% of its rows dead takes 1.43 M instructions instead of 2.06 M (min/max 1.56 M instead of 2.19 M), and the VACUUM that rebuilds the sidecar reads them the same way.
- Improved: pg_sift: the zone and the exact sums of the leaves, and the sums of the groups of a cube measure, which the sidecar of a VACUUM and the build of a unit compute, use two new SIMD kernels (AVX2; the AVX-512 table binds the same function; NEON stays scalar): VACUUM of a 13-column index of 300k rows after 1.5% churn takes 648 M instructions, 878 M before (same bytes).
- Improved: pg_sift: the sidecar a VACUUM writes for the units with dead rows takes a sixth fewer instructions: the rollups use the grouped pass of the cube, the live rows are compacted by runs between the dead ones, a sum of keys within 2^31 of zero is added in 64 bits, and a dim whose zone is narrower than its table finds a group by the key instead of a hash (VACUUM of a 13-column index of 300k rows after 1.5% churn: 1,063 M instructions, 878 M now, same bytes).
- Improved: pg_sift: the cube of a unit adds the integer measures of a dimension in one pass per group over the rows ordered by group, with the sum, count, minimum and maximum in registers, in the build of a segment and in the sidecar of a VACUUM: the cube build of a 13-column unit takes a third fewer instructions, and the sidecar's bytes are the same.
- Improved: pg_sift: the zone of a chunk without a NULL and the exact sums of an integer leaf whose zone lies within 2^31 of zero take shorter loops (no per-value NULL or overflow tests): VACUUM of a 13-column `sift_col` index with 300k rows takes 1,045 M instructions instead of 1,595 M (100k rows: 340 M instead of 480 M), with the same index bytes.
- Improved: pg_sift: a search reads the documents of the open write buffer of a BM25 index by their hashes: the terms of a document are compared as 32-bit hashes (the bytes only on a hit), the document is tested against the query tree once from the hits, and a term that scores keeps its entry between the documents, where every query scanned the terms of every buffer document once per leaf of the query: the cost of a buffered document falls from 3.1k to 0.78k instructions per query, a search of 8,000 buffered documents runs 9.4 M instructions where it ran 27.9 (an OR of three terms), 6.9 where 20.9 (an AND), 9.0 where 27.9 (a mixed query) and 10.0 where 24.6 (a phrase) at 100,000 documents, and `search_open_buffer` 5.2 M instructions where it ran 10.8 (100,000 documents), 10.8 where 16.4 (300,000) and 27.5 where 33.1 (1,000,000); the answers are the same as of a promoted table (`wbuf_hits`)
- Fixed: pg_sift: `CREATE INDEX .. USING sift (body) INCLUDE (..)` on a `sift_table` is refused with "INCLUDE columns need a heap table" when the index is created: on an empty table it succeeded, and the first promotion then failed for ever (the values are read from heap tuples), so sealed segments piled up (11 after inserting 390,000 rows) and every VACUUM warned.
- Improved: pg_sift: the bitmaps of the sets of a `count(DISTINCT x)` give way to hash sets when they pass 1 MB and cost over 128 bytes for each value they hold (ids spread over a wide domain, a few in each group: a chunk of 512 bytes and 8 kB of directory for each group), where they went on up to the hash memory: 1,000,000 rows of 60,000 groups and 4,000,000 values take 153 ms and 346 million instructions at `work_mem` 256 MB where they took 746 ms and 1,516 million (PostgreSQL 878 ms, 6,286 million), 226 ms and 346 million at 64 MB where they took 464 ms and 1,144 million.
- Improved: pg_sift: the pairs of group and value of a `count(DISTINCT x)` that outgrow the hash memory are sorted by radix as 8-byte keys, in an array of the size of the hash memory with sorted runs on disk past it, where they went through a tuplesort of int8 datums: 1,000,000 rows of 1,000 groups and values over 2^31 take 48 ms and 393 million instructions at `work_mem` 4 MB where they took 229 ms and 1,953 million (PostgreSQL 428 ms, 4,200 million).
- Fixed: pg_sift: `sift.suggest_paths` (and the `CREATE INDEX` of a `sift_col` index on a `jsonb` column with no `paths`, which calls it) on a table of another access method, a `sift_table`, ended the server process: the sample limits a scan of heap blocks and the scan of a `sift_table` is not one. Such a table is now read by one scan from its start, up to the sample, and gives the answer of a heap table with the same documents; the stretches spread over the table are those of a heap table (`col_suggest_paths`)
- Fixed: pg_sift: a phrase query (`@@> '"two words"'`) on a column of a `sift_table`, and the maintenance of a BM25 index that needs the text of a row for its positions, failed with 'trying to store an on-disk heap tuple into wrong type of slot': the text of a row is now fetched through the table access method (`tbl_bulk_groups`)
- Improved: pg_sift: the cube of a unit takes the minimum and maximum of a measure without a branch for the first value of each group, in the build of a segment and in the sidecar of a VACUUM: VACUUM of a 200,000-row table (7 columns, 4% of the rows changed a round, a sidecar for every unit) executes 463 M instructions where 483 M before (-4%, the sidecar's part 265 M, -7%); the pre-aggregates are the same bytes.
- Improved: pg_sift: the choice of the default `key_cut` of a `sift_col` index, which `CREATE INDEX`, `REINDEX` and the rewrite make from a sample of the heap, sorts the sampled rows by a radix sort of the key where it compared them, puts the rows of each unit in the order of their addresses in one pass where it sorted them again for every leaf it prices, and no longer sorts the addresses of a unit that come out in order: the choice (`sift.key_cut_decision`) of a table of 200,000 rows with four rounds of 4% churn takes 0.83 G instructions where it took 1.28 G (-35%), a serial `REINDEX` of it 3.03 G where 3.57 G (-15%), `sift.col_maintain` 2.67 G where 3.21 G (-17%); the choice, the index and its size are the same.
- Fixed: pg_sift: the WAL budget of the automatic rewrite of a `sift_col` index (`sift.col_auto_merge_wal_pct`) no longer holds back the promotion of the write buffer: while the units with dead rows were due for a rewrite and the budget deferred it, the rows of the buffer were not promoted either, and every query walked them and the dead rows of the units for the 8 rounds of 4% churn that followed (without sidecars: 100,000 rows 14,773 rows in the buffer and a grouped sum of 13.6 M instructions and 942 blocks against 0.74 M and 244 blocks after a merge, 300,000 rows 18.0 M and 1,644 blocks against 0.39 M and 244); with it the buffer holds at most one round (2,032 rows at 100,000 rows) and the grouped sum reads 0.77 M instructions, as after a merge, where the rewrite then runs. Test `col_net_policy`.
- Fixed: pg_sift: a write buffer of more than 4,096 rows (or 8 times the `LIMIT` and `OFFSET`) turned every `ORDER BY key LIMIT n` over the sorted arrays into a scan of the units; the numeric buffer is now cut into ranges of the key that are sorted only when a read touches them, to a bound of memory (2,097,152 rows): 1,000,000 rows, `ORDER BY ts LIMIT 20` with 5,000 buffered rows 4,457 blocks and 11 ms before, 62 blocks and 0.73 ms after; 20,000 rows 14,729 blocks and 32 ms before, 125 blocks and 2.0 ms after; 100,000 rows 16,276 blocks and 47 ms before, 444 blocks and about 10 ms after (a b-tree 23 blocks).
- Changed: pg_sift: the order array of a text column (`order_array_columns`) names the strings of its ranks by their dictionary codes, and keeps no second copy of them, where the dictionaries of the index, parsed and added up, take three quarters of the cache of parsed dictionaries (64 MB) or less and the codes save a fifth of the array and the dictionary: 300,000 distinct 32-character hashes take 22.3 bytes a row where they took 55.8 (a b-tree 59.1), 22-byte e-mail addresses 23.3 to 6.2 bytes a row; strings that share long prefixes (`customer-0000123456`, 7.4 bytes a row in the sorted array) and dictionaries that pass that share (900,000 hashes, four segments: every equality parsed a dictionary again, 10 to 25 ms against 0.13; the shared ring of `sift.col_dict_shared_cache_kb`, 64 MB, was thrown out as fast as it was filled, and with 256 MB the same index took 0.13) keep their strings. The reads of the codes cost more than those of the strings, 300,000 hashes in one segment: `LIMIT 20` 0.03 to 0.15 ms, a range with `LIMIT 20` 0.09 to 0.28 ms, OFFSET 20,000 2.2 to 7.0 ms, an equality 0.08 to 0.54 ms (a b-tree 0.01, 0.01, 0.8, 0.007), until the backend has parsed the dictionary: the first backend after a server start parses it (an equality 40 ms, the first statements of a range 0.3 ms until its 80th), and a new backend or pooled connection after it reads it in the shared ring (1.4 to 2.8 ms for its first statement, 0.12 ms after, for 100,000 to 1,000,000 rows). These segments are column format minor 42, which 3.7 and older cannot read (`sift.col_format_minor` 14 and a `REINDEX` write what they read); segments of older minors are read and merged as before.
- Improved: pg_sift: the hash table and the sorted order of a dictionary that the backends share (the ring of `sift.col_dict_shared_cache_kb`) are made by the second search or the second test of the first backend that meets it, not after 8 and 16 as for one that stays private: they are written to the ring, so the cost is the server's and every other backend finds them made. A text equality on 1,000,000 distinct 9-character keys whose order array keeps its strings, so that no statement had read the dictionary by code before, took 1.98 ms and 37.9 million instructions for each of its first statements and now takes 0.05 ms and 0.47 (a prefix count 3.44 ms and 57.5 million, now 1.62 and 14.0); the first backend after a server start still parses the dictionary once, 43 to 77 ms for that dictionary.
- Added: pg_sift: the reloption `order_layout` of a `sift_col` index (`auto`, `whole`, `compact`, or `'compact: ts, created; whole: id'` for the columns it names) chooses how the sorted array of a column (`order_array_columns`) is laid out for each segment a promotion, merge, rewrite or `REINDEX` writes. `compact` keeps the heap address of every row and the key of each entry to within a bucket of a few bits, and reads the exact key from the table row where a scan needs it: 3.6 bytes a row for a microsecond timestamp where the whole order takes 6.5, an index of 1,000,000 rows 12.12 MB against 15.02 (a b-tree on the column: 22.49), and `ORDER BY ts LIMIT 20 OFFSET 100000` reads the same entries in 3.27 ms and 108 cold blocks against 3.08 ms and 142 (a b-tree: 4.10 ms, 277). `auto`, the default, keeps a chunk of 512 entries compact where that is at least a fifth smaller and has no run of 8 equal keys, and whole elsewhere, so a column of keys that tie much (a date: 5.6 bytes a row) is never larger than before; a segment with a compact chunk is column format minor 41, which 3.7 and older cannot read (`order_layout = 'whole'` and a `REINDEX`, or `sift.col_format_minor` 14, write what they read), and `CREATE INDEX` says the share of compact chunks and the bytes a row of each column.
- Improved: pg_sift: `ORDER BY key LIMIT n OFFSET m` over the sorted array of the key throws away the entries before the OFFSET without reading their keys: the cursors of the segments start where the buckets of the keys put the OFFSET, and the entries on pages that are not all-visible are confirmed in the heap in one pass, so a range bound, a write buffer and 5% of dirty pages keep the skip (1,000,000 rows, OFFSET 100,000: 47.8M instructions to 37.0M ascending, 59.7M to 37.2M descending, 7 segments 64.8M to 38.5M; 300,000 rows with a lower bound, OFFSET 10,000 over a compact order with 5% of the pages not all-visible: 10,590 blocks to 619, a b-tree 538).
- Improved: pg_sift: `ORDER BY key LIMIT n` over the sorted array of an integer, date or timestamp key that is the only column the query selects returns the keys from the entries and reads no row of the table for the entries that are exact (a page all visible, the row not dead), and `OFFSET` throws away the entries before it by the bisection of the key only from 2,048 rows on, where below it walking them costs less than the bisection's 0.4 million instructions (`topn_scattered_time`, 300,000 rows, a table in the order of a hash: 0.591 ms became 0.300 ms, the cold run read 792 blocks in 802 reads of 8 KB and now reads 31 blocks in 35 reads, 6.19 MB became 0.24 MB, 4.16 ms became 0.52 ms). The reader of a segment reads, when a page that is not in the buffers follows the last such page by a page or two, that page and the next 3, then 7, up to `io_combine_limit` pages in one read (the buffer manager asked the kernel for them page by page, so a walk of a segment in gets of a few hundred bytes was a read of 8 KB each): the cold run of `topn_numeric_wide` (300,000 rows) is 116 reads of 41 KB where it was 392 of 8 KB (4.57 MB read where it was 2.99 MB), at 1,000,000 rows 330 reads of 46 KB where PostgreSQL's scan is 982 reads of 128 KB; a page in the buffers is no read and the warm run is unchanged.
- Improved: pg_sift: `ORDER BY col LIMIT n [OFFSET m]` over a `numeric(p, s)` column of up to 18 digits reads the sorted array of the column like an integer one (`order_array_columns` takes it, and `auto` chooses it by the same rule; the key is the number scaled by 10^s, kept whole, a segment with a NaN or an infinity has no array and the units answer): an ORDER BY amount, id LIMIT 20 OFFSET 10000 over 300,000 rows read every row of 7 units (12.2 ms, 90 million instructions, the sorted runs push 47,000 rows through a heap of 10,020) and takes 1.25 ms and 7.5 million instructions (a b-tree 1.19 ms, 6.6 million); at 1,000,000 rows 21.9 ms and 189 million instructions become 1.34 ms and 8.2 million (a b-tree 1.0 ms, 6.6 million); the array is 2.9 to 3.4 bytes a row, and a condition on the numeric column (a bound) still reads the units (`topn_numeric_offset`).
- Improved: pg_sift: the walk of the global order with a condition on another column (`WHERE deleted_at IS NULL ORDER BY k, id LIMIT 20 OFFSET 10000`) asks the condition before it asks whether the entry is exact: no entry of such a walk is exact, and the dead list of every unit it passed was read for nothing (1,000,000 rows after a VACUUM: 18.2 ms and 62.9 million instructions become 4.6 ms and 26.4 million, a b-tree 2.2 ms and 20.4 million; 300,000 rows 34 to 26 million instructions, 3.4 ms against 2.3).
- Improved: pg_sift: VACUUM of a `sift_col` index writes, after each segment that has dead rows, the cube, rollups and sums of every dirty unit computed over its live rows (`SIFT_SEG_NET`, no format minor: a build that does not know the flag reads the segment as written), and a unit whose dead rows are the ones they were made for answers from them as a clean unit does: a minimum, a maximum and a float sum too, where `sift.agg_dead_delta` takes the dead rows off counts and integer sums only, and whatever the share of dead rows. The automatic rewrite no longer counts such units as dirty. 8 rounds of 4% churn (each round: promote, VACUUM), instructions of one query on the dirty index, the sidecars against the dead-row delta alone (the same index rewritten): grouped sum 100,000 rows 4.60 to 1.51 M (0.71), 300,000 rows 1.11 to 0.63 M (0.39), 1,000,000 rows 46.2 to 16.0 M (10.0); minimum and maximum 2.80 to 1.57 M (0.80), 2.37 to 0.66 M (0.42), 32.1 to 16.1 M (10.4); blocks at 1,000,000 rows 2,564 to 1,139 and 1,972 to 1,139 (677). A filter, a range, a distinct count and a full scan read about the same blocks as with the delta alone. The index is still not as cheap as the rewritten one: its three segments cost 0.1 to 0.2 M instructions a query at 300,000 rows, and 462 blocks at 1,000,000 rows, which only a merge returns. The maintenance WAL of the 8 rounds at 100,000 rows: 9.5 MB with the sidecars, 5.7 MB leaving the units dirty, 15.6 MB rewriting the index every round; the sidecars cost 3.9 MB where the rewrite costs 9.9 (-61%), and 2.3 MB of each total is the heap's own VACUUM (-39% of the totals); at 1,000,000 rows 53.8 MB, 49.7 MB and 206.9 MB (-74%). `sift.col_sections` shows the sidecar as `net`; `sift.col_verify` rebuilds it from the live rows. A distinct count or a sketch still reads the unit by its rows.
- Fixed: pg_sift: `sift.col_merge` looks at the free end of the file on every call, not only on one that merged: a cut that found the index locked (a transaction that planned a query holds a lock on every index of the table) is made by the next call, where it stayed until a VACUUM (a merge into an empty end, a transaction holding the index: 195 blocks before the next call, 89 after it).
- Improved: pg_sift: a scan whose groups the `ORDER BY` wants in order sorts the distinct values of an integer, date or timestamp key by the value and the groups by a radix sort of their packed ranks, where each compare was a call, and builds the `numeric` of a `sum` from its digits, where it wrote the text of the sum and parsed it again: 1,000,000 groups of the hour and the customer (`GROUP BY date_trunc('hour', created_at), customer_id ORDER BY 1, 2`), 4MB of `work_mem`, 634 to 469 ms and 5,858 to 4,452 M instructions (4GB: 569 to 354 ms, 4,512 to 3,342 M); the same with a status and the last year, 345 to 258 ms (244 to 210 in memory); 3,000,000 rows 1,943 to 1,506 ms (1,680 to 1,418 in memory), 300,000 rows 205 to 155 ms (151 to 113); PostgreSQL's plan takes 787, 395, 2,386 and 229 ms. The rows and their order are the same.
- Improved: pg_sift: `CREATE INDEX` of a `sift_col` index with `key_cut` sorts the order arrays by radix where it sorted them by comparison: the rows of a segment cut by a key are not in the order of their addresses, which the radix sort needs, and a radix sort of the addresses now puts them there. The build takes 10% less (1,000,000 rows, default options: 1,301 to 1,187 ms without workers, 1,116 to 1,031 ms with 4; 300,000 rows 965 to 875 and 928 to 869 ms; 100,000 rows 345 to 314 and 352 to 324 ms); the index has the same bytes.
- Improved: pg_sift: the merge of a `sift_col` index (the maintenance worker's, `sift.col_merge`) cuts the free end of the file it leaves, as the rewrite does: a stream of 1,000-row batches into an empty table (1,000,000 rows, 14 columns, `sift.col_auto_merge` on) left the segments that were merged at the end of the file free but held, 112 MB (117 B a row) at 1,000,000 rows and 60 MB of reusable blocks; with the cut the file is 85.4 MB (89.6 B a row, -24%) and 33.5 MB reusable in 2 of 3 runs, and 112 MB again in the third (the cut waits for a lock that was not free); the WAL of the stream is the same (851 and 850 MB against 846 to 868). At 300,000 rows 45 against 66 B a row. Test `col_rewrite_cut`.
- Improved: pg_sift: a scan of one table that groups is priced as starting at 0.14 (55,000 operator steps) where it was priced at 0.25 (100,000), for 0.3 ms of the clock measured warm: a tenant of 6,700 rows of 100,000 by month ran in the normal plan, 1.8 ms, and runs in the scan, 0.85 ms (a tenant of 15,800 rows 4.0 against 0.83); the tenants of 5,000 rows or fewer stay with the normal plan (a tenant of 1,300 rows: 0.43 ms against 0.59 in the scan).
- Improved: pg_sift: the choice of `key_cut`, the order arrays and the point directory that a `CREATE INDEX` of a `sift_col` index makes before it reads the heap takes 450 ms where it took 565 ms (1,000,000 rows, 7 leaves, the same choices): a column whose values are only zoned is not sorted to find its smallest and largest, and a unit with more than 256 values in a column is not sorted to count them. The build of 1,000,000 rows takes 1.53 s without workers and 1.25 s with 4 (1.59 and 1.53 s before).
- Fixed: pg_sift: `CREATE INDEX` of a `sift_col` index with workers no longer chooses `key_cut`, the order arrays and the point directory a second time in the rewrite that merges the segments of its participants (the build had just chosen them from the same data): 1,000,000 rows, default options, the build with 2, 4 and 8 workers took 2.0 to 2.3 s against 1.6 s without workers, and takes 1.4 to 1.5 s now, reading 67 to 74 MB less than before the choice and writing the same 15 to 18 MB of WAL (`bench/steps/col_build_workers.sh`).
- Improved: pg_sift: the global orders of an index with order arrays and no point directory or ends (`order_array_columns` alone, the default of the build for a scattered integer, date, timestamp or text column) also code their addresses with the bits of the largest offset of the chunk, as the segments with a point directory did since 3.7 (column format minor 38, written under `sift.col_format_minor` 15 or more; a 3.6 library refuses the segment with the `REINDEX` hint, a segment written before is read as it was and the next merge rewrites it): the arrays of 1,000,000 rows take 3.50 to 3.00 bytes a row for a unique integer, 3.20 to 2.21 for a date, 7.00 to 6.50 for a random timestamp and 4.00 to 3.50 for a scattered integer, the index of four array columns 44.6 to 42.1 MB (-5.6%; 100,000 rows 4.44 to 4.21 MB, 300,000 rows 13.52 to 12.75 MB), with the same blocks read cold and the same instructions (within 1.5%) for `ORDER BY .. LIMIT` and range reads (3 sizes).
- Improved: pg_sift: the write buffers are searched by their narrowest condition: a scan of a `sift_col` index with an `IN` list and an equality on the same key (a scope of 4,650 keys and the key of one entity) looked the rows of the buffers up by the list, the planner's first condition, and every buffered row of every key of the list went to the heap to be dropped there; the lookup now takes the condition with the fewest values, and a list alone that keeps more than a quarter of the buffer loads its batches, whose tests check every condition. 3,000,000 events with 81,586 rows in the write buffers: 4,757 index rows, 186 removed by the heap and 1,149 heap blocks, where it was 85,693, 81,122 and 4,652; a list and two range conditions, 50,000 index rows and 771 heap blocks where it was 119,086 and 3,959.
- Improved: pg_sift: a scan keeps the rows of a write buffer as batches for the next scan of the backend, which reads only the records appended since, where every scan read, sorted and decoded every record of the buffer, 5,000 instructions a row on each statement until the buffer is promoted. A burst of 2,000 inserts into 1,000,000 events, then the page of the newest events: 12.5 M instructions a read where the clean index takes 1.6 M, now 2.1 M after a VACUUM (1.9 M promoted); the grouped count 62.2 M, now 51.8 M (50.7 M promoted); at 100,000 rows 17.6 M, now 7.2 M (6.0 M promoted). With a write between every ten reads the page costs 2.7 M on average where it cost 3.9 M, the count 29.6 M where it was 34.2 M. A record VACUUM flags dead stays in the kept batch, as in the copy a long scan holds: the scan checks every row of a write buffer against the heap; a row whose address is already kept, or is below the kept ones, builds the batches again.
- Improved: pg_sift: a ranked search with a test of an `INCLUDE` column does not read a superblock where no value of the column passes it (the range is in the header of the record, which a segment written from now on carries; a merge or a `REINDEX` writes it for an older one): on a table whose column follows the heap order, `created_at >= ` the last 10% of the dates over 1,000,000 documents runs 17.6 M instructions a statement down to 4.7 M (1.67 ms to 0.68 ms; 100,000 documents 3.2 M to 1.7 M, 300,000 6.8 M to 2.5 M), and a table whose values are spread over every superblock runs 0.2% more (0.6% over 100,000 documents).
- Improved: pg_sift: the table of an `IN` test of an `INCLUDE` column (and of the keys of a small joined table) is built once per superblock, not once per block: the join with a table of the regions runs 5.1%, 7.5% and 8.9% fewer instructions over 100,000, 300,000 and 1,000,000 documents (24.6 M to 22.4 M).
- Improved: pg_sift: the exceptions of a unit (the cube of the few rows where a soft-delete stamp is not NULL, column format minor 19) are kept when they take at most half of the unit's own cube or one byte for each 32 rows of it, not only the half: a `sift_col` index cut by its grouping column holds few groups in a unit, its cube is as small as its exceptions, and `WHERE stamp IS NULL` scanned every unit with a stamped row. 400,000 rows in 31 units cut by `tenant` (1 row in 101 stamped): `SELECT status, count(*), sum(qty) WHERE stamp IS NULL GROUP BY 1` answers 31 of 31 units net of exceptions in 0.35 to 0.45 ms, where it scanned the 400,000 rows in 3.1 to 3.7 ms (8x); the index takes 3.3 KB more of 573 KB (+0.6%).
- Improved: pg_sift: the scan prices the heap of a deep page (`OFFSET` past 1,024 rows) by the rows of the offset and the page, 70 operator steps a row, where it priced it by the rows read times the log of the page: over 600,000 rows a page of 100 at an offset of 2,000, 5,000, 20,000 and 50,000 takes 1.5, 3.1, 9.8 and 19.4 ms more than at an offset of 100 whatever the rows that match, and the price is within half and twice of that at each (it was 6 to 12 times at the first three and 5 times at the last when all the rows matched, a third of it when a tenth did); a text-key page of 12,000 to 28,000 rows over 30,000 now takes the sort of the rows (8% to 19% faster), and a page of 5,000 rows behind a list and a sub-select that the planner under-estimates takes the scan.
- Fixed: pg_sift: the merge of segments of a `sift_col` index with `key_cut` read the leaves of the cut from the cached state of the index while it wrote the units, and a lock taken meanwhile that reloads the entry freed them: `sift.col_merge` ended in a segmentation fault now and then next to writes and `VACUUM` (`col_index_crash`: 2 runs of 3 with the crash; a copy of the leaves is kept for the merge).
- Fixed: pg_sift: `VACUUM` of a `sift_col` index moves the top of the segment that ends the file into a hole that holds exactly its blocks: the move left no block for the free-list page it publishes and was refused, so the hole stayed (a seeded run of inserts, updates, deletes and VACUUMs with a text column's order array: 335 of 2,338 blocks free after the VACUUM, 14%; now 1 of 2,004 and the file 14.3% smaller); the last piece of the move stays where it is.
- Improved: pg_sift: a statement that holds a literal `IN` list and a sub-select over the same column is planned once as a flat form and once as the original only when the flat form can change the plan, not twice every time: planning a list of 3,960 literals next to a sub-select with a `LEFT JOIN` takes 30.2 million instructions where it took 51.7 (31.3 with the extension off), 850 literals 7.8 where it took 12.9 (7.4 off).
- Improved: pg_sift: the ordered spill of a scan (a rollup ordered by its time bucket past hash memory) writes the partial states once: when the first key of the order is the value of a leaf or a step of it, the tables go at once to equal widths of the range the zones of the leaf give, where they went to one stream that was merged and written again by range. Median of 2 rounds of 5, ms, work_mem 4MB (4GB), 1,026,312 rows: 10 days 106 to 74 (103), 30 days 300 to 212 (257), 90 days of 1,026,312 groups 850 to 557 (528): 60% over the table in memory became 5%. A first key that is not a step of a leaf of its type, and a range that holds more than the table (a filter that leaves a tenth of the range) keep the sample (`agg_group_spill_order`). Then no partition at all when the zones of the superblocks are mostly apart: the superblocks are read in the order of their zones, the groups below the zone of the next one are final and are sorted and returned as a batch while the table keeps the others (`group stream batches` in `EXPLAIN`). `work_mem` 4MB (4GB), 1,000,000 rows (999,578 groups): 7,980 temporary blocks read and written before, none now; 4,453 million instructions before, 3,519 now (3,346); 3,000,000 rows 13,536 before, 10,890 now (10,065); median of 6, ms, 300,000 rows 121 (117), 1,000,000 rows 373 (380), 3,000,000 rows 1,424 (1,595). A table whose zones overlap (a random heap order) spills as before, after the batches it has given (`agg_group_stream`).
- Improved: pg_sift: `VACUUM` seals the open write buffer of a BM25 index once it holds a quarter of `write_buffer_limit` (1 MB at the default), as it did for a `sift_col` index, and promotes its rows: a search reads every document of the open buffer. 300,000 documents (16 MB) with 4% of the rows changed in each of 8 rounds and a `VACUUM` after each: a search reads 129 blocks in 0.17 ms where it read 684 in 1.6 ms, a top-10 361 blocks in 2.2 ms where it read 924 in 4.9 ms, the index is 18.0 MB where it was 20.9 MB, and the WAL of the 8 rounds is 3.0% more (511 against 496 MB; the heap alone writes 470), test `bm25_buffer_share`
- Improved: pg_sift: `CREATE INDEX` with `stemmer = 'en'` spends 30% fewer instructions (1,000,000 documents, serial build: 133,762 M to 93,757 M; 78,977 M with no stemmer): the stem of every word of the vocabulary is kept (at most 8 MB per process) instead of the last 4,096 words. The index is the same bytes, the tokens of 20,000 documents are the same, queries and inserts read the same instructions (or / phrase / mixed 44.508 / 6.729 / 19.728 M).
- Improved: pg_sift: a scan whose groups the `ORDER BY` wants in order (a rollup by `date_trunc('hour', ts)` with the tenant and the model, `ORDER BY` the bucket descending) no longer gives up the order when its group table outgrows hash memory: it spills in ranges of the first key, an integer, a date or a timestamp, merged in the order of the key and each sorted by itself, where the scan spilled by hash and a Sort followed. 1,026,312 rows (345,272 groups in the last 30 days) at `work_mem` 4MB, median of 5, ms: 10 days 130 to 100 (101 with the table in memory), 30 days 392 to 293 (250), 90 days of 1,026,312 groups 1,240 to 854 (520); the rows and their order are the normal plan's (`agg_group_spill_order`). The same rollup over the last year (456,044 groups) no longer spills when the zones of its superblocks are mostly apart: it gives the groups below the zone of the next superblock as a batch, with no temporary block (3,849 blocks were read and written) and 1,790 million instructions where it took 2,248 (1,736 at `work_mem` 4GB); 3,000,000 rows 6,741 before, 5,488 now (5,224); 207 ms at `work_mem` 4MB against 206 at 4GB.
- Improved: pg_sift: `count(x)` of a column the table declares `NOT NULL` is `count(*)` for the scan, whether or not the `sift_col` index holds the column: a grouped series that counts the primary key (`count(id)`, the key outside the index) kept PostgreSQL's plan because the scan computes `count` over the columns it reads; on a 100,000 / 300,000 / 1,000,000-row table it now takes 41.6 / 121.7 / 166.1 ms against 106.8 / 333.9 / 961.2 (2.6x, 2.7x, 5.8x), the time of the same statement with `count(*)` (42.0 / 129.5 / 140.6 ms); a column that may be NULL, and a column on the nullable side of an outer join, are counted as before
- Improved: pg_sift: the rewrite of the segments of a `sift_col` index (the automatic one, `sift.col_maintain`) cuts the free end of the file it leaves at once, not at the next `VACUUM`, which leaves the end to a rewrite that is due: on a table of 300,000 rows with 4% of the rows changed in each round the file after the rewrite is 2.7 to 2.8 MB in every second round (it was 5.1 to 5.2 MB) and 5.1 MB in the others, the WAL of the rewrite is the same (2.3 MB a round), test `col_rewrite_cut`
- Fixed: pg_sift: the rewrite of every segment, which chooses `key_cut` again from the data, failed with "rows must be added in ascending TID order" when it changed the key and a segment cut by the old key held more than one unit (a parallel `CREATE INDEX` of a table of 205,640 rows with 2 workers and `maintenance_work_mem = 256MB`, 6 runs of 6; a merge or `VACUUM` of an index whose cut changed): the rows of a segment loaded whole are sorted by reference, so their units need not follow the TIDs. `col_key_cut_recut`.
- Improved: pg_sift: a key `date_bin(stride, ts, origin)` of whole seconds is a step of its own, like `date_trunc`: the scan evaluates it once per step of the data (the gcd of the stride and the origin, 900 seconds for 15 minutes from `2000-01-01`; EXPLAIN says `(per 900 seconds)`), on a `timestamp` and a `timestamptz` column, where it evaluated it for every distinct timestamp: the two scan arms of the three-arm `date_bin('15 minutes', ..)` queue statement over 1,000,000 rows take 8.0 ms against 23.2, and the statement 99.6 against 171.6 ms for PostgreSQL's plan (-42%; 32.9 against 53.7 ms at 300,000 rows, -39%; 14.9 against 19.3 ms at 100,000, -23%) without parallel workers, and 53.3 against 73.1 / 21.1 against 23.6 / 14.3 against 13.7 ms with two (-27%, -10%, +4%); a stride with a month, an origin with a fraction of a second or a stride or origin that is not a constant keeps the key per distinct timestamp, the answers are the same
- Improved: pg_sift: a `GROUP BY` over a `UNION ALL` of selects (a queue's depth over time from three timestamps of the same rows, one arm for each) groups each arm that runs in the scan by the keys and adds the groups up above the union, where every row of every arm went to one grouping (`count`, `sum` of ints and numerics, `min` and `max`; not `avg`, `DISTINCT`, `FILTER`, an ordered aggregate or a float sum): the three-arm `date_bin('15 minutes', ..)` statement over a job table of 100,000 / 300,000 / 1,000,000 rows runs in 15.8 / 35.2 / 110.1 ms against 20.2 / 52.4 / 155.2 (-22%, -33%, -29%) without parallel workers, with the same rows; an arm the normal plan is cheaper for (a key made of two columns, `coalesce(done_at, started_at + interval '1 hour')`) stays as written and is added in by the hash aggregate above, and the planner chooses by cost like the other flattened forms. With two parallel workers PostgreSQL's parallel plan of the same statement takes 13.1 / 24.1 / 73.3 ms against 14.4 / 24.2 / 75.9 for the scan's plan (a tie within 10%: the cost of the scan's plan does not yet count the share of the workers)
- Added: pg_sift: `stemmer` takes the name of a PostgreSQL text search configuration (`stemmer = 'german'`; `french`, `spanish`, `russian`, the 29 Snowball languages of the server, or your own with a dictionary chain: write configurations outside pg_catalog as `schema.name`). The words are the ones `to_tsvector` indexes (one pass of the dictionaries the configuration maps to word tokens, stop words dropped, the first lexeme of a dictionary that answers several), a stop word keeps its position so a phrase is two apart as `phraseto_tsquery` has it, queries use the same configuration, and accent folding is off for such an index unless `accent_folding = true` is given. The index keeps the configuration name in its metapage (format minor 1.5, which a 3.7 library refuses; `sift.verify` names it). On generic corpora of 340,000 documents the index is 25 to 32% smaller than with no stemmer and builds with the same instructions (german 3,457 to 3,446 M, french 4,247 to 3,772 M, spanish 3,310 to 2,975 M, russian 3,416 to 3,460 M), because the stop words are not indexed; `stemmer = 'en'`, the default and the English query side are unchanged (build 133,762 M and queries 40.097, 5.653, 17.820 M instructions for or, phrase and mixed, identical to 3.7). Each process keeps a memo of the words its dictionaries answered, at most 8 MB of entries and 4 MB of table.
- Improved: pg_sift: a ranked search spends fewer instructions on the page index of a superblock and on the liveness of a block of pages (same top 10 ids and scores in the 300 statements of each class): the page index reads the words of offsets its chunk can hold (1 of 8 for most) and a leaf of one word adds its bound to its pages without a row to zero and add. On 1,000,000 documents with the other 4.0 search changes: `or` 42.24 to 40.08 M instructions a statement (-5.1%), `mixed` -4.6%, `and` -3.8%, `phrase` and `filter` -3.3%; no format change.
- Improved: pg_sift: `merge_fanin` defaults to 4 (8 before): an index that takes batches keeps 4.6 segments on average and 8 at most where it kept 6.5 and 12 (1,000,000 documents ingested in batches of 5,000), and a statement reads fewer: and -9.6%, mixed -7.1%, phrase -8.2%, or -4.3% in instructions, 13% fewer MB cold. The merges write 3.10 bytes for each byte a promotion writes against 2.19, and the WAL of the ingestion grows 11%. An index that sets `merge_fanin` keeps its value; `ALTER INDEX .. SET (merge_fanin = 8)` keeps the old policy. Test: `merge_slope`.
- Improved: pg_sift: a backend keeps the blob of a term (8 kB or more) for its next statement, with the docmap directories it already kept, up to 32 MB (`bm25/segcache.c`): a statement no longer copies the blobs of its terms into memory the kernel maps and zeroes for it (365 page faults a statement of `AND` over 1,000,000 documents, 0.3 now): over 1,000,000 documents a fresh backend's 300 statements run `AND` 32% and phrase 28% faster, `count(*)` 35%, a ranked `OR` and a search with a filter or a join 10% to 12%, and the same backend again `AND` 41%; the answers and the index are the same
- Improved: pg_sift: a term that is a list (fewer postings than `sift.grouped_min_df`) is decoded where a search reads it: over 1,000,000 documents a ranked `AND` runs 33% fewer instructions (12% less time), a phrase 28% (13%), a ranked `OR` 8%, a mixed set of searches 10%, a search with a filter or a join 7%, and `count(*)` the same as before; the answers and the index are the same
- Improved: pg_sift: a ranked `OR` builds the candidates of a block from its terms' own groups when it scores the block, not as one merged group of the whole superblock: over 1,000,000 documents a ranked `OR` runs 20% fewer instructions (10% less time), a mixed set of searches 15% (5%), and a search with a filter or a join 7% fewer instructions; the union leaves out the terms that only the last bounds of a block can score (a further 5% to 10% less time); the answers and the index are the same
- Improved: pg_sift: a scan of a `sift_col` index over a table that is not all-visible (never vacuumed, or a standby, which cannot `VACUUM`) checks the rows it selects against the heap decoding the row offsets of the pages that hold a selected row only, where it decoded the offsets of every row of the unit first and walked every row of the unit to find those pages: 222.0 against 157.8 million instructions over 54 single-entity statements at 100,000 to 1,000,000 rows (a month of one entity at 300,000 rows 3.0 against 1.1 million; the b-tree plan 0.8 million); after `VACUUM` nothing changes.
- Added: pg_sift: a sift index can name `INCLUDE` columns (`CREATE INDEX ON docs USING sift (body) INCLUDE (category_id, created_at)`, up to 8 columns of `boolean`, `smallint`, `integer`, `bigint`, `date`, `timestamp` and `timestamptz`): the segments keep their values per document (segment format 5, metapage minor 1.5, written only by an index that names them; a 3.7 library refuses such an index with a `REINDEX` hint), a ranked scan drops the documents that fail `col <op> const`, `col IN (...)`, `IS [NOT] NULL` or a boolean column before it scores them, and a join with a table of at most 65,536 rows that has quals of its own adds the keys of that table's rows as one more test (`Include Tests` in `EXPLAIN`); every qual stays in the plan and is checked on the heap, so the answers are the post-filter plan's. A column costs 0.5% to 0.6% of the index.
- Improved: pg_sift: a test of `IS NULL` or `IS NOT NULL` on a column of a `sift_col` index is priced as a comparison (a bit test of each row), not as a pattern test (18 times the price): a scan of 1,000,000 rows with `parent_id = 7 AND deleted_at IS NOT NULL` costed 11,134 against the 10,883 of the bitmap scan of a b-tree on `parent_id` and took the b-tree (13.8 ms); it costs 2,583 and runs in 0.62 ms (100,000 rows 1.0 to 0.19 ms, 300,000 rows 3.1 to 0.26 ms).
- Fixed: pg_sift: a page of a list (`ORDER BY .. LIMIT`) whose rows tie on the first sort key beyond the statistics (the scan starts again with a larger heap) ended the server process, or returned no rows, when the page was served with the keys of a sub-select, an IN over a join or a joined table: the scan that starts again tested the keys it had made for the first start and ran the sub-plan, which had ended, once more.
- Improved: pg_sift: a scan with workers of many groups (thousands, a key that is an integer, a date or a timestamp) splits the groups by their key instead of the rows: each participant reads every unit and aggregates the rows of its share of the keys, the tables are disjoint and the leader only adds them to its own (`EXPLAIN`: `Groups Split By`). 3,000,000 rows and 100,000 groups take 35.2 ms against 40.6 ms in one process and 59.7 ms with 4 workers splitting the rows; 300,000 groups 86.8, 99.3 and 207.4 ms. The planner prices both plans and takes the cheaper; the rows and float sums are those of the other plans to the bit.
- Fixed: pg_sift: `VACUUM` of a table that was never analyzed, with a `sift_col` index of many segments (1,000,000 rows), failed with "ANALYZE cannot be executed from VACUUM or ANALYZE": the rewrite of the segments chooses the order arrays again, and the choice analyzed the table; inside a `VACUUM` it chooses from the statistics it has.
- Fixed: pg_sift: a `GROUP BY` of a text key with many repeats (30,000 groups over 1,000,000 rows) was priced for every dictionary entry of every unit as the growth of its table of ids, six times the integer key of the same groups (36,084 units against 5,395) when the two run in the same time (13.7 and 12.1 ms): the table is charged for the ids it gains, and the plan the workers are compared with is no longer made too dear (a near-unique key keeps its price, `agg_key_entries_cost`).
- Improved: pg_sift: a deep `OFFSET` over the sorted array of the first key with a tie-break key after it (`ORDER BY created_at DESC, id LIMIT 20 OFFSET 5000`) fetches the heap rows of the groups that hold the first row to keep and after it only: the groups of equal first keys wholly before the offset are counted from the array, an entry on an all-visible page that is not among the dead rows of its unit without its heap row, the others of the group by the heap. Over 1,000,000 rows the page takes 0.7 ms, 4.3 M instructions and 25 blocks read cold where it took 4.0 ms, 16.6 M and 3,277 (10,040 heap fetches, now 40); the b-tree on `(created_at, id)` reads 24 blocks cold in 0.5 ms.
- Improved: pg_sift: an `OFFSET` over the sorted array of a key does not confirm in the heap the entries it throws away after a `VACUUM` that followed deletes and updates: the entry is exact when its page is all-visible and its row is not among the dead rows of its unit (listed once for the unit from the liveness, where a unit with one dead row confirmed all its entries), which also covers the segments of one unit that parallel workers write (they have no summary of the pages of their units). `SELECT id, s FROM t ORDER BY s LIMIT 20 OFFSET 20000` over 300,000 rows with 1% of them updated, 0.1% deleted and a `VACUUM`: 20 heap fetches where it made 20,020, 37.2 M instructions and 4.9 ms where it took 58.7 M and 12.8 ms (2,047 buffer hits against 22,083); the b-tree on `(s)` takes 13.0 M and 2.0 ms and reads 83 blocks cold where the array reads 302 (the strings of the keys). A page that is not all-visible (rows updated since the last `VACUUM`) and the rows of the write buffer are confirmed as before.
- Improved: pg_sift: an `OFFSET` over the sorted array of the key of a `key_cut` index does not confirm in the heap the entries it throws away: a segment cut by key has no unit per superblock, so no entry of it was exact and every skipped entry cost a heap fetch (about 1,800 instructions and one heap page when the heap does not follow the key); the entry is now exact when its heap page is all-visible (looked up in the map by block) and its row is not among the dead rows of the segment's units (listed once, from the units that have any). `SELECT id, ts FROM t ORDER BY ts LIMIT 20 OFFSET 5000` over 100,000 / 300,000 / 1,000,000 rows (index `key_cut = 'customer_id'`, scattered timestamps, heap not in order of the column), 5,048 / 5,052 / 5,049 buffer pins, 9.2 / 9.4 / 9.2 M instructions and 0.93 / 1.11 / 3.04 ms before, 54 / 65 / 82 pins, 3.4 / 3.5 / 3.4 M and 0.45 / 0.43 / 0.44 ms after (the b-tree on `(ts)` 5,035 / 5,036 / 5,036 pins, 8.7 M, 0.95 / 0.92 / 1.71 ms); `OFFSET 50000` 89.5 / 91.5 / 89.6 M and 11.0 / 12.5 / 15.9 ms before, 31.3 / 32.5 / 31.3 M and 3.9 / 7.2 / 6.1 ms after (the b-tree 85.7 M, 11.8 to 15.2 ms, 50,158 pins). A page that is not all-visible and the rows of the write buffer are confirmed as before.
- Improved: pg_sift: a page of a list that excludes a scope with `NOT EXISTS (SELECT .. WHERE k = f.k)` (an anti join on a `NOT NULL` integer column of the scanned table) is served by the top-N scan too, the keys of the sub-select being the conjunct `f.k NOT IN (keys)`: 300,000 events, a page of 3 with a list of 20 and a `NOT EXISTS` scope: 0.38 ms and 292 blocks where the normal plan took 3.4 ms and 1,933; at 1,000,000 events 0.68 ms and 576 blocks against 47.7 ms and 6,361. A `NOT IN (SELECT ..)` is a filter of PostgreSQL's plan and keeps it.
- Improved: pg_sift: a page of a list (`ORDER BY .. LIMIT n [OFFSET m]` over a table with a column index) is served by the top-N scan when its keys come from a plan (`k IN (SELECT coalesce(a.k, g.k) FROM a LEFT JOIN g ON ..)`, an `IN` over a join, `EXISTS`), when the scope is written on the column of an inner-joined lookup table (`d.id IN (..)` with `d.id = f.k`), and when the select list reads columns of a table joined by its unique key: the units are walked from the end of the order and the scan stops at the page, taking the lookup columns from the rows it loaded, so the blocks read no longer grow with the rows that match (1,000,000 events, a page of 3 over 20,000 matching rows: 1.2 ms, 18 M instructions and 132 blocks, where the normal plan took 32 ms, 192 M and 59,150). `sift.explain_why` names the join or the column that keeps the normal plan.
- Improved: pg_sift: the aggregate scan takes workers where they gain and keeps one process where they do not, priced from the measured start (3 ms and 0.5 ms for each worker), the leader scanning as one of the participants, and the leader's merge of the workers' tables (50 units a group): a scalar aggregate over 600,000 rows (7.4 ms alone, 6.7 ms with 2 workers) and over 1,000,000 rows (12.3 ms alone, 8.2 ms with 3 workers) now runs with workers where it ran alone, a group by key of 300,000 rows or fewer stays in one process, and the plan is the faster of the two in 87 of 88 cells of the measured classes (72 of 80 before). A `parallel_setup_cost` of 0 still asks for the workers.
- Improved: pg_sift: a `count(*)` of one text equality (`WHERE label = 'x'`) on a `sift_col` index whose dictionary for the column holds no row counts (a column of few strings, where `count_columns` leaves them off) is answered by the groups of units (superunits), where it read every unit: 1,000,000 rows, 217 units, 150 values, cold 168 blocks to 2 and warm 0.87 to 0.33 ms (a b-tree on the column: 22 blocks, 0.52 ms).
- Improved: pg_sift: a ranked search scores a block only on the pages where the terms present there can still beat the k-th best score: over 1,000,000 articles a ranked `OR` runs 18% fewer instructions (29% fewer blocks scored), a mixed set of searches 14%, a search with a filter or a join 7%, and `AND`, phrase and `count(*)` the same as before; the answers and the index are the same
- Added: bench: ten cases of the statements that list, queue, ranking, search and write endpoints send (scope_sublink_count, page_newest_lateral, page_since_cursor, page_deep_offset, scope_exists_any_array, search_scope_filter, flags_or_null_range, time_event_buckets, live_counts_by_dim, upsert_returning_conflict): a membership sub-select with a `LEFT JOIN`, a newest-first page with lateral sub-selects, a cursor page, an `OFFSET` page, a per-row `EXISTS` over `= ANY(ARRAY(sub-select))`, a ranked text search with a membership filter, `OR` of a flag and `IS NOT NULL`, `date_bin` buckets over `UNION ALL` arms, conditional sums per dimension and `INSERT .. ON CONFLICT .. RETURNING`; each asserts the plan the planner shows (the scan or the normal plan it falls back to) and compares every answer with the normal plan.
- Added: bench: three cases of the statements that a price-history report and a usage ledger send (series_snapshot_windows, ledger_rollup_grain, count_key_outside_index): the newest snapshot of each series with `lag`, `first_value` and `row_number` windows over two aggregate arms, a usage ledger summed to a day or month label with `to_char(date_trunc(..))`, and `count(id)` of a primary key the column index does not hold; each asserts the plan the planner shows and compares every answer with the normal plan.
- Improved: pg_sift: the rewrite of every segment of a sift_col index chooses again, from the data, what a build chose and kept in an option of its own (`key_cut`, `order_array_columns`, `point_tid_columns`): an index built on a table in the order of its tenants and rewritten after rows of every tenant were appended is 9.94 MB where the rewrite kept the build's choices (13.41 MB; a `REINDEX` 9.75 MB), 300,000 rows after six cycles of 5% updated, 2.5% deleted and 2.5% inserted, with 9.92 MB of WAL for 13.43 MB and 0.27 s more CPU a 100,000 rows; the rows are held in `maintenance_work_mem` (about 350 bytes a row) and the cut is left to a `REINDEX` when they do not fit. A merge of segments that do not fit no longer counts a segment twice against `maintenance_work_mem`.
- Improved: pg_sift: the sorted arrays that the build chooses (`order_array_columns = 'auto: a, b'`, every integer, date and timestamp column of scattered values) keep the whole order of a segment or only its ends, by one rule from the zones of the units (column format minor 40 for the ends, which 3.7 and older refuse; a list the user wrote keeps complete arrays): a scan with a bound in the middle of the values reads the units whose zone holds the bound (whatever key the units are cut by: a range across the keys reads them all), and the whole order pays when those cold blocks, at 64 kB of index each, outweigh its bytes. On `facts` of 5,000,000 rows random timestamps (and a random integer) keep the whole order (index 45.7 MB, against 20.5 with the ends: `>= middle ORDER BY LIMIT 20` 49 blocks / 0.35 ms against 2,529 / 9.3, count of a range 33 / 0.42 against 2,531 / 7.2), the same column in an index cut by a tenant of 2,500 rows keeps the whole order too (an index of the ends there was 37% smaller and read 898 cold blocks and 3.8 ms for a range across the tenants against 30 and 0.14 ms at 1,000,000 rows), a column in insertion order gets none, and `ORDER BY LIMIT 20` and `>= recent ORDER BY DESC LIMIT 20` read the same 20 to 50 blocks as before in every case; `CREATE INDEX` names the choice of each column and why in a NOTICE (the figures at `debug1`). The point directory auto choice no longer skips the columns that the build chose for an array.
- Improved: pg_sift: a merge of segments of a `sift` (BM25) index (`sift.merge`, the maintenance worker, `VACUUM`) is written by the backend and up to `max_parallel_maintenance_workers` workers, each from the dictionaries and postings of the input segments themselves: the terms are cut into ranges of about the same bytes (a sample of every dictionary), each participant writes the postings of the ranges it takes and the backend the docmap and the ranges in term order, the segment a single process writes, byte for byte; the merge no longer dumps every input segment to a run file first, which is why one worker already takes less time. Promote and merge at `merge_fanin = 4`, 1,000,000 documents of 12 terms from a vocabulary of 200,000 in 40 batches of 25,000, medians of 3 interleaved runs: the merge of 4 x 25,000 documents took 226 ms and takes 141 ms (1 worker), 114 ms (2) and 72 ms (4); the 12 merges of the cycle 4,981 ms to 2,846, 2,167 and 1,463 ms, the cycle 8,694 ms to 6,501, 6,210 and 5,027 ms (the merge's share of promote plus merge from 57% to 44, 35 and 29%); an index of few distinct terms (1,122), where the merges are small, 1,432 to 770, 664 and 761 ms of the cycle (a merge of 4 x 25,000 documents 28 ms to 24, 27 and 25 ms). The merge costs more CPU time for the same instructions (4 x 25,000 documents: 229 ms of CPU to 263, 292 and 287 ms; the instructions fall 10%, the dump is gone; a cycle 8.8 s of CPU to 9.1, 9.9 and 9.6 s) and writes more, smaller temporary files (the ranges, two a range: 204 files over the cycle to 348, 444 and 636; 334 MB to 359 MB); the WAL is the same. A merge of segments that get their positions from the heap (an index written by 3.7.0), of segments too small to cut and `max_parallel_maintenance_workers = 0` are merged by one process as before.
- Improved: pg_sift: the participants of a parallel `CREATE INDEX` or `REINDEX` of a `sift` (BM25) index write the segment together: the terms of the runs are cut into ranges of about the same bytes (the runs keep a sample of their terms every 64 kB), each participant writes the postings of the ranges it takes and one record for each term, and the leader writes the docmap and appends the ranges in term order, where it merged every run and encoded every posting alone (1,000,000 documents, 4 workers: 217 to 154 ms with 1,122 distinct terms, 1,243 to 871 ms with 200,000; 8 workers: 208 to 113 and 1,232 to 620 ms; 2 workers: 258 to 224 and 1,402 to 1,033 ms; a serial build is unchanged; the same WAL and index bytes, 0.4 to 3% more instructions; CPU time +19% (1,122 terms, 4 workers, two builds on a loaded host; +4.5% interleaved on a quiet one) and +2% (200,000 terms), the same work on more cores at a lower IPC; 8 temporary files become 48, 16.0 to 21.5 MB (+34%) and 84.8 to 133.4 MB (+57%)). The index is the one a serial build writes, byte for byte. No format change.
- Improved: pg_sift: while a physical standby streams, `VACUUM` of a `sift_col` index moves the segments that lie above the end the file can have into the free runs below it, with no exclusive lock, and the online cut takes the free end: an index a merge left at 2.1 times its live bytes with the last block live (a 200 MB lab index, hot standby with feedback) is at 1.00 after three `VACUUM`s, where the compaction by rewriting the tail segments into the holes stopped at 1.76 after eight and took 7 to 8 s a pass, against 0.13 s. A new run of blocks goes to the lowest free run that holds it, so the file fills from its start and the end stays free for the cut. The pages a killed move copied are orphans of the sweep, not blocks of the segment.
- Improved: pg_sift: the layout and the free space of a `sift_col` file can be read in SQL: `sift.col_extents(index)` lists every run of blocks of a `sift_col` file (segment extents, headers, liveness runs, write buffers, the free list and the free extents, with the ones a snapshot still holds), and `sift.col_space(index)` gives its free space in one row: used, reusable and pending blocks, free extents, the owner of the last used block, where the shrink would end the file and the blocks it has still to move. Both run on a standby (extension version 4.0, `ALTER EXTENSION pg_sift UPDATE`).
- Improved: pg_sift: the `Sort`s above a window over an aggregate scan that the scan's own order serves are dropped from the finished plan (the `ORDER BY` of a CTE that keeps one period of a per-category report, the `Sort` before a second window): the scan emits the order of the Sort above its window (the groups in the order of the sort, then the rest of the window's partition keys and the window's own keys). One-entity per-breakdown report at 1,000,000 events: 2.9 s to 1.75 s (user instructions 34.4 to 20.4 G), the same rows.
- Fixed: pg_sift: planning a statement with a `MATERIALIZED` CTE, a `DISTINCT` sub-select joined to it and a second read of a table with a column index could crash the backend (SIGSEGV): the rows of a lookup were read through the sub-select with the range table of the outer query.
- Improved: pg_sift: the planner hook does less work for each statement that reads a table with a column index: the settings the verdict key depends on are listed once for a backend, and one walk places the single-reference CTEs of a WITH where PostgreSQL, and this pass, walked the query once for each CTE. The plans are the same.
- Fixed: pg_sift: planning a statement with a `MATERIALIZED` CTE, a `DISTINCT` sub-select joined to it and a second read of a table with a column index could crash the backend (SIGSEGV; 3.7.0 and 3.6.3 too): the rows of a lookup were read through the sub-select with the range table of the outer query.
- Improved: pg_sift: the load that `sift.table_bulk_groups` (on by default since 3.3) writes, `COPY` into a `sift_table` of 8 MB of rows or more, keeps its parallel workers for the command: the backend reads the next group while a worker writes the last, and the segments are published as they are written (4 CPUs: 1,000,000 orders 1153 to 980 ms, 500,000 orders 576 to 566 ms against the parallel flush of the previous change, 7 to 10% more table bytes and 5% more WAL than that flush, the same rows). A table with a `PARALLEL UNSAFE` column default or check expression, an identity column, a trigger or an index that is not a plain b-tree, `CREATE TABLE AS` and `REFRESH MATERIALIZED VIEW` keep that flush; `max_parallel_maintenance_workers = 0` keeps the load in the backend.
- Improved: pg_sift: a top-N scan (`ORDER BY <enum column> LIMIT n`) takes an enum column as a sort key, in the order of its type, through the kernels of an integer key: it ranks the OIDs of a unit's values with one pass for each value (up to 16) and reads the unit as it reads an integer (the best value of a segment, the rows that can pass the bar), where it tested every row against the heap. 1,000,000 rows, `ORDER BY lvl LIMIT 10`: 11 ms against 16 ms before, 61 ms for the normal plan.
- Improved: pg_sift: the top-N scan that keeps every row that passes (an `ORDER BY .. LIMIT n OFFSET m` with `n + m` past 100,000) keeps only the best `n + m` rows: its buffer is cut to them (a quickselect) whenever it holds twice that, and a row after the last cut is dropped when it arrives, so the final sort is of `n + m` rows and not of every row that passes (a `LIMIT` under `TOPN_TIES` and a `Tuplesort` that spilled are unchanged). 1,000,000 rows, `ORDER BY amount LIMIT 20 OFFSET 150000`: 89 ms against 193 ms before, 477 ms for the normal plan.
- Improved: pg_sift: `point_tid_columns = 'auto'` no longer picks a column that has a unique b-tree of its own, a primary key for instance: the planner takes the b-tree for an equality and the key lists answer an `IN` list, so the directory was 2.1 to 3.3 bytes a row of size and no speed (1,000,000 rows of a 13-leaf index with a primary key on the column: 38.4 MB with it, 36.3 MB without, an `IN` list of 60 keys 75 cold blocks either way, 0.2 ms warm either way)
- Improved: pg_sift: the cells of the arms of a two-level grouping (a grouping of per-event groups regrouped by category: `avg(avg(score))`, `count(DISTINCT id)`) are inlined into the scan loop (`cell_value`, `cell_add`, `cell_start`): 400,000 events of a jsonb list of objects, the regrouping of the per-event averages executes 207.3 M instructions, 18.7 ms to 194.2 M, 13.6 ms, and the same shape over 500,000 rows 290.5 M to 274.3 M.
- Fixed: pg_sift: a lookup through a CTE of ids whose rows PostgreSQL counts by defaults (a sub-select on an expression key such as `coalesce(a, b)`, a filter on a jsonb path or an expression of a table without statistics, a `HAVING` or a window) was priced as a few keys, like a literal list: the normal plan was priced for one probe of the btree, so an aggregate over the ids lost the comparison with the scan when the CTE held many, where the same ids written as a sub-select took the scan. The keys of such a CTE are now priced as those of a table without statistics are (`sift.agg_param_set_size = 0` turns it off); a CTE of a literal list or of tables filtered by their statistics keeps its count. A scope of one entity that is read through such a CTE takes the scan too.
- Improved: pg_sift: the arms of a `UNION ALL` read by a window or an `ORDER BY` emit their groups in the order of the parent, which the parent merges (`Merge Append`), on PostgreSQL 18 and up: the class of the parent's sort key keeps the member of each arm out of `ec_members` there, so the arms were never offered in that order and the statement sorted all the groups above the `Append`; the window report of a group (two arms, `lag`, 18,683 groups) executes 233.1 M instructions to 212.5 M and, with the json the statement builds above it, 438.8 M to 418.3 M.
- Improved: pg_sift: the build-time choices of a `sift_col` index (the elements of an array column, the sorted arrays, the point directories) size a table that was never vacuumed or analyzed from its pages, as the planner does, instead of reading `reltuples = -1`: an index created right after a load kept the whole-array image of a `text[]` column, and every element test (`labels @> ARRAY['x']`) decoded, hashed and interned each distinct array; 500,000 rows with one label filter and a group by `kind` execute 459.6 M instructions, 112.9 ms to 40.0 M, 5.4 ms.
- Improved: pg_sift: the load that `sift.table_bulk_groups` (on by default since 3.3) writes, `COPY`, `CREATE TABLE AS` and `REFRESH MATERIALIZED VIEW` into a `sift_table` of 16 MB of rows or more, writes the groups of a run in the backend and in up to `max_parallel_maintenance_workers` parallel workers (merged in 3.7.1): 1,000,000 orders, the backend's instructions 22,630 to 17,820 M (19,290 M at the default `maintenance_work_mem`), the cluster's +5.4%, the same WAL and TIDs; `max_parallel_maintenance_workers = 0` keeps the load in the backend.
- Improved: pg_sift: an insert into a table with a language index (`stemmer = 'german'`) spends 66% fewer instructions and takes 45% less time (3,000 single-row inserts into 100,000 documents, 733 to 247 M instructions, 70.9 to 39.3 ms; the GIN index of the same configuration 1,100 M, 105.8 ms): a backend keeps the words its dictionaries answered between statements, for its 4 most recent configurations (2 MB of entries and about 1 MB of table each), where every statement started its memo empty; the name of the configuration is resolved once and again only after a change to the catalog. The memo is dropped when a configuration, a dictionary, a parser or a schema name changes (`lang_memo`).
- Improved: pg_sift: a ranked search whose `ORDER BY` has a key after the score (`ORDER BY sift.score(ctid) DESC, id LIMIT 20`) reads its rows in one search, not two: PostgreSQL sorts the rows tied with the last one by the next key (an Incremental Sort) and asks the scan for the row after the limit, which started a second search of 16 times the limit; the scan now asks for 16 times the limit and one row at once, and the rows are those of the sort of every match (`rank_ties_sort`). Instructions of the top 20 over 100,000, 300,000 and 1,000,000 documents: `body @@> 'qaaaf OR qaatg'` 16.3 to 10.3 M, 38.8 to 22.4 M and 115.7 to 64.3 M (-37%, -42%, -44%; 3.8 to 2.1 ms and 12.6 to 5.1 ms at 300,000 and 1,000,000), with an `INCLUDE` test 22.1 to 16.1 M, 46.2 to 30.5 M and 118.3 to 67.3 M, with a join to a table of regions 19.1 to 13.5 M, 48.3 to 34.0 M and 110.7 to 66.2 M, of two terms 9.5 to 5.0 M, 27.2 to 14.2 M and 51.5 to 42.4 M; a one-term search with many documents tied at the 20th place (0.8% of the documents) 15.6 to 11.7 M and 44.7 to 33.1 M instructions at 100,000 and 300,000. A query without a key after the score asks for the limit as before.
- Improved: pg_sift: `x = ANY (ARRAY(SELECT ..))` on an integer or date column is an IN list of the scan, read when the scan starts like a parameter of the statement and handed to the point directory (or the order array) as a constant list is, where it was a test of every row of every unit: 60 ids over 300,000 rows with a point directory on the column take 0.42 ms and 1.7 million instructions against 41.7 ms and 748 million for PostgreSQL's plan (a b-tree 0.20 ms).
- Improved: pg_sift: `ORDER BY t, id LIMIT n` over a text column with an order array reads the array when the first key is followed by another key or an id tie-break, where it read the units, the keys after the first being read from the rows of each group of equal strings (a collation that calls different bytes equal is left to the units when there is a second key): `WHERE t >= 't00100000' ORDER BY t, id LIMIT 20` over 200,000 rows takes 0.19 ms and 0.5 million instructions against 13.1 ms and 186 million for PostgreSQL's plan.
- Changed: pg_sift: `order_layout = 'auto'` (the default) keeps a chunk of an order array compact where that is at least a fifth smaller and it has no run of 8 equal keys, where it was the whole order, so an index can hold both kinds of chunk and the column format minor is 41 only for a segment that has a compact chunk: the index of a microsecond timestamp over 300,000 rows takes 4.09 MB against 4.87 MB for the whole order (a b-tree 6.76 MB), and a range read takes 0.49 ms in 120 blocks against 0.53 ms in 100.
- Improved: pg_sift: a write buffer of any size gives the order of a key in ranges that are sorted when a read touches them, where it was sorted whole or not at all and the sort cost more than the units: `ORDER BY created_at OFFSET 30000 LIMIT 10` over 300,000 rows with 5,000 of them in the write buffer takes 1.54 ms and 13.4 million instructions in 595 blocks against 12.1 ms in 30,077 blocks for a b-tree (PostgreSQL's plan 54.8 ms).
- Improved: pg_sift: `ORDER BY k, id LIMIT n [OFFSET m]` over the order array of k with a condition on another column of the row (the soft-delete flag of a list page, a status) walks the array, where any condition outside the key sent the scan to the units: `WHERE deleted_at IS NULL ORDER BY k, id LIMIT 20` over 120,000 rows with 20% of the heap pages not all-visible takes 0.14 ms in 63 blocks against 9.3 ms in 651 blocks for PostgreSQL's plan.
- Improved: pg_sift: the `OFFSET` skip over the order array of a key goes on past the entries whose pages are not all-visible, where it stopped at the first entry a unit could not vouch for and a table with a few percent of dirty pages lost the skip (and, after a `VACUUM` that followed updates, past the entries it had already counted): `ORDER BY created_at OFFSET 30000 LIMIT 10` over 300,000 rows after a `DELETE` of 3%, an `UPDATE` of 3% and a `VACUUM` takes 23.4 million instructions where it took 307 million (a b-tree 42.6 million, PostgreSQL's plan 287 million).
- Improved: pg_sift: the equality, range or prefix of a string of a column that is not the sort key (`WHERE g = x AND src = 'web' ORDER BY k DESC LIMIT n`) is tested on the heap row of each entry of the order-array walk, under the collation of the comparison, as a status or a flag is, where the query went to the units: a `LATERAL` top-1 with that filter for each of the 100 entities of a group over 300,000 events takes 1.47 ms and 10.6 million instructions where it took 7.93 ms and 54.8 million (a b-tree on the entity and the time 1.71 ms and 5.2 million; no index 1,121 ms).
- Improved: pg_sift: `DISTINCT ON` and `row_number() = 1` over an `ORDER BY` that has no unique column keep the first row of a tie in the scan, where the normal plan was kept because the scan wanted a unique index to hold the sort columns (SQL leaves the row of a tie unspecified): the latest row of each of the 100 entities of a group over 300,000 events takes 2.30 ms and 14.6 million instructions in 88 blocks against 18.4 ms, 180 million and 2,220 blocks (a b-tree on the entity and the time: 21.0 ms, the same plan).

## 3.7.1 - 2026-10-31
- Fixed: pg_sift: a page of a list (`ORDER BY .. LIMIT`) with a join to a table or an `IN` sub-select, whose rows tie on the first sort key beyond the statistics (the top-N scan starts again with a larger heap), ended the server process: the scan that starts again tested the conjuncts of the keys that the first start made, without their state, and a sub-select read for it a second time came back empty.

- Fixed: pg_sift: an index with `positions = on` (the default) lost its word positions at the first merge that followed a promotion of the write buffer (3.7.0 and before): the segment a promotion wrote had no positions and a merge keeps them only when every segment it merges has them, so every phrase statement of the index was decided by heap rechecks (the index 56% of the size of a fresh build, a ranked phrase +17.7% instructions, heap checks 2,505 to 23,855 over 300 statements). A promotion now reads the text of its documents from the heap and writes the position records of its terms (114 to 137 ms against 57 to 72 for 10,000 documents, in the maintenance worker; the INSERT statement is unchanged), and a merge that finds a segment without positions analyzes its documents again from the heap (+0.45 s per 100,000 documents of such a segment, once), so an index that already lost them has them back at its next merge, with no `REINDEX`. No format change. Tests: `phrase_churn`, `merge_heal`, `merge_parts`.
- Fixed: pg_sift: planning a statement with a `MATERIALIZED` CTE, a `DISTINCT` sub-select joined to it and a second read of a table with a column index could crash the backend (SIGSEGV; 3.7.0 and 3.6.3 too): the rows of a lookup were read through the sub-select with the range table of the outer query.
- Fixed: pg_sift: a scan of a `sift_col` index with an `IN` list of the keys of a scope and an equality on one of them tests the 4,650 keys of the list for every unit and buffered row (6 ms a scan against 0.8 ms for the equality alone, the same rows and blocks) and priced the plan for the list's scans: the list is cut to the constants the equality holds (none left means no row) and the estimate takes the fewest keys.
- Fixed: pg_sift: the rows of the write buffers of a `sift_col` index were looked up by the first condition the scan found, which is the `IN` list when the planner puts it first, so every buffered row of every key of the list went to the heap to be dropped there (85,693 rows and 4,652 heap blocks for 3,455 matches, with 81,586 rows in the buffer); they are looked up by the condition with the fewest values (4,757 rows, 1,149 heap blocks) and the heap checks the others as before.
- Fixed: pg_sift: a lookup of the write buffers by an `IN` list that keeps more than a quarter of their rows narrowed nothing and sent every other condition to the heap (119,086 rows and 3,959 heap blocks for 37,500 matches, with 81,586 rows in the buffer); the buffers' batches now test every condition (50,000 rows, 771 heap blocks). A list with an equality keeps the lookup.

## 3.7.0 - 2026-10-30

- Improved: pg_sift: the fetch scans of a `sift_table` (index scans, bitmap, UPDATE, DELETE by TID) keep 256 superblocks and 64 MB per table, where they kept 32 and 8 MB, so the UPDATEs of a table that has more superblocks than that no longer parse the heads of a superblock again on every fetch: the 40,000 UPDATEs by key of `tbl_update_keys` on 1,000,000 rows take 3,538 M instructions to 2,381 M once the write buffer is settled (300,000 rows 2,320 M to 2,336 M, 600,000 rows 2,803 M to 2,336 M: the loop no longer grows with the table; the heap twin 4.2 G), and 5,621 M to 4,451 M with the 166,000 rows the case leaves in the write buffer, whose promotion is the rest of the growth. Test: `scripts/instr-budget-fetch.sh` (836 M to 691 M callgrind instructions, budget 760 M).
- Improved: pg_sift: a `LEFT JOIN LATERAL (.. LIMIT 1)` whose columns nothing else in the statement reads is not run: a left join to a sub-select of at most one row keeps each row of its left side once, so the planner hook drops it before planning (not with `FETCH FIRST .. WITH TIES`, a volatile function, a row lock or a `DELETE`/`UPDATE`; not inside a `security_barrier` view). A grouped statement over a 1,000,000-row table with a mapping lookup that the select list never reads: 81.2 to 2.4 ms with 300,000 distinct lookup keys (4.5 to 2.5 ms with 2,000), 300,000 rows 55.6 to 1.1 ms (1.9 to 0.8 ms)
- Improved: pg_sift: the rows that own a passing element are found by a doubling search from the row before, not by a binary search over all rows for each element: `flat_lateral_key_lookup` (the scores of 100 members, 500,000 rows) executes 369.4 M instructions to 244.2 M (-34%; 30.1 to 26.4 ms; the same 310 blocks, 137 read cold, 1.07 MB), the same at 300,000 and 1,000,000 orders; the execution ceiling of the class is 280 M (was 424.9).
- Improved: pg_sift: the search that adds the members of HOT chains to its matches (the count of a BM25 term on a `sift_table` with HOT updates) walks the tombstone log once from the first matched page to the last instead of bisecting it for each page: after one round of 2,000 updates and a `VACUUM` the count of 1,000,000 rows takes 35.70 M instructions, 37.74 M before (HOT updates off: 33.54 M), and 11.06 M at 300,000 rows, 11.63 M before (10.55 M). The test counts the lookups of one page alone in the log (`sift._test_tbl_hot_probes()`: 0, one for each matched page before).
- Tests: the 3.2 downgrade of `compat_matrix` says what is true of a BM25 index at segment format 4 (metapage minor 1.4): `scripts/upgrade-check.sh --to 3.2` lists it as a blocker, the 3.2 library refuses it with the format error and the `REINDEX` hint instead of an answer, and the `REINDEX` by the 3.2 library is the way back (it then verifies and answers as before); docs/UPGRADING.md section 4 says so.
- Improved: pg_sift: the load that `sift.table_bulk_groups` (on by default since 3.3) writes, `COPY`, `CREATE TABLE AS` and `REFRESH MATERIALIZED VIEW` into a `sift_table`, writes the groups of a run (the transposition of the rows, the units, the dictionaries, the pre-aggregates, zstd) in the backend and in up to `max_parallel_maintenance_workers` parallel workers (3 at most, while the groups fit `maintenance_work_mem`: 2 groups at its default of 64 MB): 1,000,000 orders of 9 columns, the instructions of the backend 22.6 G to 17.8 G (-21%; 19.3 G, -15%, at the default `maintenance_work_mem`), the time of the COPY 2.1 s to 1.55 s, of the cluster 22.6 G to 23.9 G (+5%), the same WAL, the same TIDs; `max_parallel_maintenance_workers = 0` keeps the load in the backend.
- Improved: pg_sift: grouping sets that all lie inside one of them (`ROLLUP`, `CUBE`, `GROUPING SETS ((a, b), (a), (b), ())`) are summed from the groups of the finest set: the scan hashes a row and updates the aggregates once instead of once per set, and the other sets add the finest set's groups with the state merge of the workers' tables (counts, integer and numeric sums and averages, min, max; a float sum or a `count(DISTINCT)` keeps a pass per set). `grouping_sets_report` over 1,000,000 orders 394 M instructions to 126.5 M (300,000 orders 129 M to 49.7 M), same blocks read (456 cold, 3.56 MB); `EXPLAIN ANALYZE` shows `Sets Summed From One`.
- Improved: pg_sift: the position stream of a BM25 term is coded in blocks of 32 postings (segment format 4, metapage minor 1.4): the first positions at the width of the largest, the numbers of occurrences in Elias gamma and the gaps in Rice codes, a posting read alone. On 218,058 Wikipedia articles the index goes from 136,871,936 to 124,452,864 bytes (-9.07%; the position records 75,403,628 to 63,055,190, -16.4%), a phrase `count(*)` of three words takes 9,357.8 k instructions to 4,311.8 k (-53.9%) and the top 10 15,735.9 k to 12,808.3 k (-18.6%), two common words 384,343 k to 103,266 k and 28,599 k to 24,555 k, the same answers, and its cold reads 640 to 476 blocks; a merge costs 1.8% to 4.2% more instructions (22,474 M to 22,990 M against 22,064 M to 22,179 M) and `CREATE INDEX` 7.9% more (39,654 M to 42,780 M). A merge or `REINDEX` rewrites older segments; 3.6 and older refuse an index with a segment of format 4 (`REINDEX` with the older library, docs/FORMATS.md).
- Improved: pg_sift: the planner reads the constants of an `IN` list once for a planning, not once for each time the scan prices the list or tests its zones (seven times): a statement with 50,000 literals over 300,000 rows plans 330.1 M instructions to 313.1 M with the scan on (293.8 M with it off) and runs 502.7 M to 485.5 M (-3.4%; `keys_in_wide_50k`).
- Improved: pg_sift: `VACUUM` of a `sift_table` loaded in a few big `INSERT`s no longer keeps the holes the load left: the merged segment lay above holes its own VACUUM released and the file stayed 9 to 41% larger than its live bytes for good (2 batches of 150,000 rows: 2,709 blocks with 1,122 free; 3 batches: 2,098 with 511); the highest pieces of a segment now go down into those holes a round at a time, as they do for a `sift_col` index, and the end is cut: 1,588 blocks with 1 free after the first VACUUM for every load shape. The VACUUM takes +0.35% and +0.04% instructions and writes the pages it moves (+15% WAL for the 2-batch load, +1% for 3 batches). `sift.table_space` and `sift.col_sections` of a `sift_table` nothing was written to return zeros and no rows instead of a block read error.
- Improved: pg_sift: the hash table of an integer group key that the range table does not take (the unique id of an order in an index cut by the tenant) starts with room for the groups the planner expects, at most 4,194,304 slots, where it doubled from 1,024 slots one rehash at a time: the top orders by items of 1,200,000 orders in an index cut by `tenant_id` 555 M to 521 M instructions (429 M without the cut), results equal.
- Improved: pg_sift: the planning of a statement with a long literal list in an `IN` sorts the values once, not once for each time the scan prices the list: a list of 50,000 keys over 300,000 rows plans 375.4 M instructions to 330.1 M with the scan on (293.9 M with it off) and the statement runs 547.9 M to 502.7 M (-8%; `keys_in_wide_50k`); a list of 3,000 plans -2.2%.
- Improved: pg_sift: a scan with workers is priced for the values the leader merges of an ordered-set aggregate (`percentile_*`, `mode()`): the distinct (group, value) pairs, 90 units each of a scan operator, were 1.5 a row. Over 300,000 rows with two percentiles the 2 workers cost the leader 25 ms of merging (`osa_add_n`, `blob_dump_with`) on a scan of 70 ms, 127 ms and 146 ms of CPU against 69 ms and 74 ms for the scan alone; with the old price the planner took the workers whenever `parallel_setup_cost` was low enough, and a column of few values (a grade of 5) stays priced as before (`agg_ordered_workers`).
- Improved: pg_sift: the order of a dictionary that is searched in place (`code[]` of a text or uuid leaf with 1,024 strings or more that a point directory, a key list or `dict_lookup = 'on'` asks for) takes the bits of a code, 15 for 20,000 strings and 20 for a million, where it took 32 (column format minor 39, written under `sift.col_format_minor` 15 or more; a 3.6 library refuses it with the `REINDEX` hint of `sift.upgrade_check`): 300,000 unique uuids 29.08 to 27.44 bytes a row of an index with their point directory (21.5 to 19.9 of the dictionary alone, -7.4%), 300,000 unique md5 strings 29.79 to 28.15, the lookups read the same instructions and the same blocks. Test `col_dict_packed`.
- Fixed: pg_sift: `CREATE INDEX .. USING sift_col (doc)` on a jsonb column whose documents are nearly all distinct failed with `invalid point_tid_columns option: unknown column` (or, with a text path, `could not determine which collation to use for string comparison`) at the default settings: the build chose a path of the document for the point directory and the order array by the statistics of the whole column and named it "doc". A path has no statistics of its own and is no longer chosen by the defaults (`paths` and the options named by hand are unchanged); test `col_point_auto`.
- Improved: pg_sift: a two-level aggregate over a spine of dates takes a `WHERE` that tests `IS NULL` a column the arms all return as NULL (a column of a rollup whose arms have none, which filters nothing): the scan did not take the statement, and PostgreSQL sorted, joined and grouped the 84,504 rows of its arms (the one-entity statement of the four price-series shapes at 1,000,000 events: 199 to 64 ms, 3,964 M to 1,288 M instructions and 30 MB of temporary files to none; the four 145 to 199 ms before, 44 to 64 ms now; most entities 32 to 43 ms before, 17 to 23 ms now).
- Improved: pg_sift: `VACUUM` of a `sift_col` index while a physical standby streams writes the segments that end the file again into the reusable holes below them (the ordinary rewrite, no lock the standby replays), and the online cut gives back the blocks they leave once no snapshot can read them (an index rewritten whole, 3.8 times its live bytes: 1.003 times after the standby's transaction ends; the first VACUUM 0.29 s and 9.6 MB of WAL at 200,000 rows, the replay stalls for no query of the standby).
- Improved: pg_sift: the addresses of a global order (`order_array_columns`) are coded with the bits of the largest offset of the chunk, not the 9 of the point directory, and as gaps inside a run of equal keys where that is smaller (column format minor 38, written under `sift.col_format_minor` 15 or more for a segment that also has a point directory; an index of order arrays alone stays at minor 30 to 32; a 3.6 library refuses minor 38 with the `REINDEX` hint): the array of 1,000,000 rows takes 3.50 to 3.00 bytes a row for a unique integer, 3.58 to 2.60 for 6,000 keys, 1.75 to 1.24 for 8 keys, 7.25 to 6.75 for random timestamps, with the same instructions for `ORDER BY .. LIMIT`, offset and range reads
- Improved: pg_sift: a fuzzy leaf (`term~1`, `term~2`) seeks through the dictionary instead of walking it: the terms under its first character were all decoded and tested; now a prefix more than `k` edits from every prefix of the query (a Levenshtein automaton run on the term as the scan meets it) is skipped with every term that starts with it, by a bisection of the block index. 100 `term~1` / `term~2` statements on 10,000, 25,000 and 50,000 Simple English Wikipedia articles: 9.20M, 14.76M, 21.73M instructions a query before, 5.47M, 7.71M, 9.97M after (-41%, -48%, -54%), 1.9 to 3.3 ms before and 1.1 to 1.8 ms after; the matches are the same (6,000 statements compared: counts and top-20 ids), the blocks read cold are the same; x86-64 (AVX2), PostgreSQL 18, no SIMD code touched.
- Improved: pg_sift: the sample that decides the default `key_cut` is 16 superblocks at least, not 32: `CREATE INDEX` of a 1,000,000-row `sift_col` index takes 16,554.8 M instructions to 13,774.6 M and 14,335.3 M (two runs, -13% to -17%); a table of 300,000 rows or fewer is sampled whole in both and takes the same 6,280 M; the leaf and the size of the index are the same (13,631,488 bytes), the decision text of the tests moves (the sample size and the predicted growth within 2.5 points). Tests `col_key_cut`, `col_key_cut_auto`, `col_key_cut_narrow`.
- Improved: pg_sift: `CREATE INDEX` on a `sift_col` index prices the cut by a key and its units in one pass over the sample instead of two, and sorts the groups of the pre-aggregates of the trial units with the sort of the engine: 6,919.9 M to 6,282.7 M instructions at 300,000 rows (-9.2%), 17,366.6 M to 16,971.1 M at 1,000,000 (-2.3%), 1,771.6 M to 1,752.1 M at 100,000 (-1.1%, the key was not priced there); the decision, the leaf and the size of the index are the same. Test `col_key_cut`.
- Improved: pg_sift: a BM25 `count(*)` over a `sift_table` whose updates were HOT no longer checks every row against the snapshot after `VACUUM`: the log entries of the chains VACUUM keeps are not tombstones for the superblock test (71.2 M to 13.9 M instructions at 300,000 rows with 40,000 updates; the table without HOT updates takes 17.4 M), and the match set of the search is built without a hash entry for every match (before VACUUM 288.7 M to 271.7 M, +4.2% over the table without HOT updates). Test `tbl_hot_sift_index`.
- Improved: pg_sift: the participants of a parallel `CREATE INDEX` or `REINDEX` of a `key_cut` `sift_col` index merge its segments by key range, each a slice, when the output is cut into 8 partitions or more, where one process merged them (3,000,000 rows, 8 workers, 9 slices: wall 35.6 to 24.7 s at the same CPU and WAL; an index of fewer partitions is still merged by one process, where the slices cost 28% more CPU and 63% more WAL at 1,000,000 rows). No format change.
- Improved: pg_sift: the header, the records and the parameters of the point directories of a segment end its dictionary section, next to the footer (column format minor 37, written under `sift.col_format_minor` 15 or more): a lookup reads them with the footer from one page, one block less for each segment (an equality on a nearly unique integer, cold, 300,000 rows in 2 segments: 12.0 blocks against 13.7); a library before 3.7 refuses the segment with the `REINDEX` hint, older segments are read as before.
- Fixed: pg_sift: `VACUUM` of a `sift_table` writes the frozen roots of the HOT chains it keeps even when it drops no other log entry: after one UPDATE statement (chains of a root alone) the roots stayed tombstones with their xid in every other session, and a BM25 `count(*)` on 300,000 rows with 2,000 updated rows checked 74,990 rows against the snapshot where it now checks none. Test `tbl_hot_sift_index`.
- Fixed: pg_sift: with `sift.table_async_promote`, a write buffer closed inside a transaction that was still open stayed unpromoted after the commit until another buffer closed (the worker found rows that were not final and left, and nobody asked again): a backend that reads or deletes from the table (`sift.table_auto_defrag` on) now asks again once the oldest xmin has moved while closed buffers wait. Test `018_tbl_async_promote`.
- Improved: pg_sift: a bulk load of a `sift_table` (COPY, CREATE TABLE AS, REFRESH MATERIALIZED VIEW) writes a run of 8 superblocks as one segment, where it wrote 4, so a load leaves half the segments and the dictionaries the columns repeat in each (1,000,000 orders of 9 columns: 6 segments to 3, 22.24 MB to 19.91 MB, 1,675 ms to 1,360 ms, 21.3 MB of WAL to 19.1 MB; 300,000 orders: 2 segments to 1, 6.83 MB to 6.12 MB). A load holds up to 8 write buffers of rows in the backend, where it held 4 (peak resident memory of the 1,000,000-order load 144 MB to 155 MB at the default `sift.table_write_buffer_limit`).
- Improved: pg_sift: `ORDER BY` an aggregate `LIMIT n` over a group table that spilled drops, in each partition it returns, the groups that rank after the n best of the partitions so far, where the spill returned every group to the sort above (the 10 orders with the most items of 300,000 orders, 231,151 groups, default `work_mem` of 4 MB: 519 M to 361 M instructions, 79 to 50 ms, 208,388 groups dropped; the same 794 temporary blocks read and written; at `work_mem` 64 MB, without a spill, 168 M and 25 ms as before)
- Improved: pg_sift: planning a statement the aggregate scan may answer samples the zones for its conjuncts once, not once per arm it prices (a 3-tenant `IN` list with a `GROUP BY` over 1,000,000 events: 4 samples a statement, 72 K instructions, now 13 K of the 435 K the planner spends).
- Improved: pg_sift: the walk of the pages of a key-cut unit by a filter by heap address decodes 64 pages at a time from the packed directory (search-facets: the orders of a `title @@>` word and status and amount tests per category, cut by `created_at`: 1,000,000 orders 91.8 M to 81.5 M instructions a statement, 300,000 orders 16.5 M to 15.3 M; the same blocks); what is left is about 80 instructions a page the filter names in a unit and 19 a page the unit holds, which a block index or a map from the heap address to the unit would remove
- Improved: pg_sift: a unit's rollup (`rollup_dims`, the rollups of a list) steps over the measures a query does not read by the length each now starts with, where it walked their values (column format minor 36, written under `sift.col_format_minor` 15 or more by a segment that has a rollup; both forms are read): the average of one country and status over a year in 1,000,000 orders 2.28 M to 1.99 M instructions a statement, 0.39 to 0.34 ms; 300,000 orders 1.16 M to 1.09 M; 100,000 orders 1.83 M, as before; the same 232 shared blocks and 11 MB index
- Fixed: pg_sift: a pushed aggregate with a `HAVING` clause reported all its groups to the plan above, where the clause keeps a share of them (PostgreSQL's estimate): PostgreSQL 19 counted the anti join of a `NOT IN` sub-select over such groups as one row and priced the shared CTE plan for it, where the scans are much faster
- Improved: pg_sift: `ORDER BY txt LIMIT n` over a text column with an order array reads the array within a bound on the column (`txt >= 'k'`, `>`, `<`, `<=`, `BETWEEN`, an AND or OR of them, and `LIKE 'abc%'` or `starts_with` under the C collation): the scan finds the rank of the string in each segment by the collation, the first entry by the chunk directory, and reads n entries, where the units evaluated the condition on every distinct string; 500,000 strings of 34 characters, instructions of the backend: `txt >= 'c' ORDER BY txt LIMIT 10` 221 M to 0.48 M (a b-tree 0.19 M), `txt LIKE 'abc%' ORDER BY txt LIMIT 10` 35 M to 0.68 M (0.22 M); no format change (case `order_array_text_range`, test `col_gorder_text_range`).
- Fixed: pg_sift: a wrong result since 3.5.0: a `sift_col` index with `key_cut` (set, or chosen by `auto`) and a global order array (`order_array_columns`, `auto` gives one for dates and timestamps) lost rows through a Bitmap Index Scan and a plain Index Scan when the statement had a range or an equality on the array's column that matched at most 16,384 rows (`d BETWEEN a AND b`, `d >= x AND deleted_at IS NULL`, `d = x`, alone or with other predicates): the range handed the heap the addresses of its rows and named each row's unit by the superblock of its heap page, which a key-cut unit is not, so the units that hold the rows were left unread (35% to 55% of the rows missing in the tests, 25% in a 450,000-row table); `Custom Scan (SiftAgg)`, index-only scans, `ORDER BY .. LIMIT`, a range of more than 16,384 rows and indexes without `key_cut` or without the array were exact. The range now reads the units of a key-cut segment; no format change, no `REINDEX`: install the library and reconnect. Interim: `SET sift.enable_col_scan = off` (PostgreSQL 17 and 18). Test `col_gorder_keyed`
- Improved: pg_sift: the point directory of a leaf (`point_columns`, `point_tid_columns`, and the default for integer columns of nearly unique values) is a packed table of entries by bucket where it is smaller than the slots (column format minor 35, written under `sift.col_format_minor` 15 or more): each row has an entry of the remainder of its key (the key less the smallest, 6 bits for ids as dense as the rows, else 16 bits of the hash) and the code of its row or heap address (20 bits at 1,000,000 rows), and the 4 bytes of the directory of a bucket of 64 entries, so a million random unique ids take 3.6 bytes a row where the slots took 10.7 (14.0 to 7.0 MB of index; 4.7 bytes for random 62 bit keys), a lookup reads the same two blocks and runs the same instructions (0.144 M against 0.142 M), `CREATE INDEX` writes half the WAL, and the default formula prices the real bytes (`sift_col_build_gap` too). A key in more than 1,024 rows keeps the slots and its lists. Segments of the older layouts are read and merged into the new one; a 3.6 library refuses a minor 35 segment with the `REINDEX` hint (`SET sift.col_format_minor = 14`, then `REINDEX`).
- Improved: pg_sift: reading a unit's rollup passes over the measures a query does not read eight bytes at a time, not a byte at a time (the average of one country and status over a year in 1,000,000 orders: 2.66 M to 2.28 M instructions a statement, 0.60 to 0.53 ms; 300,000 orders 1.27 M to 1.17 M, 100,000 orders 1.10 M to 1.09 M)
- Improved: pg_sift: an aggregate over the rollup of the rows (`rollup_dims`) with a range of another column next to the tests of its dims (a date window) answers the units that lie inside the range from their rollup and scans only the units the range cuts, where the range made every unit scan (the average of one country and status over a year in 1,000,000 orders: 467,221 rows scanned to 20,095, 1.62 to 0.60 ms, 6.46 M to 2.66 M instructions, 6.83 M to 2.54 M cycles, 523 to 165 shared blocks; the regress test `agg_preagg_rows` compares the answers of the three ways, case 25)
- Improved: pg_sift: a search or another filter by heap address on a table whose units are cut by a key (`key_cut`) reads the pages of each unit from its packed directory as it goes and decodes the offsets of the pages the filter keeps, where it decoded the whole directory of every unit first (search-facets: the orders of a `title @@>` word and status and amount tests per category, 1,000,000 orders cut by `created_at`: 167.7 M to 89.0 M instructions a statement, 24.5 to 13.9 ms; 300,000 orders 27.9 M to 16.1 M; 100,000 orders 1.98 M, as before; the same 325 shared blocks; the offsets of a page are checked when it is read, the pages at the end of the walk)
- Changed: pg_sift: `VACUUM` of a `sift_col` index while a physical standby streams gives the free end of the file back, where it cut nothing (an `AccessExclusiveLock` makes the replay of a standby wait for every query that holds the index, and `hot_standby_feedback` does not change a lock conflict: 29.7 s of stalled replay for a standby query of 30 s on the table, 119.7 s for 120 s). The cut takes an `ExclusiveLock`, which a standby does not replay, and only the run at the end of the file whose every block is behind the horizon (the oldest snapshot of the server, with the feedback of the standby): a block released after the oldest snapshot stays in the file until it ends, as it stays unusable for a rewrite. 1,000,000 rows, a standby query of 30 s on the table: the file 120 MB for 46 MB of live segments (2.6 times) goes to 46 MB (1.00), the replay stands still 0.1 s, no WAL; a standby without feedback gets the recovery conflict of the reuse of the blocks.
- Improved: pg_sift: a ranked scan with a `LIMIT` whose bounds tie (many matches in every superblock, as a one-letter prefix has) checks the quals the index does not answer on the rows it hands out, not on every match it meets: with no threshold before it holds `k` rows that pass, it fetched and checked 8,000 of 10,900 matches of `k*` for 20 rows of `country_code = 'US'`. It is asked for the hits that `k` rows take through the quals (the planner's selectivity, three standard deviations over) and a search that falls short is followed by one sized from the rate at which the hits passed, not by 16 times as many. Places, 34,193 rows, `filter` class: `prefix-1` 14.3 to 7.6 million instructions a query (-47%), `prefix-2` -6%, `prefix-4` -2%, `prefix-3` +5% (3 of 300 statements whose best hits seldom pass take 3 searches), other classes and the unfiltered classes unchanged; x86-64 (AVX2), PostgreSQL 18. Rows and order are those of the LIMIT (`rank_late_quals` compares 468 statements with the unpruned ranking). The ranker no longer fetches the heap row of every candidate of such a scan (the scan checks visibility and quals on the rows it hands out): `prefix-1` 7.7 to 5.6 million instructions a query and 1,157 to 524 warm blocks (851 without the late quals), `prefix-3` 1.447 to 1.387 million (the +5% is gone), `prefix-2` -1.7%; rows of equal scores may come in another order than before (unspecified in SQL; `filter_pushdown` compares the groups of equal scores).
- A difference now names its path and both values and keeps the plan of each lane; the group variant of the shapes with a count between 10 and 20 has data (entity 99), which showed that their whole-set `delta` depends on the plan in PostgreSQL itself, so the corpus skips `delta` and `scores.delta` of those two shapes (`shape.volatile`).
- Fixed: pg_sift: `REINDEX INDEX CONCURRENTLY` and `CREATE INDEX CONCURRENTLY` of a `sift_col` index under writes cut the file of the new index behind its merged segments where the cut asked once for its `ExclusiveLock` and gave up when a writer held a `RowExclusiveLock` on the index (the executor opens every index of the table, the one that is not ready included, for the length of each statement): it now asks again for up to 5 seconds, and the new index ends near its live bytes (a churn of inserts, updates, soft-deletes, reads and autovacuum on 1,000,000 rows next to a streaming standby: the file after the reindex was 2.2 to 2.3 times the live bytes and is now 1.05 to 1.08 times).
- Improved: pg_sift: an `UPDATE` of a `sift_table` that keeps the columns its indexes read gets no entry in the indexes of the table (a HOT update, as the heap's): its tombstone is flagged, the entries of the old version lead to the new one through the log, index scans and bitmap scans follow the chain, VACUUM collapses it and tells the indexes of a root only when its whole chain is dead, and `CREATE INDEX` makes one entry for a chain. 40,000 updates by primary key of 2,000 rows with two secondary b-trees: 7,152 M to 3,853 M instructions (the heap twin takes 4,168 M), 23.8 to 15.4 MB of WAL, 1.90 M to 0.74 M buffer hits; `sift.table_hot_update = off` returns the old behavior. Updates that change an indexed column are not HOT. A table with a BM25 index takes part: the visibility test of a TID the index hands out follows the chain, and the `Search:` filter of a SiftAgg count adds the members of the chains; 39,980 updates of a non-indexed column of 600,000 rows with a b-tree, the primary key and a BM25 index: 5,107 M to 2,583 M instructions (heap twin 3,331 M), 43.6 to 12.6 MB of WAL, 1.81 M to 0.73 M buffer hits, and a BM25 `count(*)` before VACUUM costs 13% more instructions (308 M against 273 M). Test `tbl_hot_sift_index`. A table with HOT updates cannot be read by an older release (it reads the flag as a plain update and loses the rows an index reaches through a chain).
- Improved: pg_sift: a statement of one query level, with no CTE, sub-select or parameter, whose cheapest normal plan costs under half of the least any scan costs (50 units: a point lookup by key, a keyset page, an `ORDER BY .. LIMIT` on a btree) is no longer offered to the arms of the scan: planning with `sift.enable_agg = on` costs what it costs with it off there (`WHERE id = 424242` aggregate: 78 us planned and run with the scan off, 104 us on, now 58 us against 54 us on an idle host; keyset page 139 against 179 us, now 117 against 110 us). `sift.explain_why` and `sift.agg_force` plan every arm as before; a plan that uses a disabled path, a generic plan and the inner levels of a CTE are not skipped. No setting; plan gate unchanged (99 shapes, 92 on the scan, 0 failures). Test `agg_cheap_normal`.
- Fixed: pg_sift: a bitmap index scan of a `sift_col` index with an equality, a range or a `BETWEEN` on a text column that has an order array (`order_array_columns`, or the default formula) returned too few rows: it took the keys of the array, which are ranks of the strings of a segment, for the codes of the segment's dictionary and handed the heap the wrong addresses (an equality on 9,002 rows: 0 rows where the heap has 4); the array answers the range of a text column in the units instead (`col_gorder_text_bitmap`). Found by the differential test of the text range reader; present since the text arrays of 3.6.
- Fixed: pg_sift: a wrong result since 3.5.0: a `sift_col` index with `key_cut` (set, or chosen by `auto`) and a global order array (`order_array_columns`, `auto` gives one for dates and timestamps) lost rows through a Bitmap Index Scan and a plain Index Scan when the statement had a range or an equality on the array's column that matched at most 16,384 rows (`d BETWEEN a AND b`, `d >= x AND deleted_at IS NULL`, `d = x`, alone or with other predicates): the range handed the heap the addresses of its rows and named each row's unit by the superblock of its heap page, which a key-cut unit is not, so the units that hold the rows were left unread (35% to 55% of the rows missing in the tests, 25% in a 450,000-row table); `Custom Scan (SiftAgg)`, index-only scans, `ORDER BY .. LIMIT`, a range of more than 16,384 rows and indexes without `key_cut` or without the array were exact. The range now reads the units of a key-cut segment; no format change, no `REINDEX`: install the library and reconnect. Interim: `SET sift.enable_col_scan = off` (PostgreSQL 17 and 18). Test `col_gorder_keyed`
- Fixed: pg_sift: a ranked or counted search on a relation the planner proves empty (a `CHECK` constraint with `constraint_exclusion = on`, the contradictory bounds of a `sift.facet_counts` bucket) is no longer costed: the page cost of the scan divided by pages the planner leaves out for such a relation and failed `T <= total_pages` on a server built with assertions (PostgreSQL 19), ending the backend.
- Improved: pg_sift: a shared CTE that yields the rows of an aggregate, a window or a `DISTINCT` (its own, or of a sub-select or CTE it reads that PostgreSQL inlines) is no longer written inline in each reader: it stays one CTE with the scans inside it. The top entities of a group (`scores--group--sort_by-top_entities`: the rows and the ranking read one CTE of grouped scores) ran the aggregates, windows and sorts twice, four scans for two; they run once, two scans, 303 to 173 ms (best of 5, 1,000,000 events, one process). Test `agg_cte_materialized` forces the inline plans and counts the scans (6 where it is now 3).
- Improved: pg_sift: an aggregate over the rollup of the rows (`rollup_dims`) with a range of another column next to the tests of its dims (a date window) answers the units that lie inside the range from their rollup and scans only the units the range cuts, where the range made every unit scan (the average of one country and status over a year in 1,000,000 orders: 467,221 rows scanned to 20,095, 1.62 to 1.27 ms, 6.46 M to 6.22 M instructions, 6.83 M to 4.56 M cycles, 523 to 321 shared blocks, all from shared buffers; the regress test `agg_preagg_rows` compares the answers of the three ways, case 25)
- Improved: pg_sift: the rows a filter keeps out of a sparse selection are read a level at a time (the unpacking of a bit-packed lane, the dictionary and the scale of a decimal once for the rows, not a descent of the encoding tree for each row), where only a run of rows of 16 or more had a cheap read. On 1,000,000 orders `count(*) WHERE amount > 100 AND created_at >= '2025-12-02'` takes 10.1 M instructions (heap table) and 10.7 M (`sift_table`) where it took 12.4 M and 13.0 M (-18%), `avg(amount)` for a status, a country and a year 13.6 M against 14.3 M (-5%), and the search facets of a `sift_table` 13.5 M against 14.6 M (-7%); one backend, same answers
- Improved: pg_sift: the top-N scan keeps up to 8 `ORDER BY` columns in its heap (it was 4, past which every row that passes was sorted, and the planner kept PostgreSQL's plan: up to 8 sort columns in the heap of the top-N scan): `ORDER BY` 5 columns `LIMIT 10` over 1,000,000 orders reads 6 of 54 units and takes 2.8 ms where PostgreSQL's top-N sort of the whole table takes 64 ms (6 columns 3.3 ms against 70 ms); a heap entry is 32 bytes larger. No format change, no setting (`topn_keys` of docs/LIMITS.md). Tests `col_topn_deep_limits`, `col_topn_deep`, `col_topn_text`; cases `topn_many_keys`, `topn_six_keys`.
- Improved: pg_sift: a ranked scan under a join is asked for the rows the join will keep, not for the LIMIT: the other relations' restrictions leave a fraction of the scan's rows (a key join passes it), and the scan searches for `limit / fraction` rows plus three standard deviations of the binomial (at most 4,096) in one search, where it searched for the LIMIT, was drained past it and searched again for 16 times as many hits, each search scoring every block again. The 50,000 article `join` class (a top-10 joined to a table whose restriction keeps 12%) takes 6.41M instead of 9.43M user instructions a query; the rows are those of ranking every match first (regression `rank_join_hint`).
- Fixed: pg_sift: a join of a column of a table with a `sift_col` index with the ranges of a list of constants (`events JOIN periods ON event_date <@ periods.range`, the two periods of a report) is counted from the statistics of the column, not at the constant 0.001 of the product that PostgreSQL uses for every join on a range: two ranges that take a quarter of a table of 30,000 rows were 60 rows where 7,400 came out, and the plan of the statement and the price the scan is compared with were made for those 60. A statement that also filters the table by the same ranges (the usual form of a report) is counted once, not twice.
- Improved: pg_sift: the automatic `key_cut` cuts by a scattered date or timestamp instead of a key whose rows fit one unit (about 5 rows a key) when the timestamp pays more: the key list and the bloom filter of the key column serve an equality, so the units it reads are those that hold the key, not every unit whose zone holds it. On 1,000,000 orders (`customer_id`, `created_at`) the index stays 11.05 MB and `count(*) WHERE amount > 100 AND created_at >= '2025-12-02'` takes 0.73 M instructions (0.25 ms, 63 blocks read cold) where it took 12.35 M (3.70 ms, 1,031 blocks), `avg(amount)` for a status, a country and a year 6.49 M against 14.25 M, the newest 10 rows of a search 8.5 M against 174.9 M, and an equality on the customer 1.7 M against 0.44 M; `key_cut = 'customer_id'` keeps the key
- Improved: pg_sift: `CREATE INDEX` and `REINDEX` on a `sift_table` decode only the columns the index uses (its keys, included columns, expression and predicate) from the column groups, not every column up to the last one it names: the primary key and a BM25 index on the last of 9 columns of 1,000,000 orders take 7,387 M instructions where they took 9,210 M (the BM25 build 791 to 541 ms), with the same WAL (30.4 MB) and the same index (test `tbl_index_build_columns`: late keys, an expression, a partial index and an INCLUDE column against a heap twin).
- Improved: pg_sift: the index scan of a `sift_table` tells the b-tree that a version deleted or updated by a transaction no snapshot can see as running is dead for everyone (`all_dead`), so the entry is marked dead on the first scan that finds it and is not fetched again, where every scan and every uniqueness check of the key fetched and rejected it again until a deletion pass or `VACUUM` removed it: 20 statements, each a transaction updating 2,000 rows by key (3 indexes, 700,000 rows), take 6,874 M instructions where they took 13,486 M, and a lookup of a key updated 3 times reads its one live version (test `tbl_index_dead_hint`: the entries marked dead are counted with `pageinspect`).
- Fixed: pg_sift: `CREATE INDEX` and `REINDEX` of a `key_cut` index with `point_tid_columns = 'off'` no longer say that the option lists the rows of the segment: `off` lists nothing, and the notice for the option now comes only when it names columns.
- Improved: pg_sift: a fetch of a row of a `sift_table` column group by its TID (an index scan, the check of a unique key, `UPDATE` and `DELETE` by key) finds the row by bisecting the page directory of its unit, and keeps that directory (some 20 bytes a page) for the superblock instead of the TIDs of all its rows (10 bytes a row, which 8 MB held for 6 superblocks, so a statement over more of them decoded them all again): 20 statements, each updating 2,000 rows by key over 700,000 rows in 9 superblocks, take 9,850 M instructions where they took 13,486 M (WAL and blocks unchanged; 2,000 updates in one transaction 7,301 M to 7,115 M; test `tbl_fetch_sparse`).
- Fixed: pg_sift: a statement with a shared CTE that reads a table with a column index no longer keeps the lifted rows of that CTE (the floor that prices the scans against the work they replace) in the plan that stays when the scans lose: the plan is the one the planner makes without the library, so a small entity gets the join PostgreSQL chooses (a nested loop where the lifted rows chose a hash join). The statement is planned once more, from a copy, when the scans lost on a plan that had lifted rows; a statement whose inline plan holds no scan is planned as before.
- Fixed: pg_sift: on PostgreSQL 16 and 17 the scan reads a group key that is an expression of the key of an unnested jsonb map (`split_part(key, '_', 1)`) in a grouping sets query below a left join, as it does on 18: the planner hands the expression over as a placeholder with no nulling of its own, which only the group RTE of 18 had been recognized by. The 3.7 corpus pushes 371, 309 and 311 statements of its three states on 17.11 where it pushed 305, 254 and 255 (373, 310 and 312 on 18.6); the shapes of the per-breakdown scores endpoints use the scan again.
- Improved: pg_sift: `ANALYZE` (and the `ANALYZE` of `VACUUM (ANALYZE)`) of a `sift_table` of more than about 120,000 rows builds the heap tuples of a share of the rows of its column groups, not of every row: the sampler keeps 30,000 rows of the ones it is given, and each row the table gives stands for the rows it left out in the row count. `VACUUM (FREEZE, ANALYZE)` of 1,000,000 orders takes 1,528 M instructions where it took 4,164 M (204 ms against 515 ms; a heap table of the same rows: 1,232 M, 199 ms), and the statistics are those of a heap twin (test `tbl_analyze_thin`).
- Improved: pg_sift: the histogram of an ordered-set aggregate over a `float8` argument (`percentile_cont`, `percentile_disc`, `mode`) spreads its slots with a full 64-bit mixer, where one multiply left the keys of prices in long chains of 18.7 probes an insert (1.1 now): the per-category median and 95th percentile of 940,327 item prices of 300,000 orders took 355.5 M instructions (53.0 ms, one backend, the same answers as `sift.enable_agg = off`) and take 167.6 M (34.4 ms); `percentile_select`, whose values have two decimals, 383.8 to 371.4 M
- Fixed: pg_sift: `sift.explain_why` prices the `cte_inline` row against the plan the planner makes without SiftAgg, where it read the shared CTE plan charged for the rows its aggregates lost (a price four orders above any plan that runs), and the arms of the plan with the CTEs inlined are `cte_cost` refusals when that plan lost the choice, where they read as used.
- Fixed: pg_sift: the `NOTICE` of an automatic `key_cut` and `sift.key_cut_decision` predict the units of 4,096 rows into which a key of 8,192 rows or more is cut when the rows of a key are sorted by a date: for 1,200,000 rows of three leaves it said +33.5% for an index that grew +141%. The date is the second key only when the prediction is within 5% of the cut by the key alone, so a narrow index is cut by the key alone (+52.5%, predicted +36%) and an index with wide rows keeps the date (+8.2%, predicted +12.2%); the widths of the columns the sample does not read come from the statistics of the table (a 64-byte text leaf was priced at one byte).
- Improved: pg_sift: a unit number that is not one of its segment's units (or an SB below the segment's first) now fails with `Unit N of M (SB .., the segment's SBs start at ..) is not a unit of this segment.` instead of a read of the liveness run at a wild offset (`Liveness byte .. lies past .. LIVE pages`), so the next report names the unit and the segment's unit count; nothing changes for a valid segment.
- Improved: pg_sift: the passes that rewrite a query before it is planned (lateral sub-selects, single-reference CTEs, shared CTEs, the `json_build_object` swap) do not walk a list of constants (`k IN (1, 2, ..)`): a statement the scan cannot run, with a list of 10,000 keys (50 KB), plans in 62.5 M instructions with the extension on against 62.0 M off, not 67.7 M (568 to 50 instructions a key); a list with an element that is not a constant is walked as before
- Improved: pg_sift: whether a table has a column index is asked of the catalog once and kept until the table's relcache entry is invalidated, instead of at every walk of the query (about 2,000 asks to plan a statement of 20 CTEs, each opening the table and looking up the access method of every index): the planning of the 64 statements of the analytics corpus takes 0.7 M instructions less at the median and up to 9.6 M less (15 of the 64 statements save more than 1 M; the largest, 46.3 M to 36.9 M), with the aggregate planner on. The answer follows `CREATE INDEX`, `DROP INDEX` and their rollback; `agg_source_cache` counts the asks and checks it.
- Improved: pg_sift: a correlated child of the scan (`LEFT JOIN LATERAL (SELECT .. LIMIT 1)` into a mapping table) that runs for each of thousands of distinct inputs asks the OS for the rest of an index or heap it has read a sixteenth of, in block order, once its runs take 25 microseconds or more (`Child Blocks Prefetched` in EXPLAIN; no setting): 25,000 distinct probes of an index of 14,600 pages, cold on an NVMe device, 0.95 to 2.7 s down to 0.55 to 0.6 s with the I/O wait 0.72 to 2.3 s down to 0.32 s; a warm run and tmpfs never trigger it.
- Fixed: pg_sift: a flattened lateral sub-select stays in the scan when the query also reads another table with a column index only through the key lookups of a CTE, also when that CTE has one more filter on the table (a type, a city) and the planner reads it as an input of a merge join: the join is priced for the few rows it reads of it, and the flat plan was set against the plan as written because the whole-table index scan made up over a quarter of its cost (three-quarters in a 4,000-entity reduction). The scan of a merge join's input now counts for at most what the join costs over its other input. Reports of that shape over 500,000 to 4.8 million rows: 713 ms to 49 ms, 2.3 s to 197 ms, 9.8 s to 730 ms.
- Fixed: pg_sift: `sift.explain_why` names the arm that made the plan: a top-N whose later sort keys the scan does not hold (a Sort above it) is reported as used, not refused for its cost, and a bare `count(*)` of a search planned as `SiftCount` reports the aggregate scan as the arm that lost.
- Improved: pg_sift: a key of 8,192 rows or more that `key_cut` sorts by a second key (`key_cut = 'tenant_id, day'`, or the date the default adds) is cut into units of 4,096 rows, the last one ending with the key, instead of the rows of a superblock (up to 32,768), so a window of the second key reads the few units that hold it: 12 keys of 100,000 rows, two one-month windows over 8 years, after a merge: 24 of 288 units read and 98,304 rows scanned (12 of 37 and 411,872 before), 3.7 ms to 1.0 ms warm, 176 to 75 blocks read cold; the price is the size, +41% for an index of three leaves (+13% with units of 16,384 rows; the same size with an `id` leaf and 100-byte rows), which `key_cut = 'tenant_id'` avoids; keys under 8,192 rows keep their units; no format change, `REINDEX`, a merge and a promotion cut the new units.
- Improved: pg_sift: the cost estimates of a `sift_col` index sort the constants of an IN list once for a planning, not once for every index path (and for every plan the planner hook makes of the statement): a statement with 4,647 keys plans in 50.9 M instructions where it took 55.9 M, 10,000 keys 94.8 M against 106.0 M, 1,000 keys 21.0 M against 22.3 M (with the aggregate planner off 48.4 M, 89.4 M and 20.4 M; PostgreSQL without the index 45 M at 4,647 keys). The keys are the same, so no plan changes; `col_cost_keys_sorted_once` counts the sorts.
- Fixed: pg_sift: a report over one entity (its id a literal in a CTE) on an index cut by that key no longer takes the scan for an entity of a few hundred rows: the planner charged the normal plan, which probes the btree for the key, the rows the table's filters keep (a factor of hundreds of thousands in the plan cost of the case), and the scan, which also loads the lookup tables of the query, lost 2.8 ms against 1.6 ms in the lab; a key that is sure to be one is now charged only the excess of its rows over the mean's, so an entity that owns a third of the table keeps the scan (4.0 ms against 16.5 ms).
- Improved: pg_sift: the cost estimate of a `sift_col` index reads the selectivity of its clauses from the cache the planner keeps in each clause, instead of making it again for every index path and every column statistic it consults: planning a statement with an IN list of 4,647 keys on a table with a `sift_col` index takes 57 M instructions where it took 443 M (PostgreSQL without the index: 48 M), and a list of 100 keys 15.9 M against 23.4 M (14.5 M). The result is the same number, so no plan changes; `col_cost_selectivity_once` counts the estimates, `diff_fuzz.py` has the family `listpage` (the page of a list endpoint) for the differential check.
- Improved: pg_sift: the planner prices the scan of a search by the matches it reads (a posting and a matching row each have a price, where a flat one stood), so the price of a word grows with the rows it matches, a top-N opens the match set at the rate of the scan, and a bare `count(*)` over a word is planned as `SiftCount` where that reads less.
- Improved: pg_sift: the chunk table of a key-cut unit comes before its directory (column format minor 34, written under `sift.col_format_minor` 15 or more), so a scan reads the head of the unit (2 kB, it was 4 kB) and the chunks it needs, and the directory only when a page of the unit is not all visible or the TIDs are asked for: 1,000,000 documents of 13 leaves, `key_cut`, cold blocks of a `GROUP BY` over 31 units 192 to 165, of a count over a window of dates 706 to 567 (-20%), of a `GROUP BY` of a low-cardinality leaf over the index 525 to 386 (-26%), 563 to 530 for a superblock index; the instructions of a warm scan and the bytes of the index are the same. The units already written are read as before; 3.6 and earlier refuse the new segments (`REINDEX` at `sift.col_format_minor = 14` writes units by superblock, `sift.upgrade_check` says so). Test `col_unit_head`, `col_enc`, `scripts/old-lib-check.sh keyed`.
- Improved: pg_sift: a scan of a `sift_col` index reads the first 4 kB of a unit (its row directory and chunk table) instead of 24 kB, and more only when the directory ends beyond (a unit of sparse pages, a key-cut unit of many rows), so a cold scan of a narrow set of leaves reads 17% to 49% fewer blocks (1,000,000 rows of an 8-leaf index: `GROUP BY` 1,214 to 816, `count(DISTINCT)` 909 to 650, a range of a measure 1,131 to 733); the warm scans run the same instructions; no format change, and the units already written are read the same way.
- Improved: pg_sift: the table of a point directory (`point_columns`, `point_tid_columns`, and the default for integer columns of nearly unique values) has `ceil(keys * 4 / 3)` slots instead of the next power of two, 10.7 bytes a key instead of 10.7 to 21.3 (400,000 unique keys: 4.27 MB instead of 8.39 MB; the index of a table of 400,000 rows and a jsonb column grows 90% instead of 180% with the directory, rounded to 10 by test col_point_auto); column format minor 33, written at `sift.col_format_minor` 15 or more, and read from the segments of every older minor; `REINDEX`, a merge and a VACUUM rewrite write it, and 3.6 and older refuse it (`SET sift.col_format_minor = 14` and `REINDEX` write the power of two; `sift.upgrade_check('3.6')` says so). The growth the NOTICE of the default choice predicts counts 11.5 bytes a row for the directory (it was 16.5).
- Improved: pg_sift: a statement with a flattened lateral sub-select (`INNER JOIN LATERAL (SELECT .. FROM jsonb_each(doc) r LEFT JOIN m ON ..) s`) stays in the scan when the query also reads another table with a column index only through the key lookups of a CTE (the members of a group): that read counted as an ordinary plan, so the flat plan was set against the plan as written by their total costs, and the low estimates of the CTE's rows and of the rows of each key made the plan as written the cheaper one. The statement of the case `flat_lateral_key_lookup` (500,000 rows, 100 keys of 3,000) took 580 ms and read 5,741 blocks cold (44.9 MB), now 42 ms and 174 blocks (1.4 MB). Test `agg_flat_key_lookup`.
- Improved: pg_sift: the scan takes the selectivity the planner kept for each comparison of the table (its `RestrictInfo`) and estimates itself only a condition inside an OR, where it estimated every comparison again for each arm of the statement; `sift.stats()` counts the estimates (`qual_selectivity_estimated`). Test `agg_plan_quals_estimated`.
- Improved: pg_sift: the planner validates the statistics of a `sift_col` index against its directory once in a planning, not at every call (the cost estimate of an index asked three times and the source once more, each a read of the metapage and the directory page under share locks): a statement over many partitions plans with fewer reads, and `sift.stats()` counts the validations (`leaf_stats_checked`). Test `agg_plan_validate_once`.
- Fixed: pg_sift: the planner counts the restriction of one column once when it prices the normal plan against the scan (a list of keys that a statement joins and also names in an `IN (SELECT ..)` of the same column (`FROM keys JOIN facts USING (k) WHERE facts.k IN (SELECT k FROM keys)`) multiplied the key ratio of each use): the normal plan was priced at the square of its cost and the scan was chosen over the btree probes. On 200,000 orders the statement took 12.5 ms as a scan (10,125 blocks read cold, 79 MB) and takes 0.95 ms as the normal plan (185 blocks, 1.45 MB). Test `agg_lookup_set_ratio`.
- Improved: pg_sift: a scan with grouping sets (`GROUP BY GROUPING SETS (..)`, `ROLLUP`) under a window or an `ORDER BY` of the query that reads it emits its groups in the order the parent sorts by, as a scan without sets already did, so that no Sort is put above it (a key a set leaves out sorts as NULL); the ranks of a group are packed in one integer when they fit in 64 bits. A daily breakdown with a jsonb dimension (400,000 events, 223,200 groups, a `lag` over the days) ran 2,062 to 2,363 ms and runs 679 to 828 ms at `work_mem` 64 MB; in the 1,000,000-event lab the first Sort of the scores breakdown (224,319 groups, 0.7 s) is gone.
- Improved: pg_sift: the ranks of a jsonb key that the scan emits in the order of the parent (`ORDER BY`, a window) are given from a string of bytes written once for each distinct value, whose memcmp order follows the jsonb comparison, and then checked with the comparator of the pathkey and repaired where they differ, so the order is the comparator's whatever the bytes got wrong. The 37,933 distinct dimensions of a daily breakdown ranked in 237 ms and rank in about 90 ms; the statement of 224,319 groups takes 2.1 s instead of 2.35 s (1,000,000 events), a case of 37,100 dimensions 590 to 777 ms instead of 687 to 852 ms.
- Improved: pg_sift: the aggregates that read the same sub-select or CTE of the FROM list (`FROM (SELECT ... ) q`, a CTE of the body, with the same quals) are one pass over the rows when the child is a plain SELECT: the child gets the identity that a table already had, so `sift_aggspec_fusable` joins the arms ("Arms In One Pass") where each arm scanned it alone. A set operation (`UNION`) child keeps its arms apart. The mapped-notes statements (three groupings of one CTE) run 250 to 201 ms at 1,000,000 events (median of 5, one process); the regress test `agg_cte_materialized` counts the fused nodes (3, 3 and 0 where it was 0, 0, 0).
- Improved: pg_sift: a shared CTE written `MATERIALIZED` is planned inline as well as shared, and the inline plan is kept when the scans cost less than the work of the shared plan, as for a CTE written without the keyword (the rows are the same; PostgreSQL itself refuses to inline a CTE with a volatile function or a write). The notes of 500,000 events (`jsonb_to_recordset`, mapped through a table, read by three aggregates) run 25 ms and read 154 blocks (1.2 MB) cold; the plan without the scan runs 368 ms and reads 14,171 blocks (111 MB). On the 1,000,000 events of the corpus statement the parent of this change already used the scan (173 ms, 2,394 blocks cold) and this change takes it to 133 ms and 2,371 blocks; the figures first published for it (1.9 s, 36,413 blocks) were not reproduced.
- Fixed: pg_sift: the planner no longer inlines a shared CTE when what the scans leave in the inline plan (the copies of the other CTEs that the statement reads more than once, one per reference) costs more than the shared plan did before its unnesting aggregates were charged, by the margin of a totals comparison: the charge makes up for the rows those aggregates lost, not for that rest. A statement of 17 `count(DISTINCT ..)` arms over a CTE set, planned at 1.9 G instructions (24 times the plain planner's 80 M, 520 ms against 13 ms) because the verdict was the inline plan, whose variant the planner plans again for each hit, now plans at 93 M after the first (1.15 times; 9 ms) and keeps the shared plan the plain planner chooses. A statement whose scans replace the work of the shared plan keeps the inline plan. Test `agg_cte_inline_rest`.
- Changed: pg_sift: the inline plan of a statement inlines the shared CTEs whose tables the scans read, not all of them, and the rule that copied only the CTEs that read the table themselves above 8 reads is gone. The inline plan wrote the body of every shared CTE over a column index at each reference, and PostgreSQL planned (and copied, level by level) every copy: the price-series statement of six such CTEs, whose scans use the tables of two, spent 1,983 M instructions (about 150 ms) on each plan the verdict cache served against 28 M for PostgreSQL alone. A plan in full inlines all of them, sees which tables its scans read (the table of the index and those of the lookups its joins make) and plans again with the CTEs that read those tables, when that plan totals no more than 1.25 times the first; the verdict keeps the set, so the plans that follow from it and the ones that check it inline those only: 150 M instructions (6 scans in a plan of 674 lines against 5,969), 13 times fewer, the plan in full that checks a verdict 150 M against 3,832 M, the plan in full that makes one 2.05 G against 3.96 G. The static rule it replaces (copy only the CTEs that read the table themselves above 8 reads) dropped two scans of 42,252 rows each from the one-entity statements (they read a CTE read twice that reads the table through one read four times): 1,000,000 events, one entity, 365 ms before the rule, 400 ms with it and 195 to 220 ms now; a second CTE, 315, 335 and 145 ms.
- Fixed: pg_sift: the planner answers once for each sub-select and CTE whether it reads a column, an aggregate or a CTE of a query above it, and no longer walks the same trees again for each join of each arm of each plan. A statement whose WITH is inside a derived table (a report wrapped in a query that totals or pages it) and reads a CTE of 17 aggregates many times planned for 4.65 s (14.4 G instructions) where the same statement unwrapped planned for 0.55 s: both now plan for 0.54 s, and every later plan of the shape takes the verdict from the cache (9 ms). On an empty scope the 4 statements of the price-series endpoint that took 4.0 to 5.1 s on the first run of the quick differential corpus take 0.45 to 0.63 s, and 11 to 13 ms after. Test `agg_outer_walks_once`.

## 3.6.3 - 2026-10-30

- Fixed: pg_sift: `REINDEX INDEX CONCURRENTLY` and `CREATE INDEX CONCURRENTLY` of a `sift_col` index with `key_cut` while a physical standby streams cut the file of the new index behind the segments the build merged, as the non-concurrent build does, where the file stayed with the free blocks of the participants' segments (1,000,000 rows, 4 workers: 7.6 MB for 2.5 MB of live segments, 3.0x, and now 2.5 MB for 2.5 MB): the new index is not ready until its build ends, so no query reads it and the cut takes an `ExclusiveLock`, which a standby does not replay, where an `AccessExclusiveLock` makes the replay wait for every query of the standby that planned on the table since the index entered the catalog; a `VACUUM` of an index that is ready still does not cut the file while a standby streams.

## 3.6.2 - 2026-10-29

- Fixed: pg_sift: a wrong result since 3.5.0: a `sift_col` index with `key_cut` (set, or chosen by `auto`) and a global order array (`order_array_columns`, `auto` gives one for dates and timestamps) lost rows through a Bitmap Index Scan and a plain Index Scan when the statement had a range or an equality on the array's column that matched at most 16,384 rows (`d BETWEEN a AND b`, `d >= x AND deleted_at IS NULL`, `d = x`, alone or with other predicates): the range handed the heap the addresses of its rows and named each row's unit by the superblock of its heap page, which a key-cut unit is not, so the units that hold the rows were left unread (35% to 55% of the rows missing in the tests, 25% in a 450,000-row table); `Custom Scan (SiftAgg)`, index-only scans, `ORDER BY .. LIMIT`, a range of more than 16,384 rows and indexes without `key_cut` or without the array were exact. The range now reads the units of a key-cut segment; no format change, no `REINDEX`: install the library and reconnect. Interim: `SET sift.enable_col_scan = off` (PostgreSQL 17 and 18). Test `col_gorder_keyed`

## 3.6.1 - 2026-10-29

- Fixed: pg_sift: a statement with a flattened lateral and shared CTEs that the verdict cache holds for another size of request no longer ends the backend: after the plan of the form that won was checked against the verdict of its size band, the planner discarded the context of the form the band's verdict named, which is never planned when the band names the other form (a wild pointer in `plan_flat`, found by the differential corpus of the analytic statements; SIGSEGV in the planner hook on `MemoryContextDelete`).
- Fixed: pg_sift: an out-of-bounds read of four bytes in 3.6.0 when an element predicate that tests two keys is pushed down (`EXISTS` over the elements of an array column, `@>`): the element test indexed the leaves of the layout with the leaf number -1 of an `AND` child that has none (`sift_aggspec_elem_test`, found by the sanitizer run of the deep nightly tier as a heap-buffer-overflow); no wrong result was found, and the test now declines such a predicate. Test `agg_elem_test_leaf`.
- Fixed: pg_sift: a parallel `CREATE INDEX` or `REINDEX` of a `sift_col` index reserves a gap in front of the segments of its participants (about the bytes the statistics say the index takes, a tenth over), merges those segments into it and cuts the file behind it, where the first `VACUUM` rewrote them (3.6.0) and left the file at twice the index for good while a standby streams (VACUUM does not cut the file then, and the output of the rewrite lay after the segments it replaced); the file of a build is made by the transaction, so no query of a standby can hold it: 3,000,000 rows, 13 columns, `key_cut`, 8 workers, a streaming standby, the file after `REINDEX` and the first VACUUM 206 MB for 99 MB of live segments (2.08x) and now 115 MB for 115 MB (1.0x), the first VACUUM 8.9 CPU seconds and 101 MB of WAL and now 0.03 s and none, the `REINDEX` 18.9 CPU seconds and 104 MB of WAL and now 39.3 s and 255 MB (the first `VACUUM` it replaces included, 27.8 s and 205 MB; the rewrite in the build of 3.6.1's first fix took 43.9 s and 325 MB) (each page is written twice, by the participants and by the merge; the gap is the estimate of the statistics, 1.8x the index on this wide table because they overstate its json columns, 202 MB for 115 MB, so the file peaks near 2.8x the index during the build and ends at 1.0x); a gap too small leaves the rest of the output above the segments it replaces, which the space step brings down; a build with no worker launched and `REINDEX CONCURRENTLY` are as before
- Fixed: pg_sift: `EXPLAIN (VERBOSE)` and `EXPLAIN (ANALYZE, VERBOSE)` of a SiftAgg top-N that computes window totals (`SELECT id, count(*) OVER () FROM t WHERE .. ORDER BY k LIMIT n`, the page of a list and its total) raised `ERROR: could not find window clause for winref 1` in 3.6.0 (PostgreSQL prints a window function only under a WindowAgg node): the scan tuple and the node's target list hold the aggregates the scan computes, `Output: id, ts, (count(*))`. Other plans with a window function (a WindowAgg over a SiftAgg aggregate) were not affected. Test `agg_topn_window`.
- Fixed: pg_sift: the planner prices the opening of the partitions of a partitioned table, which grows faster than their number, and chooses the scan for 400 hash partitions whose indexes keep different key tables: the key table of a column adds its measures to the dim of the column in the partitions that keep one (`key_columns = 'auto'` follows the data of each partition, and a column that fits in two partitions of 400 is enough), and the partitions listed other measures for the dim, which made them "not share a layout" and kept the normal plan; the scan now answers with the measures every partition holds. `GROUP BY country` over 400 partitions of 1,000,000 rows takes 8.9 ms and 26.7 MB read cold where the normal plan took 88 ms and 108.4 MB (case `partitions_scan`, test `agg_part_presence`).
- Improved: pg_sift: the planner prices a table that a semi, anti or inner join by an int4 column answers from a byte table of its keys (`k IN (SELECT ..)`, `EXISTS`, a join) as the keys alone (loaded at 10 operators a row, 453 instructions measured, where a table with its dictionary is 20) and the tuples it tests against the byte table of those keys at 5 operators (40 instructions a tuple measured on 985,000 tuples) and not at a hash probe of 86, which it priced alike: the scan of 1,000,000 rows that joins a 550,000-row table costs 25,600 where it cost 77,300 against 32,300 for the normal plan, so it is chosen again (case `keys_semi_load_first`, `keys_subq_500k`, `keys_exists_500k`, `keys_join_500k`): 7 ms and 23.2 MB read cold where the normal plan takes 149 to 160 ms and 84.5 to 85 MB, and 22.7 ms for the first statement of a backend (cache of lookup tables off).
- Improved: pg_sift: a table that a semi, anti or inner join by an int4 column answers from a byte table of its keys (`k IN (SELECT ..)`, `EXISTS`, a join) is loaded without the dictionary of its keys and the chains of its rows, which the first use that looks the table up makes: 550,256 rows read and 500,000 loaded take 35 ms where they took 59 ms, and the scan of 1,000,000 rows that joins it takes 33 ms and 442 million instructions where it took 47 ms and 531 million (the lookup cache off, 40 MB read either way; from the cache 14 ms and 123 million instructions, as before).
- Fixed: pg_sift: the default `point_tid_columns` of `CREATE INDEX` and `REINDEX` no longer lists a date or a timestamp column (an equality on one is rare, its reads are ranges and an order) and predicts the growth of the index from the values it stores, not from the width of the documents of a jsonb column: a 13-column index with paths of a table of 1,000,000 rows said +12% and +11% for two columns and grew 3.0 times (16 to 48 MB, 3.5.3 to 3.6.0), it now lists the id alone and says +44% (33 MB, the id's 13.8 MB); 400,000 rows of three columns, the directory of a nearly unique timestamp and an int8 said +10% and +9%, grew the index 4.7 to 18.8 MB, and now lists the int8 and says +106% (measured +180%); a `REINDEX` decides again, `ALTER INDEX .. SET (point_tid_columns = 'rid')` names a column by hand

## 3.6.0 - 2026-10-29

- Added: pg_sift: `sift.col_dict_shared_cache_kb` sets the size of the ring of parsed dictionaries that backends share (a `sift_col` dictionary of 128 kB or more that one backend parsed is read in place by the others), and 0 turns the sharing off: no segment is made or attached and each backend parses and keeps its own dictionaries, with the same rows. It replaces the size taken from `shared_buffers` (64 MB to 512 MB with no setting): the default is 64 MB for every server, 4 GB at most, and with posix shared memory at most a quarter of `/dev/shm` (the log says when the size is cut). The size is recorded in the shared directory by the backend that makes it, so a backend with another setting uses the ring as it is, and a change takes effect at the next start of the server. The ring is made when the first dictionary is kept, not on the first read of one: a server with no dictionary to share takes a directory of about 50 kB. This memory is in addition to `shared_buffers`, is not reclaimable like the page cache, counts in `/dev/shm` and the shared memory of a container, and gets no huge pages.
- Added: pg_sift: a plain `sift_col` index (`CREATE INDEX .. USING sift_col (r, ip)`, no operator class) returns the rows of a containment of a range (`@>`, `<@`, `&&`, `<<`, `>>`, `&<`, `&>`, `-|-`, `=`) or a network (`<<`, `<<=`, `>>`, `>>=`, `&&`, `=`) with an index scan, from the integer lanes of the column and not a sequential scan: the family `sift.col_ops` has the operators (`ALTER EXTENSION pg_sift UPDATE`; an index built by 3.5 or older with such a column needs the `REINDEX` the lanes ask for), an overflow row and the host bits of an address are decided by PostgreSQL's operator, a range the lanes do not cover (`numrange`) is left to the heap. `WHERE r @> 5000000` over 1,000,000 random ranges returns its rows in 3.12 million instructions and 0.40 ms (3,000,000 rows: 11.2 million, 1.21 ms) where the sequential scan reads 768 million and 81.7 ms (2,310 million, 218 ms), and a GiST 2.65 million and 0.36 ms (6.60 million, 0.76 ms); `r = x` 1.08 million (3,000,000 rows: 1.64 million, a b-tree 0.13 million), `ip <<= net` 19.2 million (92.0 million; a GiST 0.34 and 0.58 million), `ip >>= addr` 30.9 million (107 million; a GiST 0.26 million): the address lane of a column of random addresses has no zone to prune, so keep the GiST for them, or cut an address-only index by its node (`key_cut = 'ip#1'`: 3,000,000 addresses, `ip = x` 1.37 million, `ip <<= net` 1.57 million, `ip >>= addr` 4.29 million instructions, the index 24.6 MB against 26.6 MB). An index on the range alone, where the cut by the start of the range is not chosen (1,000,000 rows), takes 2.59 ms (the sequential scan 57.8 ms, a GiST 0.38 ms). Test `col_contain` compares the rows of every operator against every constant, `=` for every value, with the heap after writes, a merge and a `REINDEX`.
- Added: pg_sift: `order_array_columns` takes a text or varchar column (column format minor 32; `auto` chooses one by the default formula and says so in a NOTICE): `ORDER BY txt [DESC] [NULLS FIRST | LAST] LIMIT n [OFFSET m]` reads n entries of each segment, ranked by the collation of the column, merges them by comparing their strings and fetches n rows of the heap, where the scan compared the strings of every row under the collation (800 instructions a row). In instructions of one backend, 500,000 / 1,000,000 / 3,000,000 rows of 13-byte strings: `ORDER BY txt LIMIT 10` 403 / 806 / 2,429 M to 0.33 / 0.40 / 0.86 M (a b-tree 0.22 / 0.21 / 0.22 M), `DESC` 404 M to 0.36 M at 500,000 rows, `NULLS FIRST` 0.86 M at 3,000,000 rows (a b-tree: a sequential scan and a sort, 5,259 M); the default index of 9 columns (500,000 rows) takes the array of a text column of 13-byte strings: B 117 against C 48, the index +20.8% (26.4 to 31.9 MB) and `CREATE INDEX` +183% instructions under pt_BR (6.32 G to 17.90 G; 4 columns under C +62%: the sort of the strings), `order_array_columns = 'off'` drops it. A sort under another collation than the column's, a condition on the column (`txt >= 'k'`, `LIKE`), a second key, an expression or a string over 255 bytes read the units as before.
- Added: pg_sift: when the build leaves out an eligible text column of `order_array_columns = 'auto'` because the default formula says its cost is over its benefit (a narrow index under a locale), `CREATE INDEX` says so once in a NOTICE that names the column, the benefit and the cost and the `order_array_columns = 'col'` that builds it anyway. Test `col_gorder_text`.
- Added: pg_sift: `@>`, `<@`, `&&`, `<<`, `>>`, `&<`, `&>` and `-|-` of a range (`int4range`, `int8range`, `daterange`, `tsrange`, `tstzrange`) and `<<`, `<<=`, `>>`, `>>=` and `&&` of an `inet` or `cidr` column, against a constant, are answered by a `sift_col` index from integer leaves that the range or the network keeps after its image (the start and end of the set; the node of the block in the tree of prefixes), with zones and the integer kernels, not by the operator on each distinct value: 1,000,000 random ranges `r @> x` 1.5 ms where it was 552 ms (a GiST: 0.15 ms), `ip <<= net` 1.4 ms where it was 380 ms, 0.7 and 0.9 million instructions on a table in range order. The layout follows the column type (column format minor 26): an index built by 3.5 or older with a range, inet or cidr column needs a `REINDEX`; `key_cut` accepts the first lane (`r#1`).
- Added: pg_sift: `doc @> '{"k": 42, "o": {"a": 7}, "tags": ["t5"]}'` on a jsonb attribute of a `sift_col` index is answered from the leaves of the paths it names (a scalar equals its leaf, a nested object is the same of its members, an array is contained by its array leaf), where every document was read: 1,000,000 documents, one scalar member 56.5 ms to 2.0 ms, an array member 65.7 ms to 9.9 ms, a nested object 76.5 ms to 4.7 ms, three members 54.8 ms to 1.6 ms (a GIN `jsonb_path_ops` index of the same table: 1.3, 24 and 26 ms for the first three). A JSON null, a number a leaf cannot hold exactly, an empty object or array and a path without a leaf keep the normal plan (`agg_contains_object`).
- Changed: pg_sift: `VACUUM` rewrites a lone segment of a `sift_col` index when the scans of its dirty units would cost more than the rewrite, or when half of it is dead, not above a fixed 10% dead rows (`dead_rewrite_pct` 0, the default; a percent from 1 keeps the fixed rule). A query reads a unit with dead rows net of them (about 430 instructions a dead row) and by its rows once a tenth of the unit is dead (45 a row); the rewrite re-encodes every live row (900 + 463 * bytes of a row instructions); with 100 scans between two `VACUUM`s a segment of 7 bytes a row with dead rows in every unit is rewritten from about 9% dead rows, as before, one of 75 bytes a row only when half of it is dead, and dead rows kept in a few units are not. 300,000 rows, 13% dead: `VACUUM` of 6 array columns 9,528 against 292 million instructions (spread over every unit) and 9,282 against 100 (the first 13%), a grouped sum 0.5 against 17.2 and 1.8 million meanwhile; 7 bytes a row: 1,354 against 1,356 spread (unchanged), 1,171 against 96 clustered. No format change.
- Changed: pg_sift: `EXPLAIN (ANALYZE)` of `Custom Scan (SiftAgg)`, of the ranked and counted `SiftScan` and of `SiftAggRows` prints its counters in text as a few lines (`Units: 384 read, 64773 pruned (2 by set summary)`, `Rows: 27819 scanned, 183 aggregated, 17 groups`, `Lookup: 4751 rows read, 166 rows loaded, 1 index scans, 4.045 ms to load, 3 semi tables (6 kB)`, `Top-N Run`, `Scan`, `Filter Mask`) in place of one line for each counter (a typical scan printed 15 to 19 lines under the node and now prints 6 to 9); a counter that is zero is left out unless `EXPLAIN (VERBOSE)`, except the ones whose zero is the finding (units read, rows scanned, heap fetches). `FORMAT JSON`, `YAML` and `XML` keep every field with its name and type; a reader of the text lines (`Rows Scanned: 0`) reads `Rows: 0 scanned` or, better, the JSON (docs/COL-PLAN.md, "EXPLAIN in text and in the other formats").
- Changed: pg_sift: the tuples through the arms of a two level aggregate (a `UNION ALL` of aggregates over one scan, 150 operators a tuple and arm, 4,500 instructions a tuple measured on a statement with three arms) are in the price of the scan, serial or parallel, where the serial price left them out and the parallel one scaled itself by their share: the price of a scan with arms rises by what they cost (3% on the statements of the lab corpus, whose tuples the planner counts at the rows of the join), the plan text of all 156 statements and of the public-style sets is the same, and the scan of two few rows (a table of a handful of rows) now loses to the normal plan, which is the cheaper one there.
- Changed: pg_sift: the extension SQL is version 3.6 (`pg_sift--3.6.sql`, `pg_sift--3.5--3.6.sql`, `default_version = '3.6'`): run `ALTER EXTENSION pg_sift UPDATE` in every database. The update adds the operators of the typed classes to the family of the default class `sift.col_ops` (no REINDEX, no new object); the scripts of 3.5 are the released ones again.
- Changed: pg_sift: `VACUUM` of a `sift`, `sift_col` or `sift_table` relation does not cut the free end of its file, nor move segments down for it, while a physical standby streams from the server: the standby replays the `AccessExclusiveLock` of the cut first and waits for every query that holds the relation, up to `max_standby_streaming_delay`, then cancels it (`hot_standby_feedback` does not prevent it), so its replay stalled for as long as a long query on the standby ran (1,000,000 rows, a 45 s query on the standby: the replay of one VACUUM waited 30.1 s and the query was cancelled, against 0.0 s now; a b-tree is never cut and never stalled it). The free blocks stay in the file for the next rewrite and `REINDEX` returns them; without a standby nothing changes.
- Improved: pg_sift: `SELECT count(*) FROM t WHERE arr @> ARRAY['a']` (or `'a' = ANY (arr)`, an integer or text element) over a `sift_col` index whose segments are clean and all-visible is the length of the list of the value in the point directory of each segment, and reads no unit: 1,000,000 rows, 6 array columns, a value in 371 rows and in 557 rows, 4.6 and 5.0 Minstr (2 ms) before against 1.3 Minstr (1 ms) after (GIN 1.4 and 2.1 Minstr, 1 ms; the heap 2,333 Minstr, 242 and 273 ms); cold, 781 and 728 blocks read (6.0 MB) before against 57 and 56 (0.5 MB) after (GIN 373 and 556, the heap 43,436); a unit with a dead row, a page that is not all-visible and the rows of the write buffer are read as before, and a scan with workers (`max_parallel_workers_per_gather`) cuts the segments into morsels and reads them; an index built before keeps its old lists until `REINDEX` or a merge that rewrites the segment.
- Improved: pg_sift: a `sift_col` index lists the rows of an array element (or of a jsonb array's element) up to `max(4,096, rows / 64)` rows of a segment, as the lookup already accepted, not up to `max(1,024, rows / 128)`: `arr @> ARRAY['a', 'b']`, `arr && ARRAY[..]` and `'a' = ANY (arr)` with an element in 1% of 1,000,000 rows (10,866 rows) read the list of the element and no longer test every element of every unit, `arr @> ARRAY['am_3', 'am_100']` 93.8 to 34.4 Minstr (15 to 9 ms; GIN 35.0 Minstr, 7 ms; the heap 2,622 Minstr, 247 ms), `arr @> ARRAY['am_100']` 47.0 to 6.9 Minstr (10 to 2 ms; GIN 64.9 Minstr, 33 ms), `arr && ARRAY[3 elements]` 52.7 to 7.5 Minstr (10 to 2 ms; GIN 74.4 Minstr, 29 ms), at the cost of 2.7% on the index (77.2 to 79.3 MB); cold, the first reads 865 blocks (7.1 MB) before and 881 after against GIN's 4,573 (37.5 MB) and the heap's 43,436 (356 MB); an index built before keeps its lists until `REINDEX`.
- Improved: pg_sift: `ORDER BY ts LIMIT n`, `WHERE ts >= x ORDER BY ts LIMIT n`, `OFFSET`, `max(ts) WHERE ts < x` and the count of a day over a `sift_col` index whose date or timestamp column is scattered over the heap read the global order array of the column whatever the width of the index (a one-column index of a date): `order_array_columns = 'auto'` chooses a date or timestamp column by the default formula (B = 66 for the 25 times fewer instructions of a range with LIMIT, C = 47 for 121% of a 3.7 bytes a row index and 34.5% of its build) instead of a ceiling of a quarter of the index, and no longer writes the sorted run with the array (2.1 bytes a row, 18% of the build, for shapes the array answers). 500,000 / 1,000,000 / 3,000,000 rows with random timestamps, instructions of one backend (a b-tree in brackets): `ts >= x ORDER BY ts LIMIT 10` 9.4 / 18.3 / 30.5 M to 0.38 / 0.39 / 0.42 M (0.19 / 0.21 / 0.20 M), `ORDER BY ts DESC LIMIT 10` 2.83 / 2.80 / 1.94 M to 0.28 / 0.27 / 0.28 M (0.19 / 0.18 / 0.16 M), `OFFSET 1000` 23 / 32 / 46 M to 2.1 M (1.6 M); the index of one timestamp column 3.7 to 8.2 bytes a row (a b-tree 22), CREATE INDEX +34.5%. The planner prices a top-N of the order array by the segments it opens (0.17 M instructions and 44 thousand a segment), so an unbounded `min(ts), max(ts)` stays with the pre-aggregates (0.38 M, was 0.53 M when the array answered). The index of a table whose date column was out of the room before grows; `ALTER INDEX .. SET (order_array_columns = 'off')` and a `REINDEX` drops the array.
- Improved: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index with no `point_columns` and no `point_tid_columns` write to `point_tid_columns`, as `auto: col`, the integer columns of nearly unique values that do not follow the order of the table when the default formula says the directory pays (the equality reads every unit otherwise, 0.17 M instructions and 5 for each row of the table): a 7-column index of 400,000 rows, an equality on the random int8 key 1.9 M instructions to 0.2 M (a b-tree 0.2 M), the index +96% (6.50 to 12.74 MB, predicted +105%), one int8 column of 1,000,000 rows 5.17 M to 0.20 M (b-tree 0.14 M) and a 16-key `IN` list 95.6 M to 0.49 M (b-tree 0.50 M; the directory of rows 0.74 M), `CREATE INDEX` +21% instructions; a one-column index is chosen from about 1,200,000 rows, text and uuid columns are not (their dictionary finds the key); the `NOTICE` names the column with the benefit and the cost, `point_tid_columns = 'off'` keeps the index without a directory
- Improved: pg_sift: a backend reads the shared dictionary of a `sift_col` segment (the ring of 128 kB records the first backend that parses one keeps in the dynamic shared memory registry) where it lies in the ring, not from a copy it makes: the record is pinned while a scan reads it and the ring takes no place a pinned record covers, so a dictionary costs a backend a lookup and no private bytes (the bytes of dictionaries a backend holds after an `ip = x`, a numeric sum and three `url LIKE` statements over 1,000,000 rows: 59 MB, with the copies 41 MB, read in place 12 MB for the backend that parses and 4 MB for each other: 1,898, 1,303 and 128 MB for 32 backends), and the hash table and sorted order a search builds (after 8 searches and 16 tests, counted over all backends) are made once by one backend and written to a new record that carries them, where each backend and statement made them: `count(*) WHERE ip = x/51` over 3,000,000 rows of IPv4 and IPv6 networks in a `key_cut` index takes 2.6 million instructions in the first statement of another backend and 0.50 in the next (1,263 and 900 without the sharing, 123 and 110 with the copies), 2,000,000 rows 2.5 and 0.43 (877 and 518), 1,000,000 rows 2.5 and 0.39 (286 and 24), with the 256 MB ring of a 1 GB `shared_buffers`; on a server at the default 128 MB the 64 MB ring holds the dictionaries of 1,000,000 rows and not the hash tables of 2,000,000 (211 and 209 million) or 3,000,000 (314 and 312). A backend gives the pins back when a transaction ends, a statement fails or the backend is terminated.
- Improved: pg_sift: `CREATE INDEX` of a `sift_col` index with `key_cut` at the default `maintenance_work_mem` of 64MB (a leader and one worker, 16 MB of rows a run) wrote the rows of 10,000,000 rows again while it scanned: the runs of 43,000 rows outgrew the 59 segments of a participant, and 6,000,000 of the rows went through a merge. A batch of rows kept the arrays it grew by doubling (a third of their bytes unused) until its run was written; it is trimmed to its rows when its superblock is done, and the run takes as many rows as the participant's share of the table needs to fit its segments, up to eleven sixteenths of its share of `maintenance_work_mem`. Instructions of the whole build on a lab table (10,000,000 rows, 6 leaves, 5 JSON paths, 8 workers asked for): 484.6 G to 372.1 G (-23%) at 64MB, 16 merges to none, private memory of the processes 87 MB to 94 MB (the largest 42 MB to 38 MB), CPU 43 s to 37 s, the index after `sift.col_merge` the same (206.1 MB to 205.8 MB); at 1GB (8 workers) 373.9 G to 373.8 G, the same 56 runs, private memory 657 MB to 471 MB (the trim, with the same rows in a run); 3,000,000 rows the same instructions and runs at 64MB and 1GB (112.4 G, 18 runs at 1GB; private memory 516 MB to 450 MB).
- Improved: pg_sift: the merge of the segments of a `sift_col` index (`sift.col_maintain`, `sift.col_merge`, the merge a VACUUM starts) copies a run of rows of one segment at a time, a leaf at a time, when the next rows of the other segments come after it and the leaves are integers, floats and dates (no list, no string), where it copied a row and a value at a time: a table of 3,000,000 rows of five numeric leaves and a date, five segments, 12.37 to 10.26 billion instructions (-17.1%); on an arm64 core, 1,000,000 rows, 2.91 to 2.43 billion (-16.4%), 155 to 134 ms. The merged segment has the same rows and size, the same answers and passes `sift.col_verify`. No format change and no option.
- Improved: pg_sift: `sum` and `avg` of an `int2` or `int4` expression of more than one operation (`sum((w + 1) * 3)`, `avg(w * 3 + 1)`, `sum(1 + w * 2)`; one operation of a smallint column and an integer constant is the sums of the units above) are the exact sum of the expression as one program for each row, not a transition function call for each row and sum: the digits of the expression (4 to 9) prove that no row leaves its type, so the errors of PostgreSQL's integer arithmetic are not reproduced where they cannot happen, and an expression that can (`w * 100000`, an `int4` column + 1) keeps the generic aggregate and raises what it raises. 300,000 rows of a smallint column, four such sums in one statement: 23.5 ms (the heap plan, which the planner chose; forced to the scan 31 to 35 ms) to 1.9 ms (1.8 forced); 16 sums `(w + k) * 3` over 1,000,000 rows 244.5 ms (heap) to 19.1 ms. The results, errors included, are those of the heap plan (test `agg_sum_int_expr`).
- Improved: pg_sift: `FROM t, unnest(array_column) e` of a `text[]`, `varchar[]` or integer array column whose elements a `sift_col` index keeps as a list (`paths`, chosen by the build for arrays that differ) is a scan of the list: `GROUP BY e`, `count(*)`, `count(DISTINCT id)`, `e LIKE ..`, `e IS NULL`, `min`/`max`/`sum(e)` and the filters of the other columns run in the SiftAgg scan, where the plan read the table and ran `unnest` on every row. 1,000,000 rows, 6 array columns, one backend: the 20 most frequent elements 8,421 M instructions (1,064 ms) to 82 M (6.6 ms), a filtered group 3,998 M to 304 M, `count(DISTINCT id)` per element 2,265 M to 113 M (299 ms to 12 ms), `LIKE` over the elements 17,550 M to 59 M (1,835 ms to 8.0 ms); heap and GIN 7,645 M to 16,900 M for the same statements, same rows. An `unnest` of a column without a list, `WITH ORDINALITY` and an outer join stay with PostgreSQL. No format change.
- Improved: pg_sift: the first equality of a backend on a text, varchar or uuid column of a plain `USING sift_col` index no longer parses the whole dictionary: a dictionary of 1,024 strings or more kept in blocks (ids, hashes, uuids) carries the hash of its strings (column format minor 25, 5 bytes a string, `dict_lookup = off` writes none), and the code of the string is found in place. On 3,000,000 uuids the first statement of a backend read 5,977 buffers in 31 ms and now reads 51 in 5.0 ms (a b-tree: 4 in 2.8 ms); on 200,000 rows 410 buffers (uuid) and 445 (text) became 19 and 20. A range, a prefix or an `IN` with a prefix parses the dictionary as before
- Improved: pg_sift: `sift.explain_why` answers a statement with no aggregate (`SELECT * FROM t WHERE id = 5`): one row, the arm `index`, says whether the plan reads a `sift_col` index and, when it does not, why: `no_source` (the table has none), `where` (no condition of the statement is one the index answers) or `cost` (the plan with the index costs more, with both costs); it was an empty result
- Improved: pg_sift: an OFFSET reads the sorted array of the key and does not confirm the entries before it in the heap: `ORDER BY key LIMIT n OFFSET m` over a key with a global order array (`order_array_columns`) gives the first m entries of a unit with no dead row and all-visible pages to the `Limit` as rows of NULLs without fetching their heap rows (the other entries, and those of the write buffer, are confirmed as before), and an OFFSET past the 100,000 rows a top-N heap holds is walked in the array (`topn_rows` bounds the LIMIT of such a scan, not the sum): the scan holds only the LIMIT, where PostgreSQL sorted the table. The array of a top-N is priced as a merge step for each entry, not as a heap of the matches. Random timestamps, instructions of one backend after `VACUUM FREEZE` (a b-tree in brackets), 500,000 / 1,000,000 / 3,000,000 rows: `OFFSET 1000 LIMIT 10` 1.96 / 1.97 / 2.18 M to 0.74 / 0.80 / 1.19 M (1.85 M), `OFFSET 100000` 761 M / 1,569 M / 4,926 M to 46.4 M / 47.3 M / 57.4 M (169 M), `OFFSET 100` 387 K to 312 K (352 K), `OFFSET 0` unchanged (214 K); an integer with NULLs, `ORDER BY ni DESC OFFSET 1000` 1.19 M to 0.61 M (1.01 M); a date and an integer with an array the same as the timestamp. An entry before the OFFSET costs about 460 instructions, where the b-tree's heap fetch costs about 1,800. `bench/bench cases` (1,000,000 rows, ms of one statement): `OFFSET 1000` 0.18 ms (a b-tree 0.23 ms, PostgreSQL's sort 31.0 ms), `OFFSET 100000` 4.2 ms (a b-tree 31.0 ms, the sort 116.0 ms).
- Improved: pg_sift: `sum` and `avg` of a smallint column plus, minus or times an integer constant (`sum(ResolutionWidth + 1)`, `avg(quantity * 3)`; the constant within 2^31 - 2^15 for + and -, 65,535 for *, so that the integer operator cannot raise) are answered from the sums of the units as `mul * sum + add * count`, where they ran the generic aggregate for every row (the result is the bigint or numeric PostgreSQL gives; an `integer out of range` is still raised by the generic path for a constant outside the bound). 90 `SUM(ResolutionWidth + k)` over 3,000,000 rows take 1.0 ms and 29 million instructions where they took 1,807 ms and 24,108 million (the heap plan 1,767 ms), one `SUM(ResolutionWidth + 1)` 0.2 ms against 94.5 ms, and the sums with a filter 49 ms against 159 ms.
- Improved: pg_sift: the list of the pages of a heap that are not all-visible, which a scan of a key-cut index makes once for the statement, reads the visibility map 32 pages at a time while they are all-visible, not a byte (4 pages) at a time: instructions of one backend for a lookup of one key over 2 years on 10,000,000 rows (20,001 keys, 416,000 heap pages) 4.1 million to 3.7 million, and `count(*) WHERE k = <one key>` 2.7 million to 2.3 million.
- Improved: pg_sift: a scan that keeps a few units of an index of thousands (one key of 20,001 over 1,000,000 rows, the index cut by the key and a date) skips the morsels of units that the zones clear without taking each as an item of the scan, which cost 260 instructions a pair of units whatever the zones kept: instructions of one backend for `count(*), sum(amount) WHERE k = <one key> AND d` over two years, 3.2 million to 2.9 million at 4,008 units and 4.8 million to 3.6 million at 15,638 (2.4 million at 500, unchanged), and `count(*) WHERE k = <one key>` 3.1 million to 2.0 million at 15,638 (`bench/cases/key_lookup_units.sql`). The units a parallel scan pruned (`Units Pruned`, `Units Pruned By Set Summary`) were counted again by each participant that built the map of the units the scan keeps, so the figure depended on how many of them took an item: it is counted once.
- Improved: pg_sift: the planner prices the scan from what it measures instead of overcharging it. A comparison of a column costs 0.5 ns a value in a `sift_col` index scan (9 ns before; the bounds of a range count once), handing a row to a bitmap scan 15 ns (500 ns before), the entries of a numeric column that a filter keeps few of are priced for the rows that pass, a column that only comparisons read pays no decode, and a key of an object read by `metrics ->> 'key'` costs 0.6 operators an element (9 before). On 3,000,000 rows the count of a one-month range of a random date column takes 12 million instructions and 1.4 ms (b-tree plan: 54 million, 2.1 ms), with a `sum` beside it 105 million and 5.7 ms (210 million, 25.6 ms); the rows of a window of a twentieth of a table come from the index (24 ms against 63 ms for the sequential scan), and a statement with two or four `metrics ->> 'key'` averages over 1,000,000 rows takes 11.8 and 16.6 ms where it took 167 and 254 ms. Workers are priced for an unnest by the factor the scan is, so a group of 100,000 to 400,000 rows no longer starts workers that make it slower (2.2 to 3.9 ms at 100,000 rows). A lookup table read whole from the leaves of a column index costs the 0.4 power of its rows past 250,000 more than the 8 operators a row it cost (61 to 127 ns measured for 1,500,000 to 6,000,000 rows), so an `EXISTS` over 6,000,000 rows of `lineitem` for the orders of three months stays with the normal plan (190 ms where the scan took 875 ms, the heap plan 178 ms).
- Improved: pg_sift: `CREATE INDEX` of a table of 100,000 rows or more that has no statistics analyzes the columns of the index, not the whole table (`ANALYZE table (columns)`): the choice of the global order reads the statistics of those columns only, so it is the same, and the other columns, which a user did not ask to have analyzed, keep no statistics. A table of 300,000 rows with a jsonb document, two text columns and an index on two integer columns: `CREATE INDEX` 3.86 to 0.79 billion instructions (-79.7%), the same index file; on an arm64 core 4.04 to 0.83 billion (-79.4%), 177 to 45 ms.
- Improved: pg_sift: `CREATE INDEX` of a `sift_col` index over a jsonb column reads the numbers of the documents faster: a number of at most 18 digits and no fraction or exponent becomes an integer by one pass over its digits, a decimal of at most 15 digits becomes a double by one division by a power of ten (both exactly the value `strtod` gives, checked on 200 million random decimals) where the generic parse and `strtod` ran, and a key of a jsonb object is looked up first at the member after the last one found (jsonb lists its keys in the order of the members) before the binary search: 300,000 documents of eleven numbers `CREATE INDEX` 6.52 to 4.94 billion instructions (-24.2%), 1,068 to 805 ms; documents of text, tags, a list of objects and four numbers 8.17 to 7.42 billion (-9.2%); on an arm64 core 5.63 to 4.82 billion (-14.4%), 405 to 356 ms, and 7.67 to 7.22 billion (-5.9%). The index file is the same bytes (compared page by page). No format change and no option.
- Improved: pg_sift: the hash table of a dictionary under construction (the strings of a `CREATE INDEX`, a merge or a defrag) keeps the hash of each string in its slot, so a probe compares the strings only of the slots with its hash and a growth of the table puts the codes back without hashing or comparing a string again: 1,000,000 rows of three text columns `CREATE INDEX` 8.96 to 8.59 billion instructions (-4.1%), two `inet` columns 4.76 to 4.64 billion (-2.5%), the merge of five segments of the same rows 16.5 to 15.4 billion (-6.8%); on an arm64 core, 1,000,000 rows with unique strings 8.57 to 8.34 billion (-2.6%). The index file is the same bytes (compared page by page). No format change and no option.
- Improved: pg_sift: the sorts of a `CREATE INDEX` and a merge on integer keys (the directory of the keys of a list or of the values of a point column, the sorted runs of `order_columns`, the global order of `order_array_columns`) are a radix sort on the bytes the keys differ in, where they were a comparison sort of two or three fields: 1,000,000 arrays of 6 to 30 integers 26.8 to 21.8 billion instructions (-18.8%), 400,000 rows with a global order on an integer 1.69 to 1.53 billion (-9.5%) and on a 64-bit column 1.95 to 1.68 billion (-13.9%). The index file is the same bytes (compared page by page). No format change and no option.
- Improved: pg_sift: the first backend that parses a dictionary of a `sift_col` segment (128 kB or more of strings) keeps it in shared memory, and the other backends (a pooled connection that was reset, the workers of a scan, the next session) take it copied from shared memory with a CRC-32C check, where each inflated and indexed it again: a ring in the dynamic shared memory registry (nothing is allocated at start, no `shared_preload_libraries`, no setting) of a quarter of `shared_buffers` (64 to 512 MB) and a quarter of `/dev/shm` at most. The first statement of a new backend over 1,000,000 rows of IPv4 addresses with `ip = const` takes 56 instead of 210 million instructions (the second statement of a backend, 42), a sum of a numeric of near unique values 697 instead of 844 (second statement 674), `url LIKE '%/p123%'` 67 instead of 278 (38), and over 3,000,000 rows 163 instead of 738 and 2,062 instead of 3,518 (shared_buffers 1 GB); a dictionary over half of the 64 MB a backend keeps, which every statement inflated again, is copied (the second statement of the 3,000,000 rows `ip = const` 140 instead of 367). The backend that parses and keeps it pays 10 to 27 million instructions once. A merge, a rewrite or a `VACUUM` that rewrites a segment names a new generation, so nothing is invalidated; a ring that throws out more dictionaries than backends find in it stops writing for a while; where the memory cannot be had (a container's 64 MB `/dev/shm` that is full) the backend logs it once and parses as before. Every backend still keeps its own copy of what it uses (`dict_cache_kb`). `sift.stats()` counts `dict_shared_hits`, `dict_shared_misses` and `dict_shared_held`, and `EXPLAIN (ANALYZE, VERBOSE)` shows `Shared Dictionaries`.
- Improved: pg_sift: an array column of a `sift_col` index whose `paths` list its elements and have no `+` keeps no image of the arrays (the head leaf is NULL in every row): where nearly every array of a table is different that is the index of a column of 800 values and 1 to 8 elements at 73.3 MB in place of 123.9 MB with the images (1,000,000 rows, two such columns; GIN 61.9 MB). A statement that needs the whole array (`GROUP BY` or `DISTINCT` of an array, `count(DISTINCT x)`, the array itself, `x = ARRAY[..]` of two dimensions or another lower bound) reads the table for that column.
- Improved: pg_sift: `ids <@ ARRAY[..]` on a `smallint[]`, `int[]` or `bigint[]` column of a `sift_col` index whose elements are listed (`paths` names `ids[] int4`) are answered by the index (the array is not NULL and no element is NULL or outside the set: one `NOT IN` test of the integer element leaf), where only a text array was and an integer one scanned the table (1,000,000 rows of 6 to 30 elements, 20,000 values: a literal of 30 elements 2,746 to 128 M instructions, one of 20 elements 2,294 to 109 M, the same 5,245 rows; the negation, which `NULL` arrays make unknown, still reads the table).
- Improved: pg_sift: a statement with a list of thousands of literals (`key IN (..19,257 numbers..)`, as an ORM sends) compared the list with the last one number by number at every begin of a range of units; it compares the bytes at once (a scan of 1,000,000 rows under a 2-year window with a list of 19,257 keys of 191,911: 568.9 to 465.0 million instructions, -18%, 217.4 million without the list).
- Improved: pg_sift: `WHERE grp = x ORDER BY ts LIMIT n` (the last events of a device, the newest rows of a tenant), `grp = x AND ts > y ORDER BY ts LIMIT n`, the page after a key inside a group and `max(ts) WHERE grp = x` over a `sift_col` index read `n` entries of the group in each segment and `n` rows of the heap, not the units: the default of `order_array_group` is now `auto`, which adds to the global order of each date and timestamp column (`order_array_columns`) a second order sorted by the integer column of most values (100 values or more, 64 rows apiece, scattered in the heap; it keeps the plain order beside it, so no other shape reads more). 500,000 rows with random timestamps, instructions of one backend: `grp = x ORDER BY ts DESC LIMIT 10` 3.34 M to 0.42 M (a b-tree 0.27 M), `ts > y` page 3.85 M to 0.52 M, `max(ts) WHERE grp = x` 3.27 M to 0.61 M; the plain `ORDER BY ts LIMIT 10` 0.281 M to 0.284 M. The index grows 13.2% (6.7 bytes a row) and CREATE INDEX 5.7%; `ALTER INDEX .. SET (order_array_group = 'off')` and a `REINDEX` drop it. Column format minor 31, which 3.5 and older cannot read (`sift.upgrade_check` says to turn `order_array_columns` off and `REINDEX`).
- Improved: pg_sift: `WHERE grp IN (a, b, c) ORDER BY ts LIMIT n` (and the same list written as `grp = a OR grp = b`: the last events of a few devices, the newest rows of a few tenants) over a `sift_col` index with a global order grouped by `grp` reads the entries of each value of the list, up to 16, one after the other and merges them by `ts`, as it reads those of `grp = x`, instead of every unit. In instructions of one backend at 500,000 / 1,000,000 / 3,000,000 rows of random timestamps, a list of 3 values and `LIMIT 10`: 3.86 / 6.75 / 15.1 M to 0.71 / 0.87 / 1.45 M (a b-tree on `ts` 3.66 M at 500,000 rows); the plain order, range, `OFFSET`, keyset and `grp = x` shapes change by at most 1% and the index by none. A list of more than 16 values, with a NULL, or `NOT IN` reads the units as before.
- Improved: pg_sift: the buffer a segment reads its unit records into (up to 400 kB for a unit of 100,000 rows) is kept by the backend between statements, as the cursor buffers of the global order are, where the heap grew by it and shrank at the end of each statement: `ORDER BY ts DESC LIMIT 10` over 100,000 rows with random timestamps 42 to 0 page faults, 0.155 to 0.100 ms (instructions 349,443 to 335,545), the other lookups of the lab table (`ts >= x LIMIT 10`, a point `id = x`, a range of 5 minutes) unchanged at 100,000 and 1,000,000 rows (0 faults, 0.07 to 0.15 ms).
- Improved: pg_sift: the pairs of group and value of a `count(DISTINCT x)` whose sets outgrow the hash memory are sorted as int8 datums, not as tuples of two columns (no tuple to form and read back for each pair): the spilled sets of 500,000 pairs over 50,000 groups take 1,010 instead of 1,570 million instructions.
- Improved: pg_sift: the rows of a unit that the prefilter of an equality of a periodic calendar step keeps (`extract(month FROM ts) = 3`, `to_char(ts, 'YYYY-MM') = '2025-03'`) are not tested by the expression again when the source narrowed the unit to the ranges of the field (a unit that meets 32 intervals or fewer, with no NULL-only or overflowed zone): the rows of the intervals are exactly those where the field has the value. On 1,000,000 rows of a timestamp, instructions of the backend: a month 130 million to 3.3, a day of the year 7.2 to 2.1, `to_char(ts, 'YYYY-MM')` 101 to 2.5; the hour and the minute of a span of years, which the prefilter does not narrow, are as before.
- Improved: pg_sift: a sum or average of a `numeric` column without a precision (an image of digits and a scale), or of one past 18 digits, which nothing else in the query reads, adds an integer and a scale made from the lanes of each row that passes the filters, where it hashed every row of the unit into a dictionary of numerics and decoded each distinct value back. Over 1,000,000 rows the sum takes 289 million instructions where it took 451 (9,973 distinct two-decimal values; the normal plan 813), 614 where it took 1,190 for almost unique values (912), and a filtered sum grouped by a key (a quarter of dates, 1,000 keys) 116 where it took 347 (210): it stops losing to the normal plan's bitmap scan. Rows that are NaN, an infinity or more than 38 digits are added as numerics, as before; a `min`, `max`, `DISTINCT`, comparison or group key on the column keeps the dictionary. A `sift_table` keeps the dictionary too. The sums keep their value and display scale (the scale of the widest value added). No format change and no option.
- Improved: pg_sift: the default `key_cut` is decided by the default formula alone, not by a limit on the growth of the index (25% for a date or timestamp, 3% for a key) or a minimum of 64 units: the benefit of the cut (the speedup of a lookup or a range, in 5% steps) against its cost (a third of the predicted growth in percent and a fifth of the growth of the build), and a table of fewer than 32 superblocks is sampled whole; the prediction counts the headers of a unit that the baseline left out (over by 23 to 34 points on rows of 56 and 190 bytes before, 2 to 19 after). 3,000,000 rows of 56 bytes scattered over 1,000 days, `sift_col (d)`: a day count 25.4 M to 0.7 M instructions (b-tree 1.6 M), a week 25.6 M to 0.9 M (10.4 M), a group by day 88.7 M to 3.8 M (b-tree 1,885 M), the index 4.9 to 7.1 MB (b-tree 21.2 MB); 500,000 rows (23 units) a day 4.5 M to 0.3 M; `@>` on an `int4range` of 500,000 rows 4.4 M to 1.0 M (GiST 0.4 M), 3,000,000 rows 24.7 M to 0.6 M (GiST 0.5 M); `CREATE INDEX` at 3,000,000 rows of 56 bytes 1,776 to 3,485 instructions a row (0.8 to 2.9 s one process, 0.5 to 1.7 s parallel), of 190 bytes 1,919 to 3,154 (1.2 to 2.2 s); the sample that decides is sorted in line (1,298 M instructions, it was 2,208 M); `key_cut = 'off'` keeps the units by superblock
- Improved: pg_sift: `ORDER BY x LIMIT n` over a column with ties (a column with NULLs under `NULLS FIRST` or the `DESC` default, a column of few values: `ORDER BY ni DESC LIMIT 10` of a column with 30% NULLs, `ORDER BY sel LIMIT 10`) no longer reads the units whose best row only ties with the n-th: with one sort key a tie wins nothing (the SQL order does not say which of the tied rows comes), so the scan stops at the first n rows of the best value, as a b-tree does, where it read every unit that holds a NULL or the same value. In instructions of one backend at 500,000 / 1,000,000 rows (3,000,000 in the pull request), `ORDER BY ni NULLS FIRST LIMIT 10` 47.3 / 94.5 M to 2.1 / 2.2 M, `ORDER BY ni DESC LIMIT 10` 48.2 / 96.1 M to 1.6 / 1.1 M, `ORDER BY sel LIMIT 10` 5.6 / 10.9 M to 0.48 / 0.49 M, `grp = x ORDER BY ni NULLS FIRST LIMIT 10` 7.4 / 10.2 M to 0.78 / 1.1 M; `ORDER BY ni LIMIT 10` stays 1.5 M (one unit, a b-tree 0.23 M). The tied rows come in table order, as before. `sift.explain_why` names the tie column of a keyset page `(ts, id) > (x, y)` whose index lacks it.
- Improved: pg_sift: `ORDER BY x [DESC] LIMIT n` and `NULLS FIRST` over an integer, date or timestamp column whose values follow the order of the rows (a counter, a timestamp that only grows, a key loaded in order; also with NULL rows in between) reads the best rows of each unit from its end, a block of 1,024 rows at a time, and no longer decodes the whole key chunk of the unit (delta chunks decode from the checkpoint of the block; the walk starts only when the rows that pass say it ends within 4 blocks, and goes back to the whole unit otherwise). 300,000 / 1,000,000 rows (units of 20,000 rows, instructions of one backend; a b-tree 0.19 M): `ORDER BY ni LIMIT 10` 0.86 / 0.76 M to 0.51 / 0.42 M, `DESC` 1.31 / 1.10 M to 0.51 / 0.41 M, `NULLS FIRST` 1.25 / 1.16 M to 0.51 / 0.42 M, `ts DESC` 6.33 / 0.87 M to 0.83 / 0.34 M, the same blocks of the index read (cold 4 + 15 blocks, 122,880 bytes; warm 19 hits); a key that does not follow the row order, two sort keys, a condition on the key, an `OFFSET` or a limit over a few blocks of a unit read the unit as before (`Units Read From Their Ends` in EXPLAIN ANALYZE counts the units).
- Improved: pg_sift: `ORDER BY x LIMIT n` over a nullable integer, date or float column no longer pushes the NULL rows of a unit through the heap one by one: with NULLs last the values are taken first and a bar that is a value spares every NULL row, with NULLs first a row that only ties with the n-th is skipped at once (one sort key). In instructions of one backend at 500,000 / 1,000,000 rows of a column with 30% NULLs, `ORDER BY ni LIMIT 10` 1.57 / 1.59 M to 0.55 / 0.57 M (a b-tree 0.23 M), `ORDER BY ni DESC LIMIT 10` 1.68 / 1.17 M to 0.64 / 0.53 M, `ORDER BY ni NULLS FIRST LIMIT 10` 2.18 / 2.20 M to 0.74 / 0.77 M, `ORDER BY ni DESC NULLS LAST LIMIT 10` 1.25 / 0.92 M to 0.50 / 0.46 M, `ORDER BY sel LIMIT 10` 0.53 / 0.54 M to 0.47 / 0.49 M; the rest is the decode of the 20,000 rows of the first unit, the same at both sizes. Every other shape of the matrix (points, ranges, keysets, groups, counts) within 0.5% of the tip.
- Improved: pg_sift: a text order array merges its segments by a tournament (a step compares strings log2 of the segments times where it compared each segment's) and decodes a string of its block of 16 from where the last one ended, not from the start of the block. In instructions of one backend, `ORDER BY txt LIMIT 10 OFFSET 1000` of 500,000 / 1,000,000 / 3,000,000 rows of 13-byte strings under a locale 2.96 / 2.97 / 7.47 M to 2.53 / 2.54 / 5.24 M (under the C collation 3.02 / 3.04 / 3.47 M to 2.60 / 2.60 / 2.91 M; a b-tree 1.9 M), `OFFSET 10000` 26.9 / 27.0 / 71.3 M to 22.4 / 22.4 / 48.6 M (a b-tree 17.0 M), `LIMIT 10` 272 / 353 K to 269 / 337 K at 500,000 / 3,000,000 rows; integer and date arrays unchanged (within 0.1%). Test `col_gorder_text`: 6 segments, ties across them, NULLs, ICU strings, both directions.
- Improved: pg_sift: a filter that is a calendar expression of a timestamp column of a `sift_col` index and no range of it (`extract(dow FROM ts) = 3`, `extract(hour FROM tz) = 5 OR extract(dow FROM tz) IN (0, 6)`, `extract(minute FROM ts) > 55 AND extract(hour FROM ts) <> 3`) is evaluated once for each day, hour or minute the rows fall in (EXPLAIN shows `(per day)`), the step of the finest calendar function in it, in the session's time zone, where it was evaluated for each distinct timestamp: over 1,000,000 timestamps of three years the weekday takes 34 million instructions where it took 1,729, the hour 93 where it took 1,903, and a weekday with an hour 133 where it took 2,504. The rows are those of the normal plan in zones whose clocks jump, for operands no row has, infinite timestamps and NULL. A text format (`to_char`) keeps its expression evaluated for each value. No format change and no option.
- Improved: pg_sift: the 27 sorts of arrays of integers, dates and addresses in the column engine and the aggregate planner (the keys of an `IN` list, the addresses of a unit, the heads of a scan, the sampled keys of the `key_cut` choice, the block list of a lane and the like) compare in line, as the entries of a global order do, where `qsort` called a function for each comparison: 1,000,000 values of 64 bits sort in 45 ms against 88 ms.
- Improved: pg_sift: a backend keeps what it read of the write buffers for the lookups that test an equality, an `IN` list or `= ANY (array)` on a plain leaf (the previous entry): it reads only the rows appended since its last lookup and probes a hash of the leaf's values, so a lookup with 10,000 rows in the write buffer takes 0.28 million instructions where an empty buffer takes 0.25 million (the previous entry: 2.5 million, before it 45 million; 1,000,000 rows; a b-tree 0.15 million), whatever the length of the list; the first lookup of a backend reads the buffer once (5.2 million at 10,000 rows). A row in a lane that a writer is still completing is read again at the next lookup. No format change, no setting, no change to a write.
- Improved: pg_sift: an index scan of a `sift_col` index that tests an equality, an `IN` list or `= ANY (array)` (up to 65,536 values, matched by a binary search above 8) on a plain integer, date, timestamp, text or uuid column finds the rows of the write buffers in their records, instead of building the batches, dictionaries and units of every buffered row for each scan: a lookup, `UPDATE` or `DELETE` by key with 10,000 rows in the write buffer takes 2.5 million instructions where it took 45 million (1,000,000 rows, 6 segments; a b-tree on the same key 0.17 million). The cost is still proportional to the buffered rows, 250 instructions a row instead of 4,500; no format change, no setting.
- Improved: pg_sift: `x = ARRAY[..]` on a `text[]`, `varchar[]`, `int2[]`, `int4[]` or `int8[]` column of a `sift_col` index with the elements of its arrays listed, against a constant of one dimension from 1 with no NULL element, is answered from the index: the cardinality of the array, then the first four different elements of the constant (the point directory lists the rows that own them), then, for an index with no image of the column, the elements of the rows that are left read in order (an array of two dimensions or another lower bound keeps its image in the head and is not the constant's); with the image kept the image is compared last. 1,000,000 rows, a column of 800 values with Zipf skew and 1 to 8 elements, the index at 72.6 MB (GIN 61.9 MB): two elements, one common and one rare, 2.5 ms and 12.5 M instructions (GIN 0.9 ms and 9.2 M; the table scan 101 ms and 1,004 M), three elements 9.2 ms and 51 M (GIN 6.4 ms and 49 M), an integer array of two 5.4 ms and 19 M (GIN 1.2 ms and 10.5 M), a column that keeps its image 0.16 ms and 0.5 M (GIN 0.07 ms and 0.3 M), one element that 26,000 rows are equal to 22.6 ms and 165 M (GIN 70 ms and 457 M). The negation is read from the table.
- Improved: pg_sift: a test `x = 'literal'` of a text leaf builds the table of the entries of its dictionary it needs by comparing bytes, as it compares them in the aggregate scan, in place of evaluating the expression for each entry: the equality of a text array with two elements (1,000,000 rows, a dictionary of 800 entries a unit) 15.4 to 12.8 M instructions (-17%), with three elements 72.0 to 53.1 M (-26%).
- Improved: pg_sift: `CREATE INDEX` of a `sift_col` index with an array column of `text`, `varchar`, `int2`, `int4` or `int8` and no `paths` (a table of 10,000 rows or more) reads a sample of 4,096 rows and picks the layout of each column by the default formula: where over half of the sampled arrays are different the elements stand in place of the whole-array image, which shares nothing there (1,000,000 rows, two such columns: 73.3 MB against 123.9 MB with the images; GIN 61.9 MB); where the arrays repeat but the element tests would decode each distinct array of the image (`regions @> x AND stars = 3` over 196,999 distinct arrays of 1,000,000 rows: 311.6 M instructions) the elements are kept beside the image (`+` in the stored paths: 8 M); a column with few distinct arrays keeps the image alone. A NOTICE names the columns and how to leave the elements out (`paths = ''`, then REINDEX); `GROUP BY`, `DISTINCT` and `=` of an array whose image is not kept read the table (`array_distinct` over 1,000,000 rows: 10,167 M instructions against 1,234 M with the image). No option.
- Improved: pg_sift: `tags @> ARRAY[..]`, `&&`, `<@` and `= ANY` on a `text[]`, `varchar[]`, `int2[]`, `int4[]` or `int8[]` column of a `sift_col` index whose `paths` name `tags[] text` (or `int4`) are answered from the list of the elements of the column (the list a jsonb array of strings has) and its point directory, where every array was read: 1,000,000 rows of 6 to 30 elements, a rare element 4.6 M instructions (GIN 1.6 M; sequential scan 1,470 M), a medium one 5.1 M (GIN 3.9 M), a frequent one 60.9 M (GIN 242 M), `&&` of three elements 75.5 M (GIN 40.6 M), `= ANY` 71.2 M (1,937 M for GIN, which cannot use its index). The column keeps no image of the whole array: its elements and lengths hold it, so `GROUP BY`, `DISTINCT`, `=` of whole arrays and the array itself read the table; a column without a list of elements keeps its image. No option.
- Improved: pg_sift: the global order of a leaf (`order_array_columns`) is written in chunks of 512 entries, not 1,024 (column format minor 30; a segment of minor 21 is read as it was, and a test setting writes it): the seek that finds the bound of a range decodes half of the keys and addresses, the directory entries are read one at a time by a cursor and the whole directory only by a seek. `WHERE ts >= x ORDER BY ts LIMIT 10`, random timestamps, instructions of the backend: 1,000,000 rows (6 segments) 0.51 to 0.43 million, 3,000,000 rows (14 segments) 0.90 to 0.74 million; a range of 5 minutes 0.88 to 0.72 million at 3,000,000; `DESC LIMIT 10` 0.28 to 0.28 and 0.50 to 0.48 million; the index grows 0.9% and 1.0%, `CREATE INDEX` 0.3% more instructions; a b-tree 0.17 million. 512 entries against 256 by the default formula: B 3.9, C 0.4 against B 5.3, C 1.2, and 256 grows the index 3.2%. A 3.5 library refuses the segment (`ALTER INDEX x SET (order_array_columns = 'off')` and `REINDEX` first).
- Improved: pg_sift: `ORDER BY key LIMIT n` with a range of the key over a global order (`order_array_columns`) decodes the chunk of each segment that holds the bound once: the seek that finds the bound leaves the chunk to the cursor that reads the entries from it, where the cursor decoded its head again (`WHERE ts >= x ORDER BY ts LIMIT 10`, 3,000,000 rows of random timestamps: 1.25 to 0.89 million instructions, `DESC` 1.13 to 0.90, a keyset page of 50 on `(ts, id)` 1.75 to 1.20; 1,000,000 rows 0.67 to 0.51, 0.59 to 0.51, 1.00 to 0.79; a b-tree 0.26 million).
- Improved: pg_sift: the sorts of the entries of a global order (`order_array_columns`), of a point directory and of a unit's sorted run compare in line (the PostgreSQL sort template) instead of through a function pointer for each comparison, so the merge, the build and the promotion of a `sift_col` index spend less on them (3,000,000 rows, 64 MB `maintenance_work_mem`, `key_cut`: one merge of the 19 segments of a serial build 65.5 against 61.5 billion instructions, 6.1% fewer; of the 71 segments of a 4-worker build 81.3 against 77.1 billion, 5.2% fewer; the segments written are the same).
- Improved: pg_sift: the merge of a `sift_col` index with `key_cut` takes the next row from a heap of its input segments, not by comparing the next row of every segment, so the rewrite after a build with many segments costs less (3,000,000 rows, 64 MB `maintenance_work_mem`: 19 segments of a serial build 65.5 against 60.6 billion instructions, 7% fewer; 71 segments of a 4-worker build 81.3 against 61.1 billion, 25% fewer; the segments written are the same).
- Improved: pg_sift: a scan with a text test that the column source runs on the encoded form (`LIKE '%x%'`) tests its other conditions on the rows that test left: the leaves are read for those rows only, the strings kept in a unit are put in its dictionary for them alone, and the test of a segment dictionary (`url NOT LIKE '%.google.%'`) is made for the entries those rows hold, not for every entry of the dictionary. A unit that the text test empties reads no other leaf (`count(DISTINCT userid)` by search phrase of 1,000,000 web pages where the title has a word and the url lacks another: 3,716 million instructions to 618 million, 406 ms to 69 ms; the rows are those of the scan before). Test: `agg_sparse_text`.
- Improved: pg_sift: a scan that unnests a list or object of a `sift_col` index and keeps a minority of the elements of a chunk finds the row of each kept element by a walk over the offsets of the rows, not by the owner array of every element of the unit (a window of two years over 1,000,000 rows of 12 elements, one backend: 221.1 to 208.7 million instructions, -5.6%; with 10% of the keys 58.8 to 53.1, -9.7%; windows of 90 and 7 days and the scans that keep every element are unchanged within 0.3%).
- Improved: pg_sift: a scan that unnests a list or object of a `sift_col` index and keeps few of its rows sets the elements of each kept row as one range instead of looking up the owner row of every element. The rows of every answer are those of the scan before.
- Improved: pg_sift: the element leaf of a list of strings or numbers of a `sift_col` index (a jsonb array under `paths`) has a point directory in every segment that lists the rows that own each value, once however many times the row holds it, as differences in LEB128 (1.5 to 2 bytes a row of a value; column format minor 16), and a test on a value (`doc @> '{"tags": ["x"]}'`, `?`, `EXISTS` over the list) takes its rows from it without reading a unit: a value no row owns reads no unit and vets no heap row (`Rows Scanned: 0`), and the rows of a few passing elements are found by a search from each element, not by a walk of every row.
- Improved: pg_sift: an equality of a month, day of the year, day of the month, hour or minute, or a `to_char` of the numeric fields with a constant (`extract(month FROM ts) = 3`, `to_char(ts, 'YYYY-MM') = '2026-03'`) on a timestamp column of a `sift_col` index narrows the units to the intervals where the field has that value, found when the scan starts in the session's time zone (a `timestamptz` window ends where its clock goes back), before the expression is tested for the rows that stay: on 1,000,000 rows, a month takes 553 million instructions where it took 1,869, a day of the year 326 where it took 2,133, `to_char(ts, 'YYYY-MM')` 48 where it took 2,691, `'YYYY-MM-DD'` 33 where it took 3,136, and a month no row has 9. The answers are those of the expression in every zone tested (the repeated hour of New York, the half hour of Lord Howe, the skipped day of Apia). The weekday and quarter fields cover more than an eighth of the values, and the hour and minute of a span of years have more intervals than a scan tries: they are tested as before. No format change and no option.
- Improved: pg_sift: an OR of ranges of one integer, date or timestamp leaf of a `sift_col` index (this week and the same week a year before: `(d >= A AND d <= B) OR (d >= C AND d <= D)`, a list of days, calendar steps such as `date_trunc('month', d) = M1 OR .. = M2`) is answered by the source from the sorted run of each unit, one search for each range, where the scan tested every row of the unit; the ranges may overlap, touch, be empty or an equality, and a union of more than an eighth of the values of the index stays with the executor. A seven-day window and its twin a year before, grouped over a 1,000,000-row table, takes 53.6 million instructions where it took 142.7 (the elements of a map), and 57.4 where it took 148.2 (a plain count and average per key); a 90-day window 112.9 where it took 167.7, and 101.8 where it took 152.6; a list of 100 keys with the seven-day window 12.7 where it took 23.9, and a list of 1,000 keys 70.0 where it took 140.8. A selection of consecutive rows (a window) is decoded as runs and not row by row, and a unit narrows to the survivors of its filters when they are fewer than a sixteenth of its rows (a thirty-second). Queries without an OR of ranges execute the same instructions (0.00% on a key, a list of 10, 100 and 1,000 keys, and every key); a window of more than an eighth of the values (two years) and a single key change by less than 0.6%. No format change and no option.
- Improved: pg_sift: a lookup through the global order (`ORDER BY key LIMIT n` over `order_array_columns`) took 50 to 100 page faults, which the kernel needs to hand back the pages a statement allocated and malloc gave back to it at the end: the 16 KB buffers of each segment's cursor are kept by the backend between statements, and a scan allocates the vectors of float aggregates and of the combos' value ids only when it has an aggregate or a combo to read them (`ts >= x ORDER BY ts LIMIT 10`, 58 to 0 faults, 0.19 to 0.14 ms at 1,000,000 rows of random timestamps and 0.27 to 0.17 ms at 3,000,000; `ORDER BY ts DESC LIMIT 10` 0.10 to 0.075 ms and 0.15 to 0.10 ms; instructions 0.4% lower, 0.71 to 0.71 million).
- Improved: pg_sift: a scan over a partitioned table narrows a unit to the rows its filters keep, as the scan of one table does (the wrapper of the partitions had no `narrow`, so a unit whose filters left 3% of its rows was still prepared and read for all of them): a one-week window of 15 years of dates, grouped by the keys of a map over 1,000,000 rows in 8 partitions, takes 54 million instructions where it took 120 (one table of the same rows: 51), and a window of both bounds 52 where it took 122. The rows are those of the normal plan. No format change and no option.
- Improved: pg_sift: the keys of a lookup join filter are sorted and made distinct only when a second filter of the same column asks, which a join of one table never does (the scan of a semi join with 640,000 keys spent half of its time in that sort).
- Improved: pg_sift: the normal plan of a hash aggregate whose table outgrows the cache of a core is priced for the memory misses of every row and the table doubling as it fills (4 to 7 misses a row at 7 to 40 MB, the cost of an operator a row before whatever the groups), where a scan competes with it: a `GROUP BY` of a near-unique text key over a table of 1,000,000 rows takes the scan, 0.5 s where the normal plan took 0.9 to 1.2 s (the plan was priced 66 ms), and a key of a few values or one that a selective filter leaves with few groups keeps the plan it had; a grouping of 1,000,000 unique keys that the sort prices below the hash table now sorts (1.6 s to 0.7 s).
- Improved: pg_sift: the merge of a `sift_col` index writes partitions of at most a third of `maintenance_work_mem` (at least 4 MB, at most 48 of them, `sift.col_auto_merge_partition_kb` still sets the size), not one segment of all the bytes of its inputs: the writer of a segment keeps the key and heap address of every row of a global order (`order_array_columns`) until the last unit is in, about 2.5 bytes of memory for each byte of segment, so a merge took memory in proportion to the index (3,000,000 rows, `maintenance_work_mem` 64 MB, serial build, 8 segments to one: 220 MB of private memory of the backend, now 86 MB in 5 partitions; 1,000,000 rows, 16 MB: 81 MB, now 29 MB; the merge 6.2 s now 4.9 s, WAL 76 and 78 MB). The file still doubles while the segments of a build, which all cover the same key range, are merged: none of them is free before the last unit is read (158 MB for 78 MB of index). A `maintenance_work_mem` larger than the index, or `sift.col_auto_merge_partition_kb` above it, writes the one segment as before.
- Improved: pg_sift: an index created with the default class (`CREATE INDEX .. USING sift_col (id, name, ts)`, no `sift.col_*_ops`) answers the filters that return rows, not only the aggregates: `WHERE id = 5`, `IN` lists, ranges, `LIKE 'abc%'` and `IS NULL` on integer, date, timestamp, boolean, float, numeric, char(n), text and uuid columns use an index, index-only or bitmap scan, and so does `UPDATE` or `DELETE` with such a `WHERE`. The key of a statement without an aggregate on 200,000 rows, instructions of the backend and milliseconds (a b-tree in brackets): `int4` unique 89.4 M and 14.6 ms to 0.295 M and 0.052 ms (0.184 M, 0.062 ms), `uuid` 128.0 M to 0.287 M (0.186 M), `text` 157.0 M to 0.286 M (0.187 M), `date` 100.5 M to 1.35 M, an `IN` list of 16 keys 110.1 M to 2.30 M (0.391 M); a lookup of `int8` on 1,000,000 rows with 6 rows a key 515.9 M and 113.7 ms to 0.94 M and 0.13 ms. A statement that aggregates, groups, deduplicates, has a window function or sorts under a `LIMIT` keeps the plans it had on such an index (SiftAgg or the normal plan), and an index with a named class is unchanged. Already built indexes gain it with `ALTER EXTENSION pg_sift UPDATE`, no REINDEX.
- Improved: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index without `key_cut` cut the units by the start of a range column when a sample shows it scattered over the units, the index does not grow and no other leaf would lose its zones (a sequential id, or a date that follows the table or drifts over it, keeps the superblock cut: the cut spreads the zones of every other leaf over all the units) (`key_cut = 'auto: r#1'`, and the `NOTICE` names it): `r @> x` over 3,000,000 random ranges beside an address and a 128-character note reads 0.67 million instructions where the superblock cut read 27.9 million (a GiST: 1.64 million), `r && range` 0.87 million where it read 28.3; the index is 6.0% smaller (66.5 to 62.5 MB) and the build 58.8% more instructions (20.4 to 32.4 G); `ip <<= net` and `ip >>= addr` over the same index read 25.6 and 95.7 million without the cut and 23.8 and 93.9 with it. The first look of the chooser lets a time leaf whose predicted growth is up to 15% reach the full sample (it was 7%), which costs a sample of 36 superblocks and decides as before. The chooser of the default `key_cut` prices the dictionary-coded image of a range or an address like its first lane, which a unit of the cut lists in the order of the lane: an index on an address alone (3,000,000 random addresses and a 128-character note) is now cut by `ip#1` with no option (predicted size -6.8%, built -7.4%, 26.6 to 24.6 MB), where it was left without the cut with a predicted growth. An inet or cidr column keeps a third lane, the offset of an IPv4 address inside its block, so `ip = x` is the node and the offset on the lanes and the aggregate no longer reads the dictionary of the segment; the blocks that contain a network (`>>=`, `>>`, `&&`) are found by the lowest bit of the node in a table of 64 slots, and a node of 3-byte values is written bit-packed (at most 1.6% larger) so that its range test has a kernel. Without any cut, 3,000,000 addresses: `count(*)` of `ip = x` reads 15.5 million instructions (323.9), `ip <<= net` 22.3 (23.6), `ip >>= addr` 69.6 (96.3), `ip && net` 83.8 (321.6); rows of `ip = x` 13.6 (21.4). With the cut: `ip = x` returns its rows in 1.34 million instructions, `ip <<= net` 1.56, `ip >>= addr` 3.43, and `count(*)` of `ip = x` reads 0.47 million (580 with the cut before this change), `ip <<= net` 0.52, `ip >>= addr` 2.62. The test of the blocks that contain a network (`>>=`, `>>`, `&&`) on the node lane is one vector kernel (`anc_match`: AVX2, AVX-512 and NEON from one source, `col/kern_anc.h`) where it was a table load a row: `count(*)` of `ip >>= addr` on 3,000,000 uncut addresses reads 53.3 million instructions (69.6), `ip && net` 67.4 million (83.8); on an arm64 core, 1,000,000 addresses, `ip >>= addr` 33.2 million instructions (48.5 with `sift.simd = scalar`), `ip && net` 38.8 million (85.7).
- Improved: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index without `key_cut` cut the units by the start of a range column when a sample shows it scattered over the units, the index does not grow and no other leaf would lose its zones (a sequential id, or a date that follows the table or drifts over it, keeps the superblock cut: the cut spreads the zones of every other leaf over all the units) (`key_cut = 'auto: r#1'`, and the `NOTICE` names it): `r @> x` over 3,000,000 random ranges beside an address and a 128-character note reads 0.67 million instructions where the superblock cut read 27.9 million (a GiST: 1.64 million), `r && range` 0.87 million where it read 28.3; the index is 6.0% smaller (66.5 to 62.5 MB) and the build 58.8% more instructions (20.4 to 32.4 G); `ip <<= net` and `ip >>= addr` over the same index read 25.6 and 95.7 million without the cut and 23.8 and 93.9 with it. The first look of the chooser lets a time leaf whose predicted growth is up to 15% reach the full sample (it was 7%), which costs a sample of 36 superblocks and decides as before. The chooser of the default `key_cut` prices the dictionary-coded image of a range or an address like its first lane, which a unit of the cut lists in the order of the lane: an index on an address alone (3,000,000 random addresses and a 128-character note) is now cut by `ip#1` with no option (predicted size -6.8%, built -7.4%, 26.6 to 24.6 MB), where it was left without the cut with a predicted growth. An inet or cidr column keeps a third lane, the offset of an IPv4 address inside its block, so `ip = x` is the node and the offset on the lanes and the aggregate no longer reads the dictionary of the segment; the blocks that contain a network (`>>=`, `>>`, `&&`) are found by the lowest bit of the node in a table of 64 slots, and a node of 3-byte values is written bit-packed (at most 1.6% larger) so that its range test has a kernel. Without any cut, 3,000,000 addresses: `count(*)` of `ip = x` reads 15.5 million instructions (323.9), `ip <<= net` 22.3 (23.6), `ip >>= addr` 69.6 (96.3), `ip && net` 83.8 (321.6); rows of `ip = x` 13.6 (21.4). With the cut: `ip = x` returns its rows in 1.34 million instructions, `ip <<= net` 1.56, `ip >>= addr` 3.43, and `count(*)` of `ip = x` reads 0.47 million (580 with the cut before this change), `ip <<= net` 0.52, `ip >>= addr` 2.62.
- Improved: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index without `key_cut` cut the units by the start of a range column when a sample shows it scattered over the units, the index does not grow and no other leaf would lose its zones (a sequential id, or a date that follows the table or drifts over it, keeps the superblock cut: the cut spreads the zones of every other leaf over all the units) (`key_cut = 'auto: r#1'`, and the `NOTICE` names it): `r @> x` over 3,000,000 random ranges beside an address and a 128-character note reads 0.67 million instructions where the superblock cut read 27.9 million (a GiST: 1.64 million), `r && range` 0.87 million where it read 28.3; the index is 6.0% smaller (66.5 to 62.5 MB) and the build 58.8% more instructions (20.4 to 32.4 G); `ip <<= net` and `ip >>= addr` over the same index read 25.6 and 95.7 million without the cut and 23.8 and 93.9 with it. The first look of the chooser lets a time leaf whose predicted growth is up to 15% reach the full sample (it was 7%), which costs a sample of 36 superblocks and decides as before. The chooser of the default `key_cut` prices the dictionary-coded image of a range or an address like its first lane, which a unit of the cut lists in the order of the lane: an index on an address alone (3,000,000 random addresses and a 128-character note) is now cut by `ip#1` with no option (predicted size -6.8%, built -7.4%, 26.6 to 24.6 MB), where it was left without the cut with a predicted growth. An inet or cidr column keeps a third lane, the offset of an IPv4 address inside its block, so `ip = x` is the node and the offset on the lanes and the aggregate no longer reads the dictionary of the segment; the blocks that contain a network (`>>=`, `>>`, `&&`) are found by the lowest bit of the node in a table of 64 slots, and a node of 3-byte values is written bit-packed (at most 1.6% larger) so that its range test has a kernel. Without any cut, 3,000,000 addresses: `count(*)` of `ip = x` reads 15.5 million instructions (323.9), `ip <<= net` 22.3 (23.6), `ip >>= addr` 69.6 (96.3), `ip && net` 83.8 (321.6); rows of `ip = x` 13.6 (21.4). With the cut: `ip = x` returns its rows in 1.34 million instructions, `ip <<= net` 1.56, `ip >>= addr` 3.43, and `count(*)` of `ip = x` reads 0.47 million (580 with the cut before this change), `ip <<= net` 0.52, `ip >>= addr` 2.62. The test of the blocks that contain a network (`>>=`, `>>`, `&&`) on the node lane is one vector kernel (`anc_match`: AVX2, AVX-512 and NEON from one source, `col/kern_anc.h`) where it was a table load a row: `count(*)` of `ip >>= addr` on 3,000,000 uncut addresses reads 53.3 million instructions (69.6), `ip && net` 67.4 million (83.8); on an arm64 core, 1,000,000 addresses, `ip >>= addr` 33.2 million instructions (48.5 with `sift.simd = scalar`), `ip && net` 38.8 million (85.7). On a chunk of IPv4 nodes held as u32 offsets (the common case) the kernel runs on those offsets with twice the lanes and without the int64 pass (`anc_match32`): 3,000,000 uncut addresses, `count(*)` of `ip >>= addr` 28.8 million instructions (53.3), `ip && net` 43.0 million (67.4); an arm64 core, 1,000,000 addresses, 20.8 million and 26.8 million.
- Improved: pg_sift: `sift.json_build_object` reads the arguments of a call site that is typed once (no `VARIADIC` array, no unknown literal turned into text) from the call itself and writes a string literal as the C string it is, where it built an array of values, one of types and one of nulls for every call; the same bytes, the same errors, and a call site that takes a `VARIADIC` array keeps the long path.
- Improved: pg_sift: a statement that reads a table with a `sift_col` index calls `sift.json_build_object` where it names `json_build_object`, in the select list of a window, a sub-select, a CTE or the top level (`EXPLAIN (VERBOSE)` shows the sift function): the same text, with the category and the output function of each argument looked up once for the call site, where the core function asks the type cache for every argument of every row. A call with no argument, `jsonb_build_object`, a statement that reads no such table and a role that may not execute `sift.json_build_object` keep the core function; `sift.enable_agg = off` keeps it for all. No SQL change.
- Improved: pg_sift: the rows a transaction inserts into a `sift_col` index after its first are written to the write buffer a page at a time, one WAL record for a run of rows instead of one for each row, before the commit record (and before the transaction reads the index, starts a savepoint or is prepared); the rows of a savepoint that rolls back are never written, and an `ON CONFLICT` statement writes its rows at once; a buffer can pass `write_buffer_limit` by the rows of one page before it seals. COPY of 50,000 rows into a table with a 6-column index: 245 to 186 bytes of WAL a row, 23,022 to 20,571 instructions a row in the backend (-10.6%), 6,205 to 3,390 on the standby that replays it (-45%); a statement of 100 rows 293 to 234 bytes, 9,138 to 6,343 on the standby; one row a transaction is unchanged (353 bytes, 166.7 k instructions; PostgreSQL 18.6, x86-64, 1,000,000 rows, the table with only its primary key 139 bytes a row for the COPY). No format change.
- Improved: pg_sift: the lookup dictionary of a text or uuid leaf (1,024 strings or more) holds a hash of its strings to their codes (column format minor 25, 5 bytes a string, no option): an equality or `IN` test reads one slot and one block in each segment, and an absent string decompresses nothing. The first equality of a new backend on 1,000,000 rows of text or uuid costs 3.2 to 3.7 million instructions (it was 65 to 329 million: the dictionary was parsed whole) and the later ones 0.3 million (33 million); `sift.col_verify` checks every slot. Segments of minor 25 are refused by 3.5 and older with the `REINDEX` hint; `sift.upgrade_check('3.5')` says to write them again under `sift.col_format_minor = 14`.
- Improved: pg_sift: `order_array_columns = auto` (the default) chooses a date or timestamp column before the others, and takes the statistics it decides from when the table has none: a `CREATE INDEX` on a table of 100,000 rows or more that was never analyzed analyzes it first (the `NOTICE` of the arrays it chooses says so; a `REINDEX` does not, as `ANALYZE` opens the index it rebuilds), where it chose nothing; each column must fit the growth the option allows (a quarter of the index) by itself, so a wide `val` no longer keeps the timestamp out. Default index, no option, random timestamps, tip to this change: `WHERE ts >= x ORDER BY ts LIMIT 10` 2.18 to 0.33 million instructions at 100,000 rows, 18.3 to 0.51 at 1,000,000 (8.7 to 0.67 ms), 54.6 to 0.89 at 3,000,000 (27.6 to 0.49 ms); `min(ts) WHERE ts >= x` 3.78 to 0.44, 34.6 to 0.62, 103 to 1.03 million; `ORDER BY ts DESC LIMIT 10` 1.65 to 0.36, 2.78 to 0.29, 3.12 to 0.50 million; a day of `count(*)` at 3,000,000 rows 28.6 to 10.7 million; a b-tree 0.11 to 0.25 million for the first and the last.
- Improved: pg_sift: `min(x)` / `max(x)` with a range of `x` over the sorted runs (`order_columns`) is priced as the top-N of the runs, which reads the heads of the runs, not as the aggregate over every unit the range touches, so the planner picks it (`min(ts) WHERE ts >= x`, random timestamps: 100,000 rows 3.81 to 3.81 million instructions, 1,000,000 rows 34.6 to 3.84 million (3.80 to 1.26 ms), 3,000,000 rows 103 to 11.3 million (15.8 to 5.5 ms); `max(ts) WHERE ts <= x` 42.1 to 3.86 million and 126 to 11.3 million at 1,000,000 and 3,000,000 rows; a b-tree 0.24 million at every size; the global order of `order_array_columns` is what takes the rest).
- Improved: pg_sift: a calendar step (`ts::date`, `date_trunc(unit, ts)`, `extract(year FROM ts)`) or a `timestamp` / `timestamptz` column of another type compared with a stable operand (`current_date - 30`, `now() - interval '30 days'`, a parameter of a generic plan) is the range of the column that a constant gave, with the bounds found when the scan starts: the rolling windows of a dashboard over 1M rows take 0.3 to 1.0 ms against 60 to 212 ms of the normal plan (`ts::date = current_date - 30` 0.4 ms against 67 ms, `ts >= current_date - 30` 0.5 ms against 60 ms, `ts >= now() - interval '30 days'` 0.7 ms against 136 ms), where the executor ran the comparison for every distinct timestamp (117 to 192 ms). An operand beyond the timestamps (the year 1, the year 200000, `infinity`) saturates the bounds, so the answer is the heap's whatever the parameter.
- Improved: pg_sift: a scan of a `sift_col` index that waits on storage for its units (two reads past shared buffers that took 25 microseconds or more; a scan of cached pages never does) reads the first pages of the kept units that come after the current one through a read stream, as deep as `effective_io_concurrency` allows (`0` reads nothing ahead); runs of neighbouring units are left to the kernel's read-ahead. The rows of every answer are those of the scan before. Test: `col_read_ahead`.
- Improved: pg_sift: a condition `x IN (...)` or `= ANY (array)` of 32 or more integer, date or timestamp keys is tested on the rows of a unit with a hash table of the keys, not a bisection of the sorted keys for each row: an index-only scan of 1,000,000 rows with 10,000 ids takes 469 million instructions where it took 662 million (a b-tree 151 million); lists of fewer keys, and lists that the point directory answers without reading the units (1,000 and 3,000 ids: 22.6 and 81 million), are not changed.
- Improved: pg_sift: a scan that unnests an array or a map reads the element offsets of a unit's rows from the scan layer, which already holds them for reading the elements, instead of summing the counts again (the wildcard aggregate of a 12-key map over a 1,000,000-row index: 11.2 M instructions fewer, 4.6% over a 2-year window, 6.7% over 90 days, 7.9% over 7 days; results unchanged).
- Improved: pg_sift: a `LIKE` with a `_` or several `%`, `ILIKE`, `NOT LIKE` and a regular expression (`~`, `!~`, `~*`) on a `text` column of a `sift_col` or `sift_table` index are no longer run on every string of a unit's dictionary: the literals of 3 bytes or more that every match holds (`shop17` and `example` of `LIKE '%shop17_example%'`, `shop17.example.com/item/` of `~ 'shop17\.example\.com/item/[a-c]'`) are found first by the substring kernel, and the pattern runs on the strings that hold them. User instructions of one backend over 1,000,000 distinct URLs, before and after: `~ 'shop17\.example\.com/item/[a-c]'` 20,196 M to 1,772 M, `~ '7f3a9c[0-9]'` 8,042 M to 962 M, `LIKE '%shop17_example%'` 2,992 M to 1,145 M, `ILIKE '%SHOP17_EXAMPLE%'` 4,313 M to 1,174 M, `NOT ILIKE '%item/a%b%'` 4,225 M to 1,426 M; 1,000,000 sentences `~ 'the (and|of) it'` 9,540 M to 229 M; 1,000,000 codes `~ '^[0-9A-F]{2}-0000'` 3,529 M to 52 M (the bench case `hits_regex_prefilter`, `count(*) WHERE url ~ 'google\.com/1[0-9]'` on 1,000,000 rows, 8,774 M instructions of PostgreSQL's plan against 266 M). A pattern the prefilter cannot use keeps the whole dictionary at the price of a sample of 256 strings: `~ 'item/[0-9a-f]{3}c9'` (the only literal, `item/`, is in every string) 14,769 M to 14,750 M, `~ '(7f3a9c|e4b2d1)'` (an alternation has none) 12,693 M to 12,743 M. The rows are the normal plan's (test `col_text_prefilter`: 702 statements of 78 patterns, 3 columns, C, POSIX, ICU and tr-TR collations, 2 storages, strings with U+0130 and U+212A, patterns the regex compiler refuses); no format change.
- Improved: pg_sift: `LIKE '%suffix'`, `starts_with(col, 'abc')` and `col ^@ 'abc'`, `strpos(col, 'part') > 0` (and `= 0`, `<> 0`, `position('part' IN col) >= 1`) and a regular expression that is a literal (`col ~ 'shop17\.example'`, `'^https'`, `'\.com$'`, `'^abc$'`, `!~`, `~*` of a substring) on a `text` or `varchar` column of a `sift_col` or `sift_table` index are tests of the encoded strings of the dictionary (the substring kernel, a suffix or prefix compare, a bisection for a prefix) where the executor evaluated the function for each distinct value: over 1,000,000 distinct URLs `LIKE '%0e41b2'` 194 ms to 71 ms (2,821 M to 984 M instructions), `strpos(url, 'item/a') > 0` 153 ms to 73 ms (1,682 M to 989 M), a literal regular expression 1,439 ms to 90 ms (19,588 M to 992 M), `starts_with(url, 'https://shop1')` 241 ms to 139 ms (2,165 M to 971 M), a short code `LIKE '%-1A'` 43 ms to 6.0 ms (710 M to 71 M). A regular expression with a dot, a class, a group or an alternative, an empty literal, a comparison of `strpos` with another number and a non-UTF-8 multibyte database keep the normal plan. The rows are the same.
- Improved: pg_sift: `ILIKE '%part%'` of an ASCII part on a `text` or `varchar` column of a `sift_col` or `sift_table` index in a UTF-8 database is the substring kernel with the ASCII letters of the strings compared in lower case (AVX2, AVX-512, NEON), where the executor lowered and matched each distinct value: `url ILIKE '%SHOP17.EXAMPLE%'` over 1,000,000 distinct URLs 324 ms to 86 ms (4,167 M to 1,074 M instructions), a short code 96 ms to 4.7 ms (1,343 M to 36 M), a sentence 230 ms to 61 ms (2,897 M to 639 M). The two characters whose lower case is ASCII (U+0130, U+212A) are read as what `lower()` makes of them in the collation of the column when the part has an `i` or a `k`; a collation that does not lower A-Z to a-z (Turkish), a non-ASCII part, a wildcard inside the part and `NOT ILIKE` keep the normal plan. The rows are the same.
- Improved: pg_sift: `NOT LIKE '%part%'` on a `text` or `varchar` column of a `sift_col` or `sift_table` index is the complement of the same dictionary test as `LIKE '%part%'` (one SIMD pass over the strings of the dictionary, NULL rows stay out) where the executor called the operator for each distinct value: `count(*) WHERE url NOT LIKE '%shop1%'` over 1,000,000 distinct URLs 211 ms to 84 ms (2,629 M to 992 M instructions); a ClickBench query of a substring match and a negated one over a title and a URL (3,000,000 rows of `hits`) 852 ms of the normal plan against 1,501 ms before and 475 ms now. `NOT LIKE` of a prefix or a suffix is unchanged.
- Improved: pg_sift: `LIKE '%part%'` on a `text` or `varchar` column of a `sift_col` or `sift_table` index (a literal without `_`, `\` or an inner `%`, a deterministic collation, a UTF-8 or single byte database) is tested on the bytes of the dictionary, in one pass of a SIMD kernel (AVX2, AVX-512, NEON) over all its strings, instead of a call of the operator for each distinct value; the units whose dictionary lacks the part are not read, and strings stored inline in a unit use the same kernel. `ILIKE`, `NOT LIKE` and regular expressions are unchanged.
- Improved: pg_sift: the strings of a text column that a `sift_col` index stores inline (over 255 bytes, or more distinct text than the segment dictionary holds) are wrapped with LZ4 instead of zstd where the default formula says so: a substring test on 3,000,000 sentences reads 683 instructions a row, not 2,371 (a heap scan 4,286), an equality 617, not 2,305, a GROUP BY on the column 8,693, not 10,381 (1,000,000 notes of 300 bytes: 1,820 M instructions against 15,775 M for the heap plan); the index of those sentences takes 54% more bytes, the 6-column one 23.7%, and a build 0.2% to 0.4% more instructions. Text that repeats stays in the dictionary, text that LZ4 cannot shrink (hex) and text that compresses so well that the extra bytes cost more than the unwrap saves keep zstd. A NOTICE at CREATE INDEX names the column and `compression = 'auto; col=zstd'` (then REINDEX) keeps zstd. The segment dictionary of a column whose parsed dictionary no longer fits `dict_cache_kb` takes LZ4 blocks by the same formula: a mixed IPv4 and IPv6 network column cut by `key_cut` answers `ip = x` in 266 M instructions at 3,000,000 rows, not 1,181 M, for 18.9% more index; a cut build now tells the writer the rows of the index, so the signatures and this rule see the real size of its dictionaries. A segment with LZ4 blocks is of format minor 29.
- Improved: pg_sift: the dictionary of a text column of a `sift_col` index that no longer fits the cache of parsed dictionaries of a backend (`dict_cache_kb`: the leaf's strings, parsed and scaled to the rows of the index, pass it, or one segment's alone pass half of it; at least 1 MiB of strings a segment) also holds a 256- to 16,384-bit trigram signature of each block of 64 strings (column format minor 29, `REINDEX` to get it; an index that fits is unchanged), so that `LIKE '%part%'` with a part of 3 bytes or more reads only the blocks that may hold it. Without signatures the dictionaries of an index of 4 columns (random-hex URLs of 63 bytes, codes, sentences) stop fitting the cache between 500,000 and 1,000,000 rows and every query inflates all the blocks: user instructions a row of a rare 6-byte part 48, 986, 982 and 892 at 500,000, 1,000,000, 2,000,000 and 3,000,000 rows; with them 49, 132, 127 and 126 (3,000,000 rows 2,677 M to 378 M instructions), the same as before at 500,000 rows. The index grows with the leaves that outgrow the cache: 18.39 to 18.53 MB at 500,000 rows, 36.45 to 40.33 MB at 1,000,000, 109.09 to 138.07 MB at 3,000,000 (random hex does not compress: the worst case; the 25-column index of 3,000,000 ClickBench rows grew 409.0 to 415.3 MB, +1.5%, with every dictionary of 4 MiB signed); a rare 12-byte part 122 ms to 2.8 ms (a GIN `pg_trgm` index 1.9 ms); a part found in most blocks reads every block as before (a frequent part of the sentences at 3,000,000 rows 576 to 674 instructions a row). A 3.5 library refuses such a segment.
- Improved: pg_sift: one `NOTICE` per `CREATE INDEX` of a `sift_col` index lists the text columns whose strings or dictionary the compression formula stored with LZ4 instead of zstd, and those that kept zstd when the benefit of LZ4 came within a quarter of its cost (hex and ids, which LZ4 does not shrink, are not named), with what a test reads a MB of text with each algorithm, the bytes LZ4 adds and the benefit against the cost (sentences: 17.1 against 5.2 M instructions a MB for about 350 kB (+54%), benefit 24 against cost 18), and the override: `compression = 'auto; col=zstd'` then `REINDEX`, `'auto; col=lz4'`, and `sift.col_format_minor = 14` for the LZ4 blocks of a dictionary that releases before 3.6 refuse. It replaces the notice per column; a parallel build reports once.
- Improved: pg_sift: where `key_cut` is chosen by default for an index with a date or timestamp leaf, the rows of a key are sorted by that leaf too (`key_cut = 'auto: key, date'`, and the `NOTICE` of an advised cut names both columns), unless the sample predicts the index more than 1% larger: a key with more rows than a unit is split by date, so a window of dates reads fewer of its units (a key and a 2-month window: 9 units read against 2 at 125,000 rows) and the index is 21% smaller (3,000,000 rows, 6,000 keys: 76.9 to 60.8 MB), the wildcard aggregate of a jsonb map over a 2-year window 5.0% and over 90 days 7.1% fewer instructions; the build is 11% longer. A `REINDEX` of an index built before decides again; `key_cut = 'off'` or a list of its own is never changed.
- Improved: pg_sift: `=`, `<>`, `IN` and `NOT IN` of a `uuid`, `bytea`, `inet`, `cidr`, `macaddr`, `macaddr8`, `bit` or `varbit` column against constants are tests of the source on the stored image, like the `numeric` and `money` ones (a conjunct of the WHERE), where the executor evaluated the operator on a per-unit dictionary of datums: `count(*) WHERE uuid_col = C` over 1M distinct values takes 3 ms against 400 ms before (the normal plan 41 ms), `bytea` and `inet` 3 ms against 47 and 49 ms of the normal plan, a 3-value `IN` 9 ms against 141 ms of the normal plan, and the planner chooses the scan where it priced it dearer than the heap.
- Improved: pg_sift: a comparison of a calendar step of a `timestamp` or `timestamptz` column with a constant (`ts::date = D`, `date_trunc('month', ts) >= M`, `extract(year FROM ts) = Y`; `=`, `<>` and the ordering operators, any unit of `date_trunc` from second to millennium, a constant of the years 100 to 9900) is a range of the column that the kernels compare with the zones of its units, where the function ran for every distinct timestamp: `count(*) WHERE ts::date = D` over 1M rows takes 1.8 ms where it took 124 ms (the normal plan 31 ms), and 3.9 ms over 3M rows (normal plan 72 ms). The bounds are found when the scan starts, by a bisection with PostgreSQL's own function in the time zone of the session, so a zone that skips a day, repeats an hour or moves the clock by half an hour (and a generic plan run under another zone) gives the rows of the normal plan.
- Improved: pg_sift: planning a statement that reads a `sift_col` index takes fewer instructions: the walk that checks a CTE for writes and volatile functions is one pass, the sublinks of a query level that has none are not looked for, and the share of the units a list of keys leaves is computed once for each leaf and list of a plan.
- Improved: pg_sift: a `sift_col` index keeps the cardinality and `array_length(x, 1)` of every array column as two integer columns next to the array (the reloption `array_lengths`, `auto` by default: `CREATE INDEX` and `REINDEX` write `on` for an index with an array column, an index built before it keeps the images alone), so that `cardinality(x)`, `array_length(x, 1)` and `x IS [NOT] NULL` in a filter, `FILTER`, `CASE`, group key or aggregate are read without decoding the arrays: `count(*) WHERE x IS NOT NULL AND cardinality(x) > 0` over 1,000,000 rows of arrays of up to 30 elements takes 0.91 ms and 10.8 M instructions against 136 ms and 1,047 M for the normal plan (538.9 ms and 4,088 M for the index before); the index grows 10% on six array columns (73.0 to 80.5 MB) and a build, merge or promotion costs 12 to 17% more instructions. A library before 3.6 refuses such an index (REINDEX).
- Improved: pg_sift: a unit that the executor's own conditions leave no row (an OR of ranges, an expression of a column, which the source does not apply and the zones cannot refuse) is left after those conditions: the counts, owners and chunks of its element group are no longer read for it. The rows of every answer are those of the scan before.
- Improved: packaging: the smoke test of every arm64 package runs `scripts/check-aarch64-baseline.sh` on the installed `pg_sift.so` (binutils in the arm64 test containers), so a `-march` or `-mcpu` in a distribution's flags, or a newer-than-ARMv8-A instruction from any other cause, fails the release before it publishes.
- Improved: pg_sift: a containment that holds in two ways (`&&`, `<<`, `-|-`) prunes the units that neither way can hold, from the union of the zone tests of its ways, where it read every unit (the networks inside the constant or the ones that contain it; the ranges that end where it starts or start where it ends); one evaluator (`sift_colw_prune`) for the index scan and the aggregate scan, which gives a containment test its alternatives (`sift_colw_alts`). 1,000,000 random IPv4 and IPv6 networks with random masks, `count(*) WHERE ip && '2001:db8:1::/48'`: 67 of 92 units pruned (0 before), 35.7 M to 15.6 M instructions, 2.26 ms to 1.10 ms, 1,008 to 334 shared buffers (all hit; a cold run reads the same 895 pages as before, the units it skips are not read); `ip << '2001:db8:1::/48'` 27.8 M to 3.3 M instructions, 1,008 to 111 buffers; with `key_cut = 'off'` `<<` goes 157 M to 24.8 M and `&&` keeps its 133 M (the zones of the superblocks hold nearly every node); the index is the same size (16.1 MB); `col_key_cut_range` compares the rows of `&&`, `<<` and `-|-` with the heap
- Improved: pg_sift: `a = x OR a = y` (any number of equalities of one column with one operator, constants, parameters or stable expressions) on a network or a numeric column of a `sift_col` index is read as `a IN (x, y)`: the source takes the list as one test and reads the units of the values, where the OR was evaluated on the rows of every unit. 500,000 random IPv4 addresses, `sift_col (ip, r)`, `count(*) WHERE ip = x OR ip = y`: 2 of 43 units (43 before), 837 M to 0.44 M instructions, 157 ms to 0.15 ms, 967 to 589 shared buffers warm, and cold 230 pages (1.9 MB) to 30 pages (0.25 MB) read (the IN list of the same values 0.42 M; a b-tree 0.15 M, a GiST 0.39 M on a copy); the same OR on a range column stays at 1,350 M instructions, like its IN list (1,072 M), because a range has no image the source compares (docs/CONTAINMENT.md); test `col_or_equalities` compares the rows of the OR with the heap after a write, an update, a delete, a merge and a `REINDEX`
- Fixed: test tools: the scaling gate (`scaling_slope`) failed a flat query on a hosted macOS runner for a pass the host had lifted (`ORDER BY .. LIMIT` over 2 units read: the last two sizes took many times longer than the first three, in one pass; the same test passed in another lane of the same job). A slope that fails is now measured again, up to twice, and each size keeps its best time, so a pass lifted by the host is forgiven and a growth that every pass shows still fails; the bound is unchanged.
- Fixed: pg_sift: the test counter of the dictionaries parsed (`loads` of `sift._test_col_dict_cache`) did not count a dictionary that went to the shared ring, which is read in place and never kept in the private cache: `col_text_tsig` saw no parse after the in-place read landed. It counts every parse of a segment now, and the test also checks that the next statement reads the dictionary in place (`dict_shared_hits`). No change to the rows or the plans.
- Fixed: pg_sift: `SELECT .. FROM fact JOIN dim ON dim.g = fact.g WHERE dim.label = 'x' ORDER BY fact.k LIMIT n` over a `sift_col` index with an order array on `k` no longer crashes the backend: the keys of the joined table's rows, which the top-N holds as a set of the scan, reached the code that reads an `IN` list of constants (the page-after-a-key and global-order conditions) with no list to read; that predicate is left to the units, as the set was before the `IN` list of a group was read from the order array. Test `col_topn_join` (the 17 shapes of the join top-N, which crashed 8 of them).
- Fixed: pg_sift: `CREATE INDEX .. USING sift_col (ip)` on a column of IPv6 addresses gets the default `key_cut` (`ip#1`, the node of the network): the offset lane of a network (lane 2, NULL for every IPv6 row) was counted as a leaf that follows the order of the table, which made the sample refuse the cut; 1,000,000 random IPv6 hosts, `count(*) WHERE ip = x` 6.28 M to 0.67 M instructions, `ip <<= /48` 2.75 M to 0.90 M, `ip >>= x/128` 20.0 M to 6.3 M (3,000,000 rows: `<<=` 7.4 M to 0.69 M, `>>=` 260 M to 170 M; `=` stays 220 M, the whole-dictionary inflate of the segment); test `col_key_cut_range` has the IPv6 table
- Fixed: pg_sift: the table of the parsed dictionaries shared between backends could hand a reader the dictionary of another segment: a backend that copied its record and published it late, after the ring wrapped over its place and another backend filled it with a dictionary of the same size, named a place that held the other record, and the copy checked (its sums are the other record's). A record now names its key and the reader compares it (the layout changed, so the table is named `sift_col_dict_2`: a server that was not restarted keeps the memory of the old one until it is).
- Fixed: build: `src/col/amscan.c` called `DatumGetTimestampTz` without including `utils/timestamp.h`, which the headers of PostgreSQL 19 (PGDG `19~rc1`, built with `-Werror=implicit-function-declaration`) do not bring in: the library did not compile against them.
- Fixed: pg_sift: the lookup of the write buffers of a `sift_col` index (an equality or `IN` list on a plain leaf, the previous entry) passed a null pointer to `memcpy` when no row of the leaf overflowed, undefined behavior that the sanitizer build aborted on (`DELETE FROM e WHERE id IN (..)`); the copy is skipped when there is nothing to copy.
- Fixed: pg_sift: a `sift_col` index of a date, timestamp or timestamptz column is an index qual for a bound of another of those types (`d < x + interval '1 day'`, `d >= now() - interval '30 days'`, `ts >= current_date`), in the session's zone as PostgreSQL compares them; before, such a statement scanned the table: a day of 3,000,000 rows 1,476 M instructions and 75,000 buffers to 9.97 M and 1,166 (b-tree 6.15 M), a month 1,576 M to 142.9 M (b-tree 141.1 M)
- Fixed: pg_sift: a statement that aggregates and reads a column a `sift_col` index of the default class does not hold (`avg(score) FROM r WHERE d >= x AND d < y` on `sift_col (d)`) can use the index or bitmap scan of its filter, not only the sequential scan: a month of 3,000,000 rows, 1,470 M instructions and 75,000 buffers to 54.9 M and 3,104 (b-tree 53.8 M); a statement SiftAgg can answer keeps its plans
- Fixed: test tools: the leak gate of the cluster (`leak_cluster`) reported a session that had just disconnected as a leak: the backend still held the locks of the temporary tables it was dropping on exit (`session N idle holds AccessExclusiveLock on pg_temp_0.r`), which a loaded host showed in `col_wbuf_crash` (1 of 2 full runs on an 8-lane macOS run, 0 of 1 on Linux). The check now waits up to 5 s for such a session to go before it reports one.
- Fixed: pg_sift: the sample that decides the default `key_cut` is 32 superblocks at least whatever their rows, not at most 400,000 rows: a table with over 12,500 rows to a superblock (rows under about 160 bytes) ended at "the units are too large for a sample" at any size; 3,000,000 rows of 56 bytes now get a decision (the first look of 8 superblocks adds 6.8% to the build, 465 M of 6,867 M instructions; the cut of its date is predicted at +51%, measured +25%, over the 25% limit, so it stays opt-in)
- Fixed: pg_sift: the planner priced the rows a `sift_col` index hands over at 0.5 units each and the values it filters at 0.009, 40 and 4 times what they cost (10 to 12 ns a row, 1 ns a value), so a row-returning window of a twentieth or more of the table took the sequential scan: `d >= current_date - 30` on 3,000,000 rows (150,000 rows) 1,938 M instructions and 75,000 buffers to 412 M and 4,889 (b-tree 405 M), three months 1,751 M to 407 M, a year (600,000 rows) 2,677 M to 1,621 M; a week and a month keep the bitmap scan (38.1 M, 142.2 M); the index-only emit cost keeps its third of the tuple scan's, so the plans of the point lookups do not change; one join of col_ioss_columns reads both sides by index-only scan
- Fixed: pg_sift: the read-ahead of a scan (`sift_col_kept_map`) copied the map of the units the scan keeps with a length rounded up to a multiple of eight words, 64 bytes past the end of the map the caller allocated (AddressSanitizer: heap-buffer-overflow, in a scan with zones that skipped superblocks); it copies the words the map has.
- Fixed: pg_sift: a grouped scan of a `sift_col` index of about 100,000 rows took the 2 workers PostgreSQL gives a 34 MB heap, which cost it more than they saved (a group by key and an unnest under a date window: 5.6 and 4.7 ms with them, 3.4 and 2.3 alone; 32.2 and 32.6 million instructions, 20.6 and 21.5 alone). The start of the workers is priced at 1,100 estimate units a millisecond, not 750: scans of 1,000,000 and 3,000,000 rows plan the same workers and cost the same instructions (within 0.1%).
- Fixed: pg_sift: a sum of a CASE over a column of a lookup table and a numeric measure (`sum(CASE WHEN d.name = 'Sun' THEN s.price END)` grouped by the week of a date table, the seven weekday sums of a weekly report) is evaluated by PostgreSQL for each distinct pair of the date and the price, nearly every tuple, 2,000 instructions an entry and 570 more for each sum, and the planner priced none of it: the scan took 1,500 ms where the normal plan took 700 ms with one sum, 2,700 against 1,100 with seven over 2.9 million rows; the entries are in the price, and so is a joined tuple past the first of a lookup table of a fact table read from a dimension scan (1,330 instructions measured), so the normal plan is chosen (7 sums: 16.4 billion instructions with the scan, 11.4 with the normal plan); the plan text of the 156 statements of the lab corpus and of the public-style sets is the same.
- Fixed: pg_sift: a probe of a `sift_col` index whose key is a parameter of the parent query (`EXISTS (SELECT .. WHERE t.k = outer.k) OR EXISTS (..)`, a scalar subselect) was chosen over the hashed subplan whenever a few correlated filters made the planner expect a handful of outer rows, and ran for each of the thousands that came: a TPC-DS customer query took 2.9 s with the index against 92 ms with `sift.enable_col_scan = off` (a probe is 0.3 ms, the pass of the hashed subplan 90 ms). A probe of that kind is now priced at 90% of the sequential scan of the table at the least, so the hashed subplan wins where it fits in memory and the probe is still the plan where none does (18 s of sequential scans for each row against 0.2 s)
- Fixed: pg_sift: a scan grouped by expressions of an integer column with as many values as rows (`GROUP BY ip, ip - 1, ip - 2, ip - 3` over 1,000,000 rows) was priced at a twentieth of its cost and chosen over the normal plan, 1.1 s against 0.56 s: the id each value takes in the output table was not priced for an integer column, the expression was priced for the values the groups have only when it filtered, and the credit for the expressions the normal plan prices at one operator counted an integer subtraction as a numeric one.
- Fixed: pg_sift: a scan whose filter leaves a few rows to a unit (`counter_id = 62` and a month: 1 row in 800) was priced as if the table of the group key were given every entry of the dictionary of each unit, though only the entries of those rows are mapped: the scan of a text key over 1,000,000 rows took 1.8 ms and was refused at a price that said 38 ms.
- Fixed: pg_sift: the planner did not price the probe of a lookup table for each tuple the filters keep, nor the rest of an ON clause that compares the scanned row's values with the table's (a join with `(a.k = 3 AND f.qty < 10) OR (a.k = 7 AND f.qty < 20)`), which PostgreSQL evaluates once for each distinct combination of the values: such a scan was priced at a fifth of its cost and chosen over the normal plan, 3 to 11 times slower on the shapes measured (a 300,000-row fact table: 12.9 ms against 1.1 ms). The same join no longer counts its join keys as cutting the units, as its table is loaded whole.
- Fixed: pg_sift: a lookup table that is a plan (a `UNION ALL` of two tables) was priced a few operators a row and the tuples of its join at the rows of the scanned table, so the scan of a dimension joined to it for seven numeric sums of a `CASE` was chosen at 540 ms against the normal plan's 215 ms (500,000 rows in the plan, 2,000 in the dimension): the tuples are now the rows PostgreSQL expects of the join, and a row of the plan costs 11 operators and 32 for each numeric column (the same price as the load of a numeric column from the leaves); the plan the scan loses to is kept (215 ms, 1,958 against 2,517 million instructions).
- Fixed: pg_sift: the planner priced the load of a lookup table from the leaves of a `sift_col` index at the same cost for every column, where a numeric column costs 260 ns a row (the value is built and allocated for each row) and an integer one 15 ns, so a join whose scan was the small filtered table and whose lookup table was the 2.9-million-row fact with four numeric columns won on paper (cost 63,241 against 102,043) and took 4,700 ms against 400 ms of the normal plan. Each numeric column of the load now adds 32 operators to the price of a row, and the normal plan is kept for those joins (scale-1 decision-support queries: 409 ms against 5,046 ms with the scan, and six more of the same shape within 10% of the normal plan).
- Fixed: pg_sift: VACUUM's space step left the file of a `sift_col` index at twice its bytes after a merge whose output was larger than the blocks its inputs had freed (3,000,000 rows, 8 columns: three segments of 99.1 MB merged into one of 99.9 MB, the file 203 MB and 199 MB after VACUUM, 99 MB free; the merge's output differs by a few percent from run to run, and in some runs it was larger than the hole): the shift of a segment gives the free blocks below it to its data runs from the bottom up, and the header page, which lies last in the segment, found none and held the end of the file. The header now takes the lowest free block below it first, and moves alone when the data cannot (3,000,000 rows: 199 MB to 99 MB in one VACUUM of 8 ms, merge time unchanged). Test: `col_space_shift`.
- Fixed: pg_sift: a scan that groups, or counts distinct values, by a float key whose values have a short mantissa (`1.5`, `100.0`, whole numbers kept as `float8`) probed its key table at the middle bits of a multiplicative hash, which such values leave constant, so 1,000 groups shared a handful of slots: 2,049 million instructions for 1,000,000 rows (137 ms) where the same groups of an integer cost 109 million (7 ms). The slot is now the top bits of the product: 86 million instructions (7.4 ms), the same groups in the same order (`agg_key_float_hash`).
- Fixed: pg_sift: the choice between a shared CTE and its inline plan did not look into the scalar subqueries of the statement, which are subplans of the plan and not nodes of its tree: for a CTE that scalar subqueries read, the shared plan was priced as if it did no work, and the scans lost to it however much it cost. The subqueries are weighed as the other readers of the CTE are.
- Fixed: pg_sift: the groups a scan reports to the plan above, for an unnest over a lookup PostgreSQL has no statistics for (its rows are the floor of one row times the 100 of the function), are the combinations of the distinct values the index holds for the keys, at most the geometric mean of that floor and the elements of the table's rows, where they were the few PostgreSQL's estimate for the keys of a document gave: a statement whose scan makes tens of thousands of groups was planned above as if it made a few hundred, with a nested loop over them.
- Fixed: pg_sift: the planner chose the index scan for `= ANY`, `&&`, `@>` and `cardinality()` over a column of long arrays (123 bytes on average, 1,000,000 rows) where the heap plan is 2 to 4 times faster (4,090 against 1,050 million instructions for `cardinality(a) > 0`, 5,140 against 1,940 for `= ANY`): the unit hashes, interns and inflates every new array before a test reads it, which the cost left out. A new entry of an array image leaf now costs 6 operators for each byte beyond 64 of the column's average width, for the leaves a test, key or aggregate reads (not those whose null flags are all it reads); the shapes that read arrays of 64 bytes or fewer, and the ones the index wins by instructions, keep their plan.
- Fixed: pg_sift: the planner priced the dictionaries of a field of a list or the keys of a map (`scores.*`, `items[].name`) from the statistics of the jsonb column they are read from, which count every document as a distinct value, so a scan that groups by such a key was priced several times its cost and the normal plan was kept where the scan ran in a fraction of its time; the keys of an object and the fields of a list have no statistics of their own, and the scan is priced without that charge.
- Fixed: pg_sift: a scan that joins a lookup PostgreSQL has no statistics for (a CTE or sub-select of ids it expects to hold one key) is priced for workers at the mean of that guess and the column's distinct values whatever units the guess leaves, as its scan price already was where the guess left more than a handful: a statement whose lookup held tens of keys and whose scan read most units ran its second node in one process (the ranking scan of a two-scan statement, 3.4.0 had 4 workers on it)
- Fixed: pg_sift: the automatic rewrite of a `sift_col` index waited for `sift.col_auto_merge_wal_pct` of WAL even when the segments to merge were those of its own build: a parallel build leaves one set of segments per worker, all covering the same superblocks (1,000,000 rows, 4 workers: 4 segments, 33 units listed for 11 covered), and the budget counts from the write of the build, so the index stayed that way until the server had written 20 times its size (`sift.col_maintenance_due(index, true)` NULL, `(index, false)` `fragmented units`). The segments of at least half the bytes of the largest that overlap each other (listed units at `sift.col_auto_merge_fragmentation` times those they cover) are the first rewrite, not churn, and the budget does not hold it; the rewrite itself is asked for by the next VACUUM of the table, by a scan (once per `sift.maintenance_naptime`, as a scan asks for a VACUUM), or from `sift.col_maintain(index)`, so a table nothing changes gets it too; a table with `autovacuum_enabled = false` does not. The first rewrite is partitioned by `sift.vacuum_rewrite_limit` like any other. Tests: `col_build_peers` (SQL and TAP).
- Fixed: pg_sift: a scan whose workers the planner prices for the rows it reads kept one process only when the units it keeps times the average unit of the index were fewer than `sift.agg_par_min_rows`, and the average unit of an index built by workers is several times the real one (its statistics count the superblocks the segments span, not the units they hold), so a scan of a few thousand rows started workers that cost more than the scan (4.2 ms with 2 workers, 0.2 ms alone, 4,500 rows). The floor now compares the rows of the units the scan keeps, which the unit directory holds.

## 3.5.3 - 2026-10-28

- Improved: packaging: the release attaches an `.rpm` for Amazon Linux 2023 (`pg_sift_N-X.Y.Z-1.amzn2023.{x86_64,aarch64}.rpm`, N = 17 or 18), built and smoke-tested in `amazonlinux:2023` containers of both architectures (4 more packages, 16 in all): the distribution's PostgreSQL keeps its files in `/usr/lib64/pgsql` and `/usr/share/pgsql`, where the PGDG `el9` package puts them in `/usr/pgsql-N`, so that package installs there but `CREATE EXTENSION` does not find the extension. One spec (an `%if 0%{?amzn}` block), and `scripts/package.sh matrix` is now the source of the release matrix (`verify` requires the same packages).
- Fixed: packaging: the `.deb` needed glibc 2.38 (`libc6 (>= 2.38)`: it was built on Debian 13) and so did not install on Debian 12 or Ubuntu 22.04 (`GLIBC_2.38 not found`); it is built on Ubuntu 22.04 (glibc 2.35, GCC 11) now and smoke-tested in fresh Debian 12, Debian 13, Ubuntu 22.04 and Ubuntu 24.04 containers, so one package serves all four.
- Fixed: build: on Linux/AArch64 the library is compiled for plain `armv8-a` (NEON) whatever flags the server's `pg_config` carries; a build against Amazon Linux 2023's PostgreSQL inherited `-march=armv8.2-a+crypto` and used LSE atomics directly, which do not run on every Arm CPU.
- Fixed: pg_sift: a text column of a `sift_col` index tested with `= ANY ($1)` or `<> ALL ($1)` crashed the backend (signal 11) when the parameter was NULL and the plan generic (a prepared statement once its plan turns generic, a PL/pgSQL variable), and a server that clobbers freed memory (assertions on) answered no row for a packed or expanded array: the hash of the list is built at the first test from the array of that test, which the evaluation freed. The list is now copied, flat, where the hash lives, and a NULL list loses the hash. Workaround on earlier releases: `SET sift.enable_agg = off`.
- Fixed: pg_sift: the build with GCC 14.2 on arm64 (Debian 13) ended with an internal compiler error in the chunk codec (`cc_put_neon`: a by-value parameter of a vector type aligned below its natural alignment); the type is the natural one now, as it is only ever a value. The machine code of the AVX2, AVX-512 and NEON kernels is byte for byte the same.
- Improved: packaging: the release attaches the `.deb` (`postgresql-N-pg-sift_X.Y.Z-1_arm64.deb`) and the `.rpm` (`pg_sift_N-X.Y.Z-1.el9.aarch64.rpm`) for arm64 too, for PostgreSQL 17, 18 and 19, built on the hosted arm runner with the compiler's default `armv8-a` baseline (no `-march`, so they run on any aarch64 Linux) and smoke-tested in the arm64 images of the same five distributions, where it attached x86-64 only (6 packages, now 12): the Debian recipe says `Architecture: any` and the spec `ExclusiveArch: x86_64 aarch64`, so there is no copy per architecture, `scripts/package.sh` (`ARCHES`) lists them, and `scripts/package.sh verify` refuses a set that lacks one.
- Improved: packaging: the release attaches the `.deb` and `.rpm` for PostgreSQL 17, 18 and 19 (`postgresql-N-pg-sift_X.Y.Z-1_amd64.deb`, `pg_sift_N-X.Y.Z-1.el9.x86_64.rpm`), each smoke-tested in fresh Debian, Ubuntu, Rocky and Alma containers, where it attached the PostgreSQL 18 pair only (2 packages): one Debian recipe and one spec serve every major, `scripts/package.sh` (`PGS`) lists the majors, and the release refuses to publish a set that lacks one (`scripts/package.sh verify`).
- Improved: tools: `scripts/check-aarch64-baseline.sh` disassembles a `pg_sift.so` for AArch64 and fails on an instruction of an extension later than ARMv8-A and NEON (LSE atomics, CRC32, dot product, SVE, crypto and the others; the libgcc outline-atomics helpers, behind a runtime check, are allowed). The libraries of the six arm64 packages of 3.5.2 pass it.
- Tests: `agg_visibility_refusal` is deterministic: the statistics of the table are those of every row (the cost of the other plan moved by 15% from one `ANALYZE` to the next), and the decisive key bound sits 16% from that cost, not 1%; it still asserts that the scan is refused for its cost.
- Fixed: test tools: `make check` on macOS arm64 failed in `pg_upgrade*`, `replica_mixed`, `leakcheck_gate`, the column crash and batch tests, `scaling_slope` and 5 checks of `scripts/lint.sh`. The scripts spelled `paste -sd` and `seq -s` the GNU way, read file sizes with `stat -c`, compared the padded output of `wc`, used a `sed` pattern with `\?`, `timeout` (missing before macOS 26), `-march=x86-64` in the MariaDB build and the module suffix `.so` (macOS: `.dylib`, so the upgrade tests ran the library the user had installed instead of the one under test); the benchmark generators and the scaling gate used Python 3.12 f-strings and `X | None` annotations that Python 3.9 of macOS rejects. The older releases that the tests build now see the lz4 and zstd directories of Homebrew. Only the test scripts change; the library is the same.

## 3.5.2 - 2026-10-28

- Fixed: pg_sift: an aggregate scan that read several wide leaves (numeric beyond 64 bits, interval, money) for the selected rows of its units, with a text or other dictionary key beside them, gave rows to the wrong groups or crashed the backend (signal 11): the eight slots of the executor's per-dictionary maps were handed out round robin, so a dictionary made for the read of a wide leaf (one for every unit) took the slot of a key map still in use; the least recently used slot goes now (test `agg_dict_slots`).
- Fixed: pg_sift: a lookup table loaded from the leaves of a `sift_col` index crashed the backend (signal 11) when it had a text column of 1,024 or more distinct strings in a segment, whose dictionary the segment keeps in blocks and the backend has not loaded: the load read the strings through the arrays of the whole dictionary, which such a dictionary does not have yet. `SELECT county, sum(x) FROM facts, addresses WHERE facts.k = addresses.k GROUP BY county` killed the server on the first statement of a connection with 1,100 counties. The string of an entry is now read with the call every other reader of a dictionary uses, and the test of a text condition of the table loads the dictionary first; the rows are those of the normal plan (`agg_lookup_lazy_dict`).

## 3.5.1 - 2026-10-27

- Fixed: pg_sift: a backend crashed (signal 11) when a parallel aggregate scan with a correlated lookup (`LEFT JOIN LATERAL (..) LIMIT 1`) was planned with workers and none could be launched because the pool of parallel workers was spent: the scan kept pointers into the parallel shared memory, PostgreSQL destroyed the parallel context and the leader read the unmapped segment (`memo_find`); the pointers are dropped before the context is destroyed (test `agg_par_no_workers`). Until the update: `max_parallel_workers_per_gather = 0` or a larger `max_worker_processes` and `max_parallel_workers`.

## 3.5.0 - 2026-10-27

- Fixed: pg_sift: a partitioned table whose partitions did not choose the same rollup of the rows (`rollup_dims = auto` picks it from each partition's data, so a partition attached later with few rows has none) lost the single `Custom Scan (SiftAgg)` over its partitions; the scan now keeps the rollups every partition has and reads the others by their rows.
- Changed: pg_sift: `sift.table_dict_lookup` is on by default: a `sift_table` stores the order of the strings of a dictionary of 1,024 strings or more, so the first test of a string in a new backend reads it in place instead of loading the dictionary whole, where the orders of a segment take a quarter of it at most; `off` writes the compact form (column format minor 14, which 3.3 and 3.4 read).
- Improved: pg_sift: the sort that gives a dictionary its order (`dict_lookup`, `count_columns`) skips the prefix that the strings of a run still share and radix sorts the bytes after it, so a column of urls or prefixed ids no longer pays a comparison of whole strings for every step; the order is the same.
- Improved: pg_sift: the bloom filter that `key_columns = 'auto'` writes for a unique id that no list fits may use up to a share of five percent of the unit together with the lists (three before), which gives it the bits a key of a column named in the option, so `WHERE id = x` skips more of the units whose filter lacks the key; the NOTICE at `CREATE INDEX` says so.
- Changed: pg_sift: `rollup_dims` defaults to `'auto'`: `CREATE INDEX` of a `sift_col` index picks the pair of low-cardinality columns whose combinations fit the cells of a unit and writes the rollup of the rows by them (column format minor 9, which 3.3 reads), with a NOTICE that names the columns; `rollup_dims = 'off'` writes none, and the regression tests that count pre-aggregated units or read the format minors of an index create it with `'off'`.
- Added: pg_sift: `sift.agg_verdict_cache` (on by default; `off` plans the choice between a shared CTE and its inline plan, and between the flattened form of a statement and the form as written, in full for every statement, so a plan does not depend on the statements planned before it)
- Fixed: pg_sift: the `cte_inline` row of `sift.explain_why` for a statement planned from a cached shared-CTE verdict showed the `scan_cost` and `normal_cost` of the statement that made the verdict; it has none now (the `verdict_cached` row says why), and a statement far from that size, planned in full after the hit, reports its own
- Improved: pg_sift: `count(*) WHERE col BETWEEN a AND b` (comparisons of one `int2`, `int4`, `int8`, `date`, `timestamp` or `timestamptz` column, nothing else in the statement) of a `sift_col` index with a global order for the column (`order_array_columns`, chosen by the build for scattered columns) counts a segment from the entries between the bounds of the range when every unit of the segment is exact (every page all-visible, no dead row), where SiftAgg filtered every row of every unit: 800,000 rows, 7,900 in the range 8.7M to 1.3M instructions (0.9 to 0.2 ms), 160,000 in the range 11.5M to 1.4M (1.2 to 0.37 ms). A segment with a dead row, a page not all-visible or no array is read as before, and so is a count with another test, aggregate or group.
- Improved: pg_sift: an index-only scan of a `sift_col` index forms its index tuples from the columns the plan reads, the others NULL, where it decoded, converted and formed every returnable column of the index for every row (`SELECT id` over a 7-column index with a `numeric` column converted the `numeric` of each row): 160,000 rows of 800,000 are 436M instructions against 272M (forced index-only scan), and the index-only scan costs 0.15 an emitted row where it was 0.5 (the same tuple scan stays 0.5), so the planner picks it for `WHERE rnd BETWEEN a AND b` over 20% of the rows: 570M instructions for the sequential scan it chose before against 272M. The node's function is replaced after `ExecutorStart` by one that tells the scan, on its first call, which columns the node's target list, quals and recheck quals reference; a scan that starts without it reads every column as before.
- Improved: pg_sift: the merge and the rewrite of a `sift_col` index copy the rows that have arrays leaf by leaf, as they always copied the rows without, and no longer build a value for every leaf and element of the row first: the merge of 19 segments of 400,000 rows with a path into an array of objects took 4,630,712,323 user instructions and now takes 4,030,012,412 (-13.0%; three interleaved runs each, 1.50e9 to 1.42e9 cycles), the merged segment the same 2,473,984 bytes in 18 units, and `sift.col_verify(index, true)` finds nothing.
- Added: pg_sift: `scripts/plangate.sh --calibrate` (docs/AGG.md, "Plan-estimate calibration"): for 30 shapes it plans the scan, a sequential, b-tree and bitmap scan and PostgreSQL's own plan with EXPLAIN only, runs each distinct plan under `perf stat` and fails when the plan the planner chooses takes more than 20% and 1,000,000 user instructions more than the best (30 seconds a lane, in the full tier; today none does and no pair with a scan is ranked the wrong way round)
- Improved: pg_sift: an index scan on a `sift_col` index with `point_tid_columns` over 65 keys or more (a text prefix of thousands of keys, an IN list) looks the keys up a chunk at a time (32, then four times as many) as the executor asks for rows, so that a `LIMIT 10` over a prefix of 4,012 keys reads 32 of them: 1,000,000 rows, 6.74 M to 1.32 M instructions a statement (0.67 to 0.22 ms warm), `LIMIT 1000` 8.82 M to 5.07 M, the whole answer +2.5% (14.78 M to 15.16 M). No format change; bitmap scans, index-only scans and aggregates are unchanged.
- Improved: pg_sift: a range of a `sift_col` column whose values are scattered over the heap (`WHERE rnd BETWEEN a AND b`, up to 16,384 rows) hands the heap the addresses between its bounds from the sorted keys of each segment (the global order, `order_array_columns`), with no unit read, where the units' zones all cover the range and every unit is read: 790 of 800,000 rows 18.4M to 2.4M instructions (covering) and 18.7M to 2.6M (rows), `count(*)` 8.3M to 2.6M, 7,900 rows 44.1M to 19.1M; with an equality of a `point_tid_columns` column (`AND grp = 7`) the shorter of the two lists answers, 8.3M to 2.8M. `order_array_columns` is now chosen by the build (`auto`, the default): columns of 1,000 or more distinct values whose correlation with the heap order is under 0.99, on a table of 100,000 rows or more with statistics, at most two and while they add no more than 25% to the index (+21% of a 7-column index of one scattered `int4` column, +22% build instructions; the NOTICE names the columns and the MB); `SET (order_array_columns = 'off')` and a `REINDEX` drops it, and an index with the option set by hand is left as it is. The segments are column format minor 21, which 3.3 and older refuse.
- Improved: pg_sift: an `ORDER BY key LIMIT n` that reads the order array (`order_array_columns`) decodes the first entries of each chunk it needs, not the 1,024, and leaves the array to the heads of the sorted runs when the LIMIT is 32 rows or fewer, the key follows the heap and the scan has no condition (the planner decides by the heads of the index, no setting). 1,000,000 rows in 4 segments, instructions of the statement: `LIMIT 10` 434 K to 307 K (a b-tree 167 K), `DESC LIMIT 10` 426 K to 297 K (b-tree 152 K), `LIMIT 1000` 2.13 M to 2.13 M (the heap fetch and the merge of the sources are what is left; b-tree 1.76 M, the sorted runs alone 4.48 M). No format change.
- Improved: pg_sift: the merge of the order arrays of a top-N keeps the source of the last entry while its next one comes before the best of the others, and no longer clears the 16 KB of each source's decoded chunk: 1,000,000 rows in 4 segments, `ORDER BY ts LIMIT 1000` 2.14 M to 1.99 M instructions (-6.9%), `DESC` 2.34 M to 2.15 M (-8%); the b-tree on the same table 1.76 M. No format change.
- Improved: pg_sift: an index scan on a `sift_col` index with `point_tid_columns` over a text prefix or a one-sided range (`<`, `<=`, `>`, `>=`) of a leaf in lookup form takes the keys from the sorted order of the dictionary, the positions between two bisections, and reads their codes a chunk at a time, where it built a table of the codes of each segment and a list of the keys first: 1,000,000 rows, a prefix of 4,012 keys, instructions of the statement: `LIMIT 10` 1.36 M to 0.57 M (0.22 to 0.09 ms warm), `LIMIT 1000` 5.17 M to 4.67 M, the whole answer 15.21 M to 15.13 M (-0.6%). No format change; the lookup limits are those of a list of keys.
- Fixed: pg_sift: where PostgreSQL's count of a lookup is a default guess, the scan path reported to the plan above up to the groups of all the rows the lookup lists can pass; that lift is bounded to the groups the plan expects plus a thousand. Two such scans joined on expression keys were reported at 56,000 and 140,000 rows, the join above at 39 million and its price at 544 million, so the plan that scans one aggregate and keeps the other on the normal path (the default) won against the one that scans both (`sift.agg_force`, 44 ms against 57 to 73 ms for the default); both now plan the same two scans
- Improved: pg_sift: the merge and the rewrite of a `key_cut` `sift_col` index sort the rows of each unit they read by the first key leaf copied next to the row reference (the sort the build uses), and not through the generic comparison of every leaf: the merge of 19 segments of 400,000 rows took 4,430,094,480 user instructions and now takes 3,993,725,232 (-9.9%; 2.19e9 to 1.9e9 cycles, three interleaved runs each), the merged segment the same 2,392,064 bytes in 13 units on 200,000 rows.
- Changed: pg_sift: VACUUM's space step of a `sift_col` index moves the highest runs of a segment larger than `sift.vacuum_rewrite_limit` down into the free blocks below it, a round at a time (up to 64 rounds, the lock let go between them), and cuts the file where it ended: the segment keeps its generation, its header page is written again at the lowest free block with the extents it now has, and nothing is read through the decoders. A merge writes its one segment above the blocks of the segments it replaced, and the file stayed at twice the live bytes for good when that segment was larger than the allowance: a 75.3 MB segment above 76.9 MB of free blocks, allowance 12 MB, went from a 152.2 MB file to 75.35 MB in 6 rounds of 8 to 18 ms each (0.10 s, 76.9 MB of WAL for 75.3 MB moved).
- Improved: pg_sift: the first plan of a backend on a `sift_col` index read the zone arrays of every leaf (about 11,000 buffers and 40 ms more than the second plan at 61,670 units, again in every backend a pooler recycles): a segment of 64 units or more now keeps the sums of its leaves and a sample of its zones in a block after its last section, and the first plan reads about 3,000 buffers at 61,670 units (11,700 before) (+0.4% bytes, no measurable cost to `CREATE INDEX`). No format minor: 3.4 reads such a segment and ignores the block; a segment written before is read from its zones. `sift.col_verify` checks the block.
- Added: pg_sift: `count_columns` (reloption, on for a text leaf read in place, `off` to leave out, column format minor 24): the dictionary of a text column of 1,024 strings or more also stores the rows of each string in byte order, and `count(*) WHERE txt LIKE 'abc%'` (or `=`, `<`, `<=`, `>`, `>=` in the C collation, nothing else in the statement) of a segment with no dead row and every page all-visible is read from two places of the dictionary, not from the rows of the matching strings: 1,000,000 unique strings, 4,012 matching, 5.3 ms to 0.14 ms (a b-tree 0.43 ms), 62,564 matching 3.7 to 0.14 ms, `< 'kf'` 5.3 to 0.18 ms; the counts are packed by blocks of 64 strings, 131 KB (+0.17% of the segments of the 8-column test table) for 1,000,000 unique strings, CREATE INDEX +0.44% instructions; a 3.4 library refuses the segments (`sift.upgrade_check('3.4')` lists them)
- Improved: pg_sift: `key_columns = 'auto'` writes a bloom filter for a column it chose that got no list, bitmap or runs, in what the share of the unit has left, so `WHERE id = x` of a unique id scattered over the units skips the units whose filter lacks the key; the NOTICE at `CREATE INDEX` names the column and what `point_tid_columns` would add in MB. An `int8` column keeps key lists and runs like `int2`, `int4` and `date`.
- Added: pg_sift: `sift.col_leaf_cache_kb` (default 4096 kB, 0 turns it off): the statistics a `sift_col` index gives the planner (a record for each leaf and a sample of the unit zones, folded from the zones of all its units) are kept in shared memory by the first backend that plans over the index, and the other backends copy them: the first statement of a new connection to an index of 10,000 units and six leaves planned in 9.4 ms and 2,494 buffers, now 0.9 ms and 213 once another backend has planned (the index of a server with 63,000 units and some forty leaves cost a new connection 100 to 800 ms). A merge, a rewrite, a `REINDEX` or a change of the options folds them again, in one backend; `sift.stats()` counts `leaf_stats_built` and `leaf_stats_shared`
- Added: pg_sift: `sift.json_build_object(VARIADIC "any")` returns the text of `json_build_object` (the same bytes for every type of argument and key, and the same errors) and looks up the category and the output function of each argument once for a call site, where the core function asks the type cache for every argument of every call; with `sift` before `pg_catalog` in the `search_path` every call of the name takes it. A grouped statement over 300,000 rows that calls it 5 times takes 166.4M instructions where `json_build_object` takes 177.1M, and one that calls it 12 times 925.5M where 975.3M.
- Fixed: pg_sift: a promotion of a write buffer of a `sift_col` index whose `key_columns` is `auto` wrote its segment without the key lists of a column the cube holds (the report after a build drops it from the option, the units keep their lists), so an equality on the key read every unit whose zone held it; it keeps the lists the segments hold, as a merge does.
- Improved: pg_sift: `key_columns = 'auto'` takes a unique `int2`, `int4` or `date` column named `*_id` when the index is key cut or the column is not in heap order, and a unit whose keys come in runs of consecutive values keeps them as a list of runs (column format minor 23) where the offsets and the bitmap do not fit 3% of the unit, so a lookup by id, an `IN` list or a range of ids reads the few units that hold them where it read every unit. A 3.4 library refuses a segment with a list of runs: `sift.upgrade_check('3.4')` says how to write it without.
- Improved: pg_sift: an index scan that returns tuples (not an index-only scan or a bitmap scan) of a lookup in a `point_tid_columns` directory (`WHERE grp = 7 LIMIT 10`, an `IN` list of a few long lists) decodes the addresses of its keys as the scan asks for them, a cursor over each list of the keys merged in order, where it decoded every address of the keys before the first row (8,300 addresses of the key at 800,000 rows, 1.5 million instructions): `grp = 7 LIMIT 10` runs in 0.021 ms (0.133 before; 0.012 to 0.015 for a btree), `LIMIT 100` in 0.036 (0.21), `LIMIT 1000` in 0.22 (0.36), and the scans that read every row of the key take what they took (4.9 ms for the 8,300 rows). The units with a dead row are read after the addresses of the others; a scan with other conditions, or an `IN` list of more than 8 lists, decodes whole as before.
- Changed: pg_sift: the planner prices the index scan of a lookup in a `point_tid_columns` directory by what it does: the start is the open of the source and a probe (12 plus 4 a key, where it was 24 plus 4: 67,000 instructions, 16 us measured), a key that is in every unit counts its listed rows and not the rows of the units (the filter of 800,000 rows was charged for `grp = 7`), and the index correlation is the share of units whose directory lists addresses, since the scan returns them in heap order. `grp = 7 LIMIT 10` of 800,000 rows (`grp` has 100 values) took the sequential scan (0.21 to 0.39 ms, cost 68.4) and takes the index scan now (0.016 to 0.024 ms, cost 22.2; a btree on `(grp, ts)` is 0.42..37.5). Planner picks of the 41 queries of `bench/col_scan.py` (800,000 rows, one index) that changed, warm median: `IN` of 10 ids count 0.063 to 0.030 ms, `IN` of 1,000 ids rows 0.92 to 0.65 (bitmap to index), rows of an equality plus a range 0.42 to 0.28, an equality on a key in every unit 3.16 to 1.22, text `LIKE` rows 0.27 to 0.06, count 0.65 to 0.05, prefix count 1.16 to 0.15; none is slower by more than 0.015 ms, and the plan-choice gate (30,000 rows, stock lane) has 99 shapes, 0 loses, 0 failures.
- Improved: pg_sift: a lookup of a `point_tid_columns` directory with other conditions (`WHERE grp = 7 AND ts BETWEEN ..`) decodes the addresses of the key only in the units the other conditions leave, and stops at the last of them, where it decoded every address of the key first (about 8,700 at 800,000 rows, 0.9 million instructions): `count(*)` with the aggregate scan runs in 0.21 ms (0.34 before; 0.12 for a b-tree), the index scan and the bitmap scan of the same statement take 25% fewer instructions.
- Fixed: pg_sift: the rewrite of a `key_cut` `sift_col` index wrote all its live bytes as one segment whatever `sift.col_auto_merge_partition_kb` and `sift.vacuum_rewrite_limit` said, and re-merged every partition into it at the next VACUUM: an index over the limit was one segment larger than the space step could ever move, and the file stayed at twice the live bytes (105 MB live after 12 churn cycles: 209.5 MB file after VACUUM). The rewrite now cuts the index into partitions of the key (under three quarters of the size, so that a step of the space step moves one whole), and a partition the churn did not touch is not read. The merge of an index over the limit (a parallel build's tier, `sift.col_merge`) writes partitions too: 19 segments of a 75 MB index became 9 of at most 9 MB, and the file went from 150 MB at the merge's peak back to 75.5 MB at the next VACUUM, where the one segment of 75 MB it wrote before stayed at 152 MB.
- Changed: pg_sift: VACUUM's space step of a `sift_col` index runs rounds, up to 16, each copying at most `sift.vacuum_rewrite_limit` under the lock and letting it go before the next, and a round moves the segments that end the file down into the free blocks even when it cannot cut the file yet (the next round or VACUUM does): a rewrite writes beside the segments it replaces, and the file went back to its live bytes only after as many VACUUMs as the limit fits in the index (105 MB live, 12 MB limit: file after VACUUM 199.9 MB without rounds, 115.1 MB with them, 12 cycles).
- Fixed: pg_sift: the scan of a statement whose lookups are a CTE or a sub-select of ids that PostgreSQL has no statistics for started no workers when it read many rows: PostgreSQL expects a handful of rows from the join, the scan was priced at a few milliseconds for them, and the 3 to 4 ms the workers take to start never paid (two aggregates over the elements of a list, 353,000 rows: 793 ms alone, 262 ms with 4 workers). The path with workers is now priced for the rows the scan reads, and the price of the scan without workers (the one the plan above and the normal plan are compared with) is unchanged.
- Fixed: pg_sift: a scan whose workers the planner prices for the rows it reads (lookups of a CTE or sub-select of ids) kept one process when its units numbered fewer than `sift.agg_par_min_rows` (128), which a scan of 49 units of unnested documents (101,000 rows, 990 ms alone, 870 with 4 workers) did: the setting is now `sift.agg_par_min_rows` (default 80,000), the rows of the units the scan keeps; at 65,000 rows (60 units of 900 bytes) the workers made a 40 ms scan 58 ms, and the setting is 0 to start them for any scan.
- Fixed: pg_sift: a query that joins a table to a `sift_col` index with `point_tid_columns` could crash the backend (signal 11) in the executor: the bitmap path that reads the addresses of a lookup keyed by another relation (`WHERE EXISTS (SELECT 1 FROM t WHERE t.id = e.key)` over partitions, in a merge join) was planned without that relation, so the key of the outer side was read from a plan that did not wait for it. The path needs the relation now, as the index path under it does. The releases 3.4.0, 3.4.1 and 3.4.2 have the fault.
- Fixed: pg_sift: the builds of a release line could not be told apart in a running server: `sift.version()`, the extension version and, on PostgreSQL 18, `pg_get_loaded_modules()` said `3.4` for 3.4.0, 3.4.1 and 3.4.2. The module version is now the release and the commit (`3.5.0+2a7e47598`; `scripts/build-id.sh`: `git describe` of a checkout, the `BUILD_ID` that `git archive` fills in for the deb and rpm builds, `unknown` for a tarball with neither). `sift.version()` and the SQL scripts are unchanged.
- Changed: pg_sift: the planner prices a lookup of a point directory by the rows it lists: the aggregate scan charges the tests of an `IN` list or an equality only for those rows, where it charged every row of the kept units, and the index scan no longer charges the filter of the kept units' rows when the directory is what keeps them. A `count` or `sum` over 1,000 ids of 800,000 rows took SiftAgg with two workers (5.6 to 7 ms) over the index (0.9 ms); it runs in 1.3 to 2 ms on one process now (cost 2,048 against 8,167 for the index), and a list of 100 stays on the index scan (0.1 ms).
- Improved: pg_sift: an `IN` list of up to 65,536 keys is looked up in the point directory of a `sift_col` index (`point_columns`, `point_tid_columns`) by the aggregate scan whatever the number of units, where a list of more keys than twice the units made it read every unit; and the scan's start for an `IN` list no longer walks the table of its range twice (about 0.4 ms of 800,000 rows for any list). `count`/`sum` over 100 keys of 800,000 rows run in 0.6 ms (1.1 to 1.8 before), over 1,000 keys in 1.3 ms (3.3 before), with the scan forced on one process.
- Fixed: pg_sift: a scan with two lookup tables keyed by the same column (a chain `B.id = A.id` that the planner rewrites to the scanned column) tested the column against the keys of each: the keys of one within the keys of the other are now the only join key conjunct (`Join Key Filters` counts it once), so the wider set is not tested over every scanned row, which cost the platforms without a vector kernel for the lookup table (arm64) a pass over the rows.
- Improved: pg_sift: the first lookups of a backend in a text or uuid column of a `sift_col` index with `point_columns` (`WHERE s = x`, a range or a prefix of a dictionary of 2,048 strings or more) no longer pay for the whole dictionary at the 33rd test: the backend keeps the head of the dictionary and the sorted positions its searches read (about 300 kB for a segment and a leaf, under `dict_cache_kb`, by segment generation: nothing to invalidate after a merge, `VACUUM` or `REINDEX`), and parses the dictionary only once the searches have cost what the parse does. 1,000,000 rows, a new backend, the first 100 different keys: uuid 69-92 ms to 14-23 ms (480 M to 51 M instructions), text 92 ms to 16-23 ms (790 M to 123 M); 300 keys: uuid 85 ms to 32-37 ms, text 88 ms to 52-56 ms; 1,000 keys: uuid 107-148 ms to 85-96 ms, text the same (117-163 ms to 140-164 ms); from about 2,000 lookups the old parse at the 33rd test is paid back, and 3,000 keys cost uuid 175-217 ms to 246-304 ms (1,040 M to 950 M instructions, the bisections miss the cache more than the parse does) and text 202-214 ms to 224-241 ms; no format change
- Improved: pg_sift: `count(*) WHERE col IS NULL` (or `IS NOT NULL`, alone, one `count(*)` and nothing else) of a `sift_col` index reads the null counts of the zones of the units, also for a column that has no histogram in them (more than 256 values in a unit): 1,000,000 rows, 30% NULL, 1.01 to 0.03 ms (9.6M to 0.26M instructions); a unit with a dead row, an exception or a NaN is read as before.
- Fixed: pg_sift: a merge or `sift.col_refresh` (and the rollup worker after `ALTER INDEX .. SET (cube_minmax = ..)` or `rollup_dims`) of a `sift_col` index with `key_columns = 'auto'` rewrote the units without the key lists, and the key table, that the build had kept for a column the cube holds (the segment came out as column format minor 14), so after churn an equality on that column read every unit whose zone held it and the planner priced the scan above a btree plan (a SiftAgg node lost on statements with a filter on the key and a lookup join); every rewrite now writes the lists again where a segment holds them.
- Improved: pg_sift: `WHERE key = x` (an `IN` list, a range of the key, `GROUP BY key`) with `count`, `sum`, `min` and `max` of a numeric, date or timestamp column of a `sift_col` index reads one record of each segment, not its units, when the key is a foreign key or a `*_id` column of `key_columns` with more than 256 values in a unit (the cube of a unit keeps no histogram of it): a key table per segment of at most 3% of the segment, written by the build, promotion, merge and VACUUM (+2.6% instructions of `CREATE INDEX`, +0.9% of an insert and promotion, +4.3% of a merge, +3.0% of a VACUUM that rewrites a segment, at 100,000 rows and 3,000 keys). 300 keys at 900,000 rows: `count(*), sum(amount), min(ts), max(ts) WHERE key = 17` 5.20 ms before, 0.107 ms now (0.49 ms and 0.054 ms at 100,000 rows); the segments of minor 22 are not read by 3.4, and `key_table = off` writes the minor 15 ones as before.
- Improved: pg_sift: a `sift_table` keeps the key table of its foreign key and `*_id` columns too, so `WHERE key = x` (an `IN` list, a range of the key, `GROUP BY key`) with `count`, `sum`, `min` and `max` of a numeric, date or timestamp column reads one record of each segment whose superblocks are all plain, not its units. 300 keys at 900,000 rows: `count(*), sum(amt), min(amt), max(amt) WHERE key = 17` takes 0.577 ms without the table and 0.068 ms with it (`k IN (3 keys)` 0.835 and 0.069 ms), the segments grow by 0.27% (2.7% at 100,000 rows), and the write paths cost +0.4% (load and promotion) to +1.5% (a rewrite) instructions. The segments of minor 22 are not read by 3.4; `sift.table_key_table = off` writes the minor 15 and 18 ones as before.
- Fixed: `sift.col_verify` could report that a unit's exceptions differ from its chunks after a writer reused an empty overflow bitmap. Writers now inspect the bits; verification checks the stored exception leaf and accepts missing optional summaries from older writers. Existing indexes need no rebuild for this fault.
- Fixed: pg_sift: `sift.upgrade_check('3.3')` did not list an index with a global order (`order_array_columns`, column format minor 21) as one that 3.3 cannot read: the table of the first release that reads each minor stopped at 20. It is a blocker now, and the check to 3.4 finds it ok.
- Changed: pg_sift: the extension SQL is version 3.5 (`pg_sift--3.5.sql`, `pg_sift--3.4--3.5.sql`, `default_version = '3.5'`): run `ALTER EXTENSION pg_sift UPDATE` in every database. The update moves the version, adds `sift.json_build_object(VARIADIC "any")` and replaces `sift.upgrade_commands()`; no other SQL object changes; the scripts of 3.4 are the released ones.
- Fixed: pg_sift: the normal plan of a lookup of one key that owns many rows of the table (a key of the column's most common values, listed as a constant of a CTE) was priced for the mean's rows of the column, which PostgreSQL counts a probe of the key by, so the scan lost the choice for a statement of the heaviest key that it wins by a factor of ten: the normal plan is charged the excess of the key's rows over the mean's, as many times as the statistics give (the key of 1,726 rows of 95,916 in 500 keys: 9 times), on top of the lost-estimate charge of the rows out of the joins. Default settings, 2 workers allowed, a column index, the 148 statements of the heaviest key, medians of three alternated builds against the build without it: 100,000 rows 62.8 ms to 7.5 for a statement of two arms over the key (PostgreSQL's plan 72.9), 12 of the 148 statements 20 to 90% faster (the sum 1,615 ms to 1,490); 300,000 rows 94.7 to 63.7, 90.1 to 60.0, 40.7 to 27.5 (the sum 2,773 ms to 2,710), 5 statements 1.5 to 11 ms slower (51.1 to 62.3).
- Fixed: pg_sift: the keys a statement lists as constants of a CTE were not followed through a second CTE: when a CTE of constant ids was joined to the table in a CTE (`WHERE s.id = k.id`, with or without a `LEFT JOIN`) and a third CTE passed the table's key on, the shared CTE and the arms of the inline plan were priced for the mean's rows of the column, as if the listed key were an average one, so the plan that keeps the shared CTE won where the scans win in time (a statement of the heaviest key through two CTEs). The listed constants are now followed to the CTE that reads the key, and the rows they own (as the statistics give them) lift the CTE's scan as for a CTE of the list itself. Default settings, 2 workers allowed, a column index, 904 statements of 148 shapes, medians of three alternated builds against the build without it: at 100,000 rows 46 plans change, 12 from the normal plan to the scan and none the other way, and the 43 timed statements take 264.4 ms in all against 374.4; at 300,000 rows 53 change and the 53 timed take 959.0 ms against 1,177.0; one statement is slower (2.3 to 6.2 ms).
- Changed: pg_sift: the planner prices the dictionary a worker inflates for a numeric stored as images (35 operators a row, measured with 2,000,000 rows of almost unique values) by the number of values the column has, to the 0.13 power over 2,000,000 (20 operators with 30,000 values over 300,000 rows, 20 ns a row measured): the workers of `sum(numeric)` over such a column were priced above the scan without them and the scan kept one process, though they shorten it by 28 to 40% (300,000 rows, the scan 15 to 17 ms against 22 to 28, with the workers' CPU 43 to 48 ms against 25 to 33). Default settings with `parallel_setup_cost = 1` to bring the break-even to this size, 34 shapes of int, text and numeric keys, `count(DISTINCT)` and wide sums: 7 of the 34 launch workers (3 of them since this change), all 7 are shapes the workers shorten, and none of the shapes they lengthen does.
- Changed: pg_sift: the planner prices what the workers of a scan cost with the measured numbers: a millisecond of the scan is 750 cost units when it weighs the 3.35 ms the workers take to start (it was 500), and the leader's merge costs 25 operators for each group key of each group of each process's table, on top of the 8 for the group (35 ns measured with one key over 30,000 groups, 85 ns with three over 100,000). A scan of 4 to 13 ms no longer launches workers that add 7 to 17 ms of CPU and no wall: 1,000,000 rows, default settings, 2 workers allowed, median of 3, 2 rounds: 9 groups of text and an average 6 ms to 4 (CPU 16 ms to 8), an average over two text keys of 30,000 groups 16 ms to 12 (CPU 36 to 16, instructions 178 to 135 million); the 8 other shapes of the 10 timed keep their plan and their times, including the 8 ms scan over three text keys that the workers shorten from 11 ms.
- Fixed: pg_sift: `VACUUM` of a `sift_table` with a unit of more than 65,000 rows failed with `could not read blocks N..N in file ...: read only 0 of 8192 bytes` once rows were deleted from every such unit, and never finished: the liveness bits of a unit of up to 74,496 rows lie on three pages of the segment, which with the header page are four, and clearing them kept room for three (a page number overwrote the buffer of the first page). 3.3.0, 3.4.0, 3.4.1 and 3.4.2 have it (1,000,000 rows of one integer column, `VACUUM FULL`, `DELETE ... WHERE id % 20 = 5`, `VACUUM`); the rows stay readable and the dead ones unreclaimed. Test `tbl_vacuum_dead_span`.
- Fixed: pg_sift: the normal plan of an aggregate that reads a lookup of a few keys was kept at PostgreSQL's price whatever the keys own, and PostgreSQL counts the rows of a key at the mean of the column: a lookup of one id whose rows are a fiftieth of the table (a key of the column's most common values) met a normal plan priced for a fiftieth of that, and the scan lost the choice it wins in time. The normal plan is now charged the excess of the rows the keys own (the share the statistics give the listed constants of the lookup, as the lift of a shared CTE counts them) over the rows PostgreSQL expects out of the joins, as for a lookup of many keys. Default settings, 2 workers allowed, a column index, 904 statements of 148 shapes: at 100,000 rows 14 plans change and take 65.3 ms in all against 20.5 (35.7 ms to 4.7 where PostgreSQL's plan takes 55.4, 14.4 to 3.8 (15.4), 4.2 to 1.4, 3.2 to 1.3); at 300,000 rows 21 change, 320.1 ms against 245.6 (26.5 to 9.5, 29.5 to 16.2, 31.1 to 20.6, 63.5 to 52.1); none is slower by 20% and 1 ms, and the statements within 1.2x (or 5 ms) of the faster of the two plans go from 12 to 14 of the 14 at 100,000 rows and from 14 to 19 of the 21 at 300,000.
- Fixed: pg_sift: the heap check the planner charges the scan for the rows on pages that are not all-visible counted the rows the table's own filters keep, not the rows that reach the groups after the lookup joins have filtered them: a statement whose lookups keep a few hundred of 288,000 rows was charged the pages of all of them. A scan whose arms are priced for the check that the data needs takes the arms it lost: 300,000 rows, 12-key lookups of a table with a column index, 16% of whose pages are not all-visible, default settings, 2 workers allowed: 39.7 ms to 28.6, 38.5 to 28.9, 41.8 to 31.1 (the normal plan takes 4 to 5 ms: the pick of these three statements is still the scan); of the 57 statements timed at 100,000 and 300,000 rows 3 change their plan and none is slower. Test `agg_visibility_refusal`.
- Improved: pg_sift: the scan gives the ids of a text value (a `count(DISTINCT)` of a text column, a group key, the values the leader takes from the workers) by its bytes when the collation is deterministic, hashing and comparing them in place instead of through `hashtext` and `texteq` (each looks up the collation and detoasts both values): 9 to 10% fewer instructions for the scan of `count(DISTINCT author)` over 1,000,000 rows of 245,000 values, 394 to 357 million without workers and 1,009 to 918 with 2 (300,000 rows: 121 to 109 and 303 to 272); the same ids, the same answers; a nondeterministic collation keeps the type's equality.
- Changed: pg_sift: the planner prices the ids that the dictionary entries of a text column get for a `count(DISTINCT)` of it (35 operators an entry at 100,000 distinct values, growing with their number: 6 ns a row measured at 10,000 values, 83 at 100,000, 199 at 736,000 over 3,000,000 rows; it was free), and the pairs that the workers send to the leader for it (a distinct text value costs the leader 0.4 to 1.2 us to merge: 250 operators; it was 7 for every row). The scan with 2 workers took 1.9 times as long as the scan without (226 against 119 ms over 1,000,000 rows, 1,051 against 622 ms over 3,000,000) and PostgreSQL's index-only scan 66 and 307 ms, but was priced at 65% of the serial scan. Default settings, 2 workers allowed, median of 5 runs: `count(DISTINCT author)` of 245,000 distinct values over 1,000,000 rows 166 ms to 95, 736,000 values over 3,000,000 rows 680 ms to 327, the plan PostgreSQL's with an index on the column; a column of fewer than 30,000 values keeps the scan.
- Improved: pg_sift: an `ORDER BY k DESC LIMIT n` over a key that grows with the table (a timestamp in insert order) reads the superblocks of each window of the walk from the window's best end, so a window that holds the rows of the page reads one unit and not every unit of the window in table order: `WHERE ts <= x ORDER BY ts DESC LIMIT 10` with x in the middle of a table of 3,000,000 rows (1,000,000, 100,000) ran 2.67 million instructions (1.92, 0.74) and runs 0.65 million (0.74, 0.74), 8 units read (5) become 1 (100,000 rows: 2 before and after). Test `col_topn_reverse_window`.
- Fixed: pg_sift: `sift.agg_force` priced every scan path at 0, so the path with workers and the one without tied and the planner kept the one without: a forced scan never got workers, and what was timed under it (`count(DISTINCT uuid)` over 1,000,000 rows, planned with 2 workers by default) was the serial scan. The paths now keep the order of their prices, a millionth of them, so the forced scan launches workers wherever the natural plan does.
- Changed: pg_sift: the planner prices the call of a transition function (`min`/`max` of text, `bool_and`, `variance`, `array_agg`, `string_agg`, `corr`, ...) at 1 operator (it was 4) plus the function's cost and the 2 of its argument: the 32 to 88 ns a row the scan measures is 3 to 7 operators in all, and a scan 10 to 50% faster than PostgreSQL's plan lost the choice. Default settings, 2 workers allowed, on top of the pricing of the wide columns: a numeric group key next to an int one 254 ms to 131 (PostgreSQL's plan 211) at 1,000,000 rows, 683 to 356 (474) at 3,000,000, 32 to 19 (37) at 100,000; `bit_and(int)` at 100,000 rows 11 ms to 3; of the 51 aggregates of a set of wide and narrow columns timed against both plans, the choice is the faster one for 50 (39 without it, 29 on the tip); the 4 cells of the matrix (254 shapes at 100,000, 1,000,000 and 3,000,000 rows) that change their plan take 979 ms in all against 510 ms, none is slower than PostgreSQL's plan.
- Fixed: pg_sift: an `ORDER BY k, expr LIMIT n` over a range of the sort key on an index with `order_columns` (`WHERE k >= x ORDER BY k, k * 2, id LIMIT 50`) failed with "sift_col sorted run is corrupted": the search of the run for the first entry of the range read the key through the buffer of the leaf, which holds the whole key of the unit when a sort expression follows it, and overwrote that; it reads the key from the unit's vector.
- Changed: pg_sift: the planner prices the scan of a wide column (a numeric without a precision, a uuid, an interval, a bytea) by what the scan does with its dictionary entries: an entry that is only copied (`array_agg`, `string_agg`, `count` of the column, a group key) costs 5 operators and one that a sum or an average of the column is added to in the same scan 25 (it was 60 to 180 for every use), and a group key of a wide column pays for its table of groups by the number of distinct values (45 operators an entry at 30,000 of them, growing with their count to the 0.42 power; it was free). The scan of `array_agg(uuid)` or `array_agg(numeric)` or of a numeric group key, 10 to 40% faster than PostgreSQL's plan and serial by design (the spelling of `1.0` and `1.00` depends on the order of the rows), was priced above it and lost the choice. Default settings, 2 workers allowed: 1,000,000 rows, `array_agg(vnb)` 171 ms to 109 (PostgreSQL's plan 201), `array_agg(vu)` 124 to 78 (141), a numeric key 148 to 107 and 197 to 144; 3,000,000 rows, 613 to 341, 527 to 229, 423 to 306 and 565 to 338; the 11 cells of the matrix (254 aggregate, key and filter shapes at 100,000, 1,000,000 and 3,000,000 rows) that change their plan take 2,824 ms in all against 1,676 ms, none is slower than PostgreSQL's plan.
- Improved: pg_sift: an `ORDER BY .. LIMIT n` that reads a sorted run (`order_columns`) over conditions on integer, date and time columns that bound the sort key (`ts >= x`, `ts BETWEEN a AND b`, `ts = x`, an OR of ranges, the page after a key `(ts, id) > (x, y)`) reads a unit from the first entry of the range without running the unit's filters, and tests the conditions on the rows it takes (a condition on another column only where an equality of the key pins it; a unit in which the walk meets many rows that fail goes back to the filters); the search of the range probes where evenly spread keys would put the bound, after the zone of the unit has answered the units that the range holds or misses whole. Instructions a statement over 3,000,000 rows of a scattered timestamp, runs only (1,000,000 rows, 100,000 rows in brackets): `WHERE ts >= x ORDER BY ts LIMIT 10` 18.7 to 8.4 million (6.6 to 3.1, 1.0 to 0.65), the page after a key `LIMIT 50` 63.6 to 10.4 million (24.2 to 4.7, 3.6 to 1.9); EXPLAIN ANALYZE shows `Units Read Without Filters`.
- Fixed: pg_sift: the cap of a CTE of a literal list's keys to the constants of its list (3.5) applied to every column the CTE returns, though the list says nothing of the values of a second column (the partner of a listed row, joined to the parents of the rows on that column); it applies to the column the list is of. The scan of an index cut by the key is priced for the keys PostgreSQL expects there again: on a fixture of 400,000 rows the scan costs 876 for the listed column and 1,547 for the partner column, where both cost 876. No plan of the 600 statements measured for the cap changes.
- Fixed: pg_sift: a request for the keys of a literal list (a CTE of ids kept to the keys a sub-select allows) priced the units its keys leave for the rows the keys own in the column's statistics counted as keys, so one key of many rows was priced as thirty and the scan of an index cut by the key (`key_cut`) as a read of a fifth of its units where it opens 3 of 149: the planner kept the normal plan, or chose the scan and started four workers for three units. The keys of such a CTE are no more than the constants of its list. Measured on 1,000,000 rows with the index cut by the key, 600 statements over seven shapes, one to a thousand keys, warm, 4 workers: the pick is within 1.2x of the faster of the normal and the scan plan (or 5 ms of it) for 99.0% of the statements, up from 86.8%; those that lose more than 2x and 50 ms fall from 3 to 1, and the milliseconds lost against the faster plan from 2,304 to 335. Of the 106 statements whose plan changes, 38 are faster by 20% or more and none slower (1,439 -> 517 ms in total; the 36 that go from the normal plan to the scan: 901 -> 184 ms, 6 to 90 ms -> 1.4 to 14 ms). On a plain index 54 plans change (the scan runs without workers: instructions equal, 416 -> 333 ms) and one statement of 1 ms takes the scan (0.7 -> 3.5 ms).
- Changed: pg_sift: an `ORDER BY .. LIMIT n` that reads a sorted run (`order_columns`) over a range of the sort key (`ts >= x`, `ts BETWEEN a AND b`, `ts = x`, a row comparison of the page after a key, an OR of ranges) starts at the first entry of the range, found by two binary searches of the run, and no longer walks the entries before it, which fail the filter: `WHERE ts >= x ORDER BY ts LIMIT 10` over 1,000,000 rows of a scattered timestamp takes 6.1 million instructions where the plain index takes 17.8 and the whole run walk 56 (100,000 rows: 1.0 million, 2.2 and 6.0); a key of floating point or text, and a filter on another column, are read as before.
- Fixed: pg_sift: where PostgreSQL's count of a lookup is a default guess (a CTE or a sub-select of ids), the scan path reported to the plan above the groups of all the distinct values of the key, so a request for a few keys met a plan priced for thousands of groups (a hash or merge join over them that the forced scan did not need) and the planner kept the normal plan; it now reports them at the geometric mean of PostgreSQL's count and all of them, as the normal plan's price already takes the keys. Measured on 1,000,000 rows with the index cut by the key (`key_cut`), 600 statements over seven shapes, one to a thousand keys, warm, 4 workers: the pick is within 1.2x of the faster of the normal and the scan plan (or 5 ms of it) for 86.8% of the statements, up from 74.7%; those that lose more than 2x and 50 ms fall from 9 to 3, and the milliseconds lost against the faster plan from 4,853 to 2,304. Of the 73 statements whose plan changes, 52 are faster by 20% or more and none slower (5,551 -> 2,768 ms in total); on an index not cut by the key no pick changes (599 statements at 1,000,000 rows, 28 at 3,000,000), and at 3,000,000 rows on `key_cut` 6 of 28 plans change, all faster (15.0 -> 5.3 ms, 18.2 -> 6.9, 21.0 -> 13.2, 27.7 -> 19.4, 41.9 -> 32.7, 90.1 -> 77.5).
- Improved: pg_sift: a statement with a shared CTE whose verdict is the inline plan no longer plans the plan with the shared CTEs as well to read the size of the request from it: the verdict keeps the size of the inline plan too (the most rows its scans read) under a band of its own, and the hit plans the inline plan alone. On the 156 statements of the corpus over 1,000,000 rows, 84 fresh-backend hit plannings of 28 shapes and classes go from 78 to 72 million instructions on average (the 9 plans of the inline form drop from 94-171 to 81-137 million, 3.4 ms median each at about 3 million instructions a millisecond); the plan chosen is the one planned in full in all 84.
- Fixed: pg_sift: `CREATE INDEX` of a `sift_col` index with `hll_columns` and a leaf that is NULL in nearly every row (a soft-delete stamp, which keeps exception cubes) but no list of elements failed with "segment 1 is corrupted": the offset of the sketches counted a directory of exception element blobs that the segment does not hold; the index is built and the sketches are read (a table of 60,000 rows: `col_verify` passes and `approx_count_distinct` answers from the two units' sketches).
- Changed: pg_sift: the planner prices the state and final function of each group of an aggregate that runs on PostgreSQL's own transition function at 20 operators (it was 150, the price of the sort of an ordered-set group, which keeps it): the 40 to 200 ns a group the scan measures for `min` of text, `bool_and`, `max` of a timestamp, `variance` and `array_agg` over 100,000 to 620,000 groups. A group-by on a column of hundreds of thousands of values with such aggregates kept PostgreSQL's plan where the scan is 2 to 3 times as fast: 1,000,000 rows, `count(*)`, `min(text)`, `max(timestamptz)`, `sum(numeric)` by an integer column of 621,000 values 753 ms to 397 ms, by a rounded float 682 to 221 ms, and by a column of 100,000 values (100,000-row tables) 25 to 7 ms. The 6 cells of the matrix that flip are all faster than PostgreSQL's plan.
- Fixed: pg_sift: with `sift.explain_refusals` on (or in `sift.explain_why`), a `min()` or `max()` that PostgreSQL plans as a `LIMIT 1` sub-select and whose scan arm lost left its arm in a list that outlived the statement, and the next plan of the session read the freed list: the backend crashed (segmentation fault) on the next `EXPLAIN` of the session (`EXPLAIN ANALYZE SELECT min(v4) FROM t` with `sift.enable_agg` on, then off). The list is dropped with the context it was made in. Test `agg_refusals`.
- Changed: pg_sift: the planner prices a row of an aggregate that runs on PostgreSQL's own transition function (`min`/`max` of text, `bool_and`, `variance`, `stddev`, `array_agg`, `string_agg`, `corr`, ...) at 4 operators plus the function's cost (it was 12), the 32 to 88 ns a row the scan measures; it priced the scan at four to five times its cost, and kept PostgreSQL's plan where the scan is 2 to 5 times as fast. Default settings, 1,000,000 rows, 4 groups: `min(vt)` 156 ms to 35 ms, `bool_and` 193 to 22, `variance(vf8)` 174 to 40; the shapes that flip, 41 of the 504 cells of the matrix that ran (254 aggregate, key and filter shapes at 1,000,000 and 100,000 rows, 2 workers allowed), take 1,032 ms in all against 210 ms; none is slower than PostgreSQL's plan.
- Improved: pg_sift: the scan of a numeric without a precision (or an interval, a uuid, a bytea: any wide column) fetches the dictionary entry of a row some rows before it reads it: the dictionary of a segment of millions of almost unique values does not fit the cache, and every row was two misses (the offset, then the image). `GROUP BY` 6 groups, `count`, `sum`, `avg` of 10,000,000 almost unique numbers in one segment, a new backend, no workers: 2,174 to 1,242 ms (8.99 to 5.10 billion cycles; instructions +1.8%); with 4 workers allowed and the scan forced 842 to 501 ms (PostgreSQL's plan 317 to 366 ms), 2,000,000 rows unchanged within the noise (a dictionary of that size is cached).
- Improved: pg_sift: a statement with a lookup table of a first-row child (`LEFT JOIN LATERAL (SELECT .. LIMIT 1)`) that no cache holds no longer reads every page of the table to ask a cache for it: the request for a cached table first looks for an entry of the same query in the backend and the shared memory.
- Added: pg_sift: a `sift_table` reads the rollups of the elements of its declared jsonb lists (`sift.table_set_paths`) that its units already carried and nothing read: a `GROUP BY` of the unnested elements by their leaves of few values with `count`, `sum`, `min` and `max` adds the tuples of each unit up instead of reading its elements, so a `GROUP BY` of an element key over 500,000 rows takes 0.121 ms where it took 22.977. The groups are not written differently, so they grow 0% and the writers cost 0%; `SET sift.agg_rollup_cost = 1000` reads the elements as before. Test `tbl_rollup`.
- Added: pg_sift: the cube of a `sift_table` keeps the minimum and maximum of each integer measure per group, as the cube of an index does by default (`cube_minmax`, column format minor 7): `GROUP BY g` with `min` and `max` of an integer column over 500,000 rows takes 0.140 ms where it took 2.860. The groups grow 0.7% (8 columns) to 3.1% (4 integer columns) and the writers spend 0.0% to 0.2% more instructions; the calendar cells of a date or timestamp column hold none, and a table written before keeps its cubes until a rewrite. `SET sift.col_format_minor = 6` writes none; a 3.2 library cannot read minor 7. Test `tbl_preagg_minmax`.
- Added: pg_sift: a `sift_table` keeps the key list or bitmap of each unit for its int2, int4 and date foreign key and `*_id` columns that are not unique (the default of `key_columns` for an index), and the sorted run and run head of each unit for its `timestamp` and `timestamptz` columns the rows follow (a run of any other takes more than 3% of a unit and is not written): `count(DISTINCT fk)` over 500,000 rows takes 0.053 ms where it took 2.10 (36.2M to 0.69M instructions), and `ORDER BY ts DESC LIMIT 10` 0.038 ms where it took 0.140. The groups grow 0.0% to 1.4% (500,000 rows, 7 columns) and the writers (INSERT, COPY, VACUUM and its merge) spend 0.0% to 1.6% more instructions; groups written before keep their format and are read as before, new and rewritten groups are column format minor 15 (lists) or 18 (run heads, which a 3.3 library cannot read). The top-N does not walk a run that is the rows in their own order (the walk read LIMIT 100 to 1,000 up to 37% more instructions than the unit). No setting: `SET sift.col_format_minor = 12` writes none.
- Fixed: pg_sift: with `sift.enable_agg = off` the planner lifts no more the rows of a CTE scan or of a sub-select scan (a floor made for the choice between the scan and the shared CTE): the normal plan is the plan of the 3.3 library, text and cost, where 3.4.0 planned another one for 92 of the 156 statements of a replay corpus (`Memoize` x3 became x10, `GroupAggregate` x8 became x5 in the largest one); the same 156 statements planned without the column index are identical in both libraries.
- Fixed: pg_sift: the units a lookup's keys leave were priced for the geometric mean of the keys PostgreSQL guesses for a lookup it has no statistics for (a CTE or a sub-select of ids) and the distinct values of the column, so one key was priced as tens of them and an index cut by the key (`key_cut`) kept the normal plan's btree probes where the scan reads a few units: the keys PostgreSQL expects are the keys priced where they leave a handful of units (an index cut by the key, or a key that owns a few units of a clustered table); where they leave more (an index not cut by the key reads tens of units for one key) the geometric mean stays, because the planner cannot tell a key of a few hundred rows from one of thousands and the scan is not worth its planning for the small ones. Measured on 1,000,000 rows, 600 statements over seven shapes, one to a thousand keys, warm, 4 workers: the pick is within 1.2x of the faster of the normal and the scan plan (or 5 ms of it) for 74.7% of the statements on a `key_cut` index, up from 67.2%; those that lose more than 2x and 50 ms fall from 32 to 9, and the milliseconds lost against the faster plan from 8,165 to 4,853.
- Improved: pg_sift: a correlated `LEFT JOIN LATERAL (SELECT .. LIMIT 1)` lookup whose key has no unique index, or whose condition reads the joined row alone, is read as a table once the child has run for an eighth of the rows of the table (or at once when a cache holds it): a key whose rows agree on the columns the query reads is answered from the table and a key whose rows differ runs the child, so the result is that of the child under every plan. `EXPLAIN` shows `First Row Lookups` and `First Row Keys Differ`; test `agg_first_row_dup`.
- Fixed: pg_sift: the choice between a shared CTE and its inline form was kept for a statement whatever the size of the request (the constants of a dashboard move the rows its scans read by decades), and the cost guard compared plans of different variants, so a backend that met a size after another one took the plan of the other size: the same statement planned as two plans across fresh backends. A verdict now holds within 1.4 times the rows of the largest scan of an indexed table in the shared plan, the verdict of each size band is kept as well, and a hit that is far from the size of its verdict is planned in full (or from the verdict of its own band); an interleaved run of 44 plannings of eleven sizes gives the plan of each size planned alone, 0 of 44 differ (28 of 44 before).
- Changed: pg_sift: a partitioned table whose partitions are `sift_table`s runs its aggregates as `Custom Scan (SiftAgg)`, as one whose partitions carry a `sift_col` index does (the partitions of one kind, with their columns in the same order, are one source; the partition list, pruning, workers and the top-N work as for indexes): over 200,000 rows in two partitions, `GROUP BY` a 20-value column with `count(*)` and `sum` takes 0.20 ms and 0.6 million instructions instead of 32.0 ms and 350 million, a filtered `count(*)` 0.17 ms instead of 10.6 ms, `count(DISTINCT)` of a column under a filter 2.4 ms instead of 94 ms, and a top-5 by a column 0.17 ms instead of 29 ms; with two workers the first takes 0.21 ms instead of 15.1 ms.
- Changed: pg_sift: the planner prices the scan of a numeric without a precision (stored as images) for the workers that read it cold at more operators a row the more distinct values the column has past 2,000,000 (10 more for each million, up to 40 more): the dictionary of a segment outgrows the cache and each row is a miss, so a column of millions of almost unique values keeps PostgreSQL's plan with 4 or 8 workers allowed, where the scan took 1.4 to 2.3 times as long. `GROUP BY` 6 groups, `count`, `sum`, `avg`, default costs, 5,000,000 rows 257 / 266 ms (4 / 8 workers) to 140 / 135 ms, 10,000,000 rows in one segment 801 / 681 ms to 294 / 338 ms. Columns of fewer distinct values (a million rows of 30,000 prices keep the parallel scan: 36 ms against 61), the 2,000,000-row plans and the plans with 0 or 2 workers are unchanged.
- Changed: pg_sift: `count(DISTINCT x)` of a `timestamptz`, `float4`, `float8` or `uuid` column (a leaf with no set of its own) takes its ids from a combo, as the distinct values of a joined column do, and runs in the workers; it was a serial sort of the values (a `GROUP BY` of 8 groups over 1,000,000 rows, 2 workers allowed: `timestamptz` 675 to 201 ms, `float8` 656 to 242 ms, `float4` 605 to 238 ms, `uuid` 1,542 to 582 ms, and 2.5 to 0.9 billion instructions for the `timestamptz` one without workers; the scalar `count(DISTINCT uuid)` 1,472 to 465 ms against PostgreSQL's 388 ms). The counts are exact, as before; the planner prices the new path and keeps PostgreSQL's plan where it is cheaper. Test `agg_distinct_parallel`.
- Changed: pg_sift: a `sift_table` runs the text tests of an aggregate scan (`=`, `<>`, ranges, `IN`, `NOT IN`, `LIKE 'p%'`) on the encoded strings of its groups and uses them to skip groups, as `sift_col` does, with the same code: over a million rows a group-by filtered by `s = 'x'` takes 15 million instructions instead of 44 (`sift_col`: 15), and one filtered by a prefix of a high-cardinality text column 59 instead of 448 (`sift_col`: 504). `sift_col` is unchanged.
- Improved: pg_sift: a statement with a shared CTE whose scans cost less than the work they replace by more than the margin, and whose inline plan is still refused for its total cost, is no longer planned again at ten and a hundred times the rows (that check is for the close calls): two of the eight planner runs of such a statement without a cached verdict, 473 to 261 million instructions for a four-way `min` over categories, the plan the same on 156 statements at three request sizes.
- Fixed: pg_sift: a column scan no longer sends the rest of its rows to the heap check when a promotion of the write buffer (or any other change of the index directory that replaces no segment) runs while it is open: the directory version moved, the scan stopped trusting the visibility map for all its later units, and every selected row of every page was read from the heap. The scan now asks only whether the segments it opened are still in the directory, which a merge answers and a promotion does not. 3,000,000 rows, a wide grouped scan with one promotion in the middle: 155,388 to 22,139 rows and 38,592 to 5,924 heap pages checked (158,337 to 27,971 and 39,549 to 7,574; 192,221 to 33,441 and 48,000 to 9,144).
- Improved: pg_sift: a `LEFT JOIN LATERAL` over the elements of a list that holds a lookup of its own (`LEFT JOIN LATERAL (SELECT .. FROM jsonb_to_recordset(doc -> 'list') AS p(..) LEFT JOIN LATERAL (SELECT .. FROM lookup WHERE .. LIMIT 1) m ON true) e ON true`), the shape of a per-row score that keeps the rows with no element, is answered by the scan, with the placeholder of its columns reading NULL for a row with no element (they kept the normal plan, `from_shape`, which unnests every row): the two-level ranking of 3,000,000 rows (15,000 groups, all of them) takes 0.52 s against 3.2 s with 5 workers (654 ms against 3,022 ms and 4.1 against 14.4 billion instructions in one process; 1,000,000 rows 161 ms against 745 ms, 300,000 rows 90 against 222, 100,000 rows 33 against 99).
- Fixed: pg_sift: a group-by on a text column read through a dictionary stored in blocks looked up the dictionary entry of every row of a chunk, kept or not, so a filter that keeps 82 of the 39,000 rows of the units it reads decompressed the blocks of all the entries of those rows and went on to inflate the whole dictionary; the chunk now looks up the entries of the rows it keeps when it keeps fewer than half. The first statement of a backend that groups 82 rows by a 100,000-entry dictionary takes 1.1 ms where it took 9.5 (warm 0.7 ms, unchanged); the result and the plan are the same.
- Improved: pg_sift: opening the head of a stored unit validates the bitmap of each sparse page 8 bytes at a time, where it counted the bits of each byte: a unit of a table that churned (gaps in its pages) costs 40,000 instead of 86,000 instructions to open, and the grouped sum of 1,000,000 orders after churn (store question 3, 196 units, 57 of them dirty) runs 10.9 million instructions instead of 13.5 million (-19%), with `sift.agg_dead_delta` on or off; no format change, no setting (scripts/instr-budget.sh holds the number: 908,000 to 435,000 instructions for 12 units)
- Changed: pg_sift: a CTE of the keys a request names (a literal list kept to the keys a sub-select allows, `WHERE id IN (7, 12) AND id IN (SELECT ..)`) is estimated at the keys its list is worth in the statistics of the column of the indexed table it is joined to, where PostgreSQL multiplied the two selectivities and priced the plan that keeps the CTE for one key: a request for one key that owns 6% of 3,000,000 rows planned the normal plan at 3,400 ms where the scan took 363 ms, and one for 100 keys 3,159 ms where it took 397 ms (warm medians of 3 rounds); a key of the mean size stays at one, so the requests that the normal plan answers in a few milliseconds keep it.
- Changed: pg_sift: a scan that reads a numeric without a precision (or any other wide column: interval, uuid) goes to the workers of a parallel plan unless a group key shows it: the counts, the sums of an expression, `stddev` and `variance` of a bare `numeric`, and a `uuid` key, took one process (a group key of the numeric, `min`, `max` and the lists of it still do: they show which of 1.0 and 1.00 came first). 1,000,000 rows, 8 workers and `seq_page_cost = 100`: `count(n) GROUP BY g` 100 to 44 ms, `stddev(n)` 102 to 36 ms, `sum(n * 2)` 112 to 46 ms, `GROUP BY uuid_col` 41 to 17 ms (the normal plan: 39 to 52 ms). `EXPLAIN (VERBOSE)` says `Serial Because` for the reasons the planner has (a function that is not parallel safe, a group key of equal numerics, a generic aggregate: DISTINCT, ORDER BY, no combine function) as it did for the state of the aggregate. Test `agg_wide_sum_dict`; the diff fuzz family `widew`.
- Fixed: pg_sift: a `CREATE INDEX` (also `CONCURRENTLY`) or `REINDEX` of a `sift_col` index with `key_cut` builds a table of any size with the memory it has: when the runs fill the 121 segments of the directory the build merges the oldest of the lowest level into one (the streaming merge of the key-cut segments, a row written once for each level it reaches) and gives the pages of the inputs back, instead of stopping before the table is read; the check that counted the runs and the automatic cut that was not chosen for it are gone (6,000,000 rows of 480 bytes at 64 MB with 1 worker: 114 segments, 10 s, 383 MB of segments against 381 MB for one run built with 12 GB, 54 MB of private RSS a process; the file keeps about 8% of free pages that a REINDEX with more memory does not)
- Improved: pg_sift: the scan's start is priced from what it does: a top-N that reads the order array opens no unit (the executor starts, the array of each segment is opened and its first chunk decoded: 25 to 35 us measured), and an aggregate with no groups and no search starts at 0.1 ms, not 0.25 ms. The scan now wins over the normal plan's Sort or Aggregate on 100,000 to 300,000 rows: `WHERE grp = x ORDER BY ts DESC LIMIT 10` 0.23 to 0.10 ms, with a range of `ts` 0.23 to 0.09 ms, keyset page 0.28 to 0.10 ms, `max(ts) WHERE grp = x` 0.36 to 0.10 ms (100,000 rows), `max(ts) WHERE grp = x AND ts < t` 0.69 to 0.13 ms (300,000 rows); where the scan has no unit to skip (`count(*) WHERE k = x` on a column of 100,000 distinct values) the normal plan keeps winning. No setting, no format change.
- Added: docs: [docs/COL-TAM.md](docs/COL-TAM.md) section 13 lists the features of `sift_col` and what `sift_table` offers of each (works, partial, missing, n/a with the reason).
- Improved: pg_sift: an ungrouped aggregate that the pre-aggregates do not answer (a filter on a column the units do not summarize, a unit with dead rows) counts and sums the rows of a chunk in one pass over its selection, where it scattered each row through an array of group ids that are all 0: `SELECT count(*), sum(v), sum(m)` over 1,000,000 rows (`sum(v)` an integer, `sum(m)` a `numeric(12,2)`, `sift.agg_preagg = off`) runs 54.9 million instructions instead of 90.9 million (-40%; 3.3.0: 86.1 million); a grouped aggregate is unchanged; no format change, no setting (scripts/instr-budget-ungrouped.sh holds the number: 18,136,000 to 10,987,000 instructions for 200,000 rows)
- Improved: pg_sift: an ungrouped `sum` of a `numeric` expression that the pre-aggregates do not answer adds the sum of each block of 512 values with one kernel (`sum_i64`, written once for AVX2, AVX-512 and the scalar level) instead of scattering each value through the group ids: the ungrouped `SELECT count(*), sum(v), sum(m)` over 200,000 rows (`sum(m)` a `numeric(12,2)`, `sift.agg_preagg = off`) runs 9.99 million instructions instead of 10.99 million (-9%); a grouped sum is unchanged; no format change, no setting (scripts/instr-budget-ungrouped.sh holds the number)

## 3.4.2 - 2026-10-26

- Fixed: pg_sift: the extension builds against PostgreSQL 19 again: the planner's read of a lookup's plan down to its inputs named `UpperUniquePath`, which 19 folded into `UniquePath`.
- Fixed: pg_sift: the key-cut test of the few-keys rule (an index cut by the key leaves a few-keys lookup to the scan) counts the keys PostgreSQL expects of the lookup, as the rule itself does, and no more the mean between those and the column's distinct values that prices the scan: a request for one key as a CTE or sub-select of ids (estimated at one row, priced as several) kept the normal plan's price on an index cut by the key, and a statement with a shared CTE took a plan split into one scan per arm, 1.3 to 5 times slower, or a lookup by the index of the table.
- Fixed: pg_sift: a lookup of a few keys no longer keeps the normal plan's price when the index is cut by the key (`key_cut`): the keys leave a handful of units there, so the scan reads them in a millisecond or two while the probes of the btree cost the time of the rows of the largest keys. An index not cut by the key keeps the choice it had.
- Fixed: pg_sift: a CTE of the keys a request names (a literal list kept to the keys a sub-select allows, `WHERE id IN (7, 12) AND id IN (SELECT ..)`) is estimated at the keys its list is worth in the statistics of the column of the indexed table it is joined to, where PostgreSQL multiplied the two selectivities and priced every plan that keeps the CTE for one key: a request for one key that owns 6% of 3,000,000 rows planned the normal plan at 3,400 ms where the scan took 363 ms, and one for 100 keys 3,159 ms where it took 397 ms (warm medians of 3 rounds); a key of the mean size stays at one, so the requests that the normal plan answers in a few milliseconds keep it (`agg_cte_key_list`).
- Fixed: pg_sift: the few-keys rule no longer keeps the normal plan's price for a lookup whose keys PostgreSQL estimates at a fraction of the smaller list a join of its plan reads (ten ids semi-joined to forty: one, or two): the plan of the lookup is read down to its inputs, a CTE as its query, so a request for ten ids takes the scan instead of an index nested loop that reads the rows of the largest id.
- Changed: pg_sift: the planner prices the scan of a numeric without a precision (stored as images) for the workers that read it cold: a worker inflates and indexes the dictionary of images of every unit it takes, where a backend that repeats the scan finds them decoded, so the parallel scan of a column of almost unique values scaled to 1.7x with 3 processes while PostgreSQL's plan scales to 3.4x. `GROUP BY` 6 groups, `count`, `sum`, `avg` of 2,000,000 almost unique values (2 / 4 / 8 workers allowed, default costs): the planner took the scan at 103 / 91 / 86 ms against PostgreSQL's 96 / 70 / 70 ms (up to 31% slower), it now keeps PostgreSQL's plan (79 / 62 / 62 ms); 500,000 rows 27.7 against 24 ms, now 24 to 26 ms. Without workers the scan is still taken (135 against 213 ms). No effect when `parallel_setup_cost` is 0.
- Fixed: pg_sift: an `ORDER BY .. LIMIT n` that reads a sorted run (`order_columns`) over a filter that few rows pass (`WHERE ts >= x ORDER BY ts LIMIT 10` on 1,000,000 rows of a scattered timestamp) read the sort key of every entry it walked by position, 83 instructions each, where 3.3.0 decoded the unit's column; it reads the keys of the rows that pass only: 113 million instructions a statement (3.4.0) become 56 million, 3.3.0's 59 million.
- Fixed: pg_sift: an `UPDATE` of a row of a `sift_table` that is the referencing table of a foreign key failed with "don't have transaction information for this type of tuple" in 3.3.0, 3.4.0 and 3.4.1, whatever column it changed: the integrity trigger asks the slot of the old row whether the current transaction made it, and the table's slot answered with the error of a virtual slot. The slot now answers from the xmin of the row, as it already did for `xmin`. `INSERT`, `DELETE`, and every statement on the referenced table were not affected; `tbl_foreign_keys` runs them all, with the three actions, on heap and `sift_table` pairs.
- Fixed: pg_sift: `CREATE INDEX` of a `sift_col` index with `hll_columns` and a leaf that is NULL in nearly every row (a soft-delete stamp, which keeps exception cubes) but no list of elements failed with "segment 1 is corrupted", and the same index created before its rows could be written by an `INSERT` and then refuse `VACUUM` and every scan of the segment: the offset of the sketches counted a directory of exception element blobs that the segment does not hold. The index is built and the sketches are read (a table of 60,000 rows: `col_verify` passes and `approx_count_distinct` answers from the two units' sketches); an index already built with the fault needs a `REINDEX`.
- Improved: pg_sift: the heap check of the rows of an index that no VACUUM covers (a table loaded and not vacuumed, or one an old transaction holds back: no page is all-visible) tests a row without a call through a test of its TID and without the call of `TransactionIdPrecedes`: the grouped sum of 1,000,000 rows with 1% dead rows and no VACUUM (`SELECT k, sum(v), count(*) .. GROUP BY k`, PostgreSQL 18, user instructions of the backend) runs 164.5 million instructions instead of 204.4 million (-19%; 3.3.0: 180.2 million, 3.4.1: 206.3 million), `SELECT count(*), sum(v), sum(m)` 197.1 million instead of 237.0 million (-17%); a table with the pages all-visible does not check its rows and is unchanged; no format change, no setting (scripts/instr-budget-vet.sh holds the number: 20,024,000 to 12,030,000 instructions for 200,000 rows)
- Fixed: pg_sift: a statement with workers failed with `ExtensibleNodeMethods "SiftAggRows" was not registered` ("parallel worker") when a scan with workers whose filter starts a sub-plan (`x IN (SELECT ..) OR ..`) sat beside another sub-plan of the statement that holds a `Custom Scan (SiftAggRows)` node, such as a CTE of per-row aggregates grouped with `count(DISTINCT ..)`: the workers registered the `SiftAgg` node and not the `SiftAggRows` one when they read the sub-plans of the statement; both are registered now (test agg_rows)

## 3.4.1 - 2026-10-26

- Fixed: pg_sift: a request for one or a few keys of a lookup (a CTE of ids joined to the table) took the scan at a price far above the normal plan's, and lost: the 3.4.0 charge on the normal plan of an aggregate over the elements of a jsonb array (the excess of the table's rows over the rows PostgreSQL expects, and of the keys over its count) read the rows the keys pick as a lost estimate, hundreds of times the plan's price for a single key, so the scan won whatever it cost. A lookup of up to four keys, as PostgreSQL counts them, keeps its btree probes at the price they are planned for; longer lists keep the charge. Measured (1,000,000 rows, one key of about 200 rows): the normal grouped aggregate was charged 64,555.85 where its plan is priced 48.29, and lost to the scan priced 3,507.02; with 3,000,000 rows the grouped aggregate over a jsonb map for one small or one medium key took 8.8 and 12.4 ms in the scan and takes 0.9 and 6.0 ms now (the normal plan takes 1.1 and 7.0 ms), and a workload of single-key lookups that took 340 ms each takes between 2 and 6 ms. Limits: one key of a very large entity takes the normal plan, as in 3.3 (3,000,000 rows: 13.9 ms in the scan, 20.9 ms in the plan), and lookups of 10 keys or more keep the charge, so the scan still wins there (22 ms for 10 keys where the normal plan takes 76 ms); a choice made at run time is 3.5 work (`agg_few_keys_pick`).
- Fixed: pg_sift: a `CREATE INDEX` (also `CONCURRENTLY`) or `REINDEX` of a `sift_col` index with `key_cut` on a table whose rows made more runs than the 121 segments of the directory with `maintenance_work_mem` failed after reading as many rows as the runs held ("needs more than 60 segments", 136,750,000 rows at 64 MB with 1 worker): the build now holds the rows of up to 32 superblocks before it reads the table, counts the runs, uses fewer parallel workers when that makes them fit (6,000,000 rows of 580 bytes at 64 MB: 2 participants failed at 60 segments each, serial builds in 52 runs, 28 s), and otherwise stops at once with the `maintenance_work_mem` it needs; the automatic decision gives up a cut that does not fit with a `NOTICE` (`col_key_cut`, `col_key_cut_auto`). Limits: an explicit `key_cut` build of a table of wide rows still needs `maintenance_work_mem` of about (workers + 1) x 2 or 3 GB (the message names it), and a table that needs more runs than the directory holds at the memory given cannot be built with `key_cut` until the runs are merged inside the build, which comes in 3.5.
- Fixed: pg_sift: a session that used the column aggregate pushdown, a partitioned table, a shared CTE or a `sift_table` took up to four of the ten relcache callback slots PostgreSQL gives a backend, one for each cache, and the session ended with `FATAL: out of relcache_callback_list slots` when the libraries loaded next to pg_sift (a distributed-table or time-series extension, a scheduler, ...) had taken the rest. pg_sift takes none now: the partition bounds and facts follow the rows of `pg_class` through syscache callbacks (sixty-four slots of their own), a `sift_table` notices the invalidation of its relation by the `rd_amcache` mark the relcache frees with it, and the CTE verdicts, whose keys carry the size, file and indexes of each table, are no longer dropped by an invalidation. Measured on PostgreSQL 18: the server takes 2 of the ten slots, `citus.so` 1 at load and 4 more on the first query when the `citus` extension is created in the database, `pg_cron` 1 and `timescaledb` 1, so the second cache of pg_sift was the eleventh registration; the test `relcache_slots` finds 8 free slots without pg_sift and 4 with 3.4.0, 8 and 8 now. The ten slots are shared by every library, so another library can still use them up: with the `citus` extension created, `timescaledb`, `pg_cron` and the server leave one spare.
- Fixed: pg_sift: a statement whose filter is a set of ranges on the zones of a `sift_col` index (the index with `key_cut`) no longer starts the setup of a scan for each morsel: the setup is made once for the scan, and an index with many units no longer pays about 0.3 s for each statement (a typical scan of the lab went from 8.3 ms down to 1.6 ms; an index without `key_cut` gains in proportion to its units, and `scripts/instr-budget-zone.sh` bounds the pruning of the case).
- Fixed: pg_sift: a filter on a text column (`LIKE`, a function of the column) is evaluated once for each entry of the dictionary of the column before the scan reads a row, and that loop could not be interrupted: `statement_timeout` and a cancel waited for the whole loop (about 200 ms over 2,800,000 rows, seconds with a dictionary of a hundred thousand long names and an expression that costs microseconds an entry; the test `interrupt_dict_filter` ends a 3,192 ms statement in under a second). The evaluation checks for interrupts now.
- Fixed: pg_sift: the first N rows of each group asked with a window function (`row_number() OVER (PARTITION BY k ORDER BY o)` and `rn <= N`, N > 1) ran in the scan with the groups' arrays when its plan priced less than the window's, which left out the work above the scan: the executor spends 1.4 to 2 us on each group to unnest its rows (PostgreSQL prices the rescan of a function scan at the rows it returns, 0.11), so 6,263 groups of the top 3 rows cost 12.3 ms there, priced 4.5. The plan of the arrays now pays 0.9 for each group it unnests (360 operator costs). 1,000,000 rows, 31,675 of them in range, 6,263 groups: the window plan 26.2 ms, the arrays 32.8 ms (was chosen, priced 17,693 against 18,857); with 30 or 40 groups the scan stays (22.4 ms against 33.2); 100,000 rows unchanged (2.3 ms). Test `agg_topn_rownumber`.

## 3.4.0 - 2026-10-25

- Changed: pg_sift: the byte budget of the cube of a `sift_col` unit counts what the cube holds (every measure of a kept dimension, with its minimum and maximum) and is 40% of the unit's encoded bytes, at least 8 kB, where it was 2% of the first three measures, so an index of sixteen columns whose dimensions have 50 to 200 values keeps the dimensions in every unit and answers a `GROUP BY` of any of them from the pre-aggregates: the last of eight dimensions of 50 values over 500,000 rows goes from 2.3 ms to 0.28 ms (the normal plan takes 63 ms), the index grows by 8.9% for the cube instead of 6.7% at 1,000,000 rows (at most 41% for 32 columns of 200 values) and the index of the store table is the same file. Dimensions of alike size keep the order of the columns, so units do not keep other ones each, the calendar cells of the dates share 10% as built, and a column the cube holds in every unit no longer keeps a key list (`key_columns`). Segments written afterwards; `REINDEX` or `sift.col_refresh(index)` rewrite the older ones. No format change.
- Fixed: pg_sift: `count(DISTINCT d)` of a `date` column that is not among the sixteen dimensions of the cube, but whose calendar cells the units keep, counted the months (or years) of the cells instead of the days: 14 for a column of 400 days over 14 months, with `Units From Pre-aggregates` of every unit. The count reads the histogram of the column itself, and the units of a column without one are scanned.
- Changed: pg_sift: the sum and average of a `numeric` column without a precision (read through a dictionary of the unit's values) run in the workers of a parallel scan, where they kept the scan in one process: the scan was then priced 10,086 against 16,063 for PostgreSQL's plan, which runs with the default 2 workers, and was slower than it (1,000,000 rows, 16 groups: 70 ms against 52 ms). A sum that leaves the 38-digit integer form in a worker (NaN, Infinity, past 38 digits or 128 bits) crosses to the leader as a numeric and is added there with the same arithmetic; the sums of an expression of a nullable input take the same route. 1,000,000 rows: 33 ms against 55 ms for the normal plan at the default 2 workers (leader instructions 743M to 293M), 30 ms against 50 ms with 8 workers and the page costs of a large database; 100,000 rows unchanged (7.5 ms). Test `agg_wide_sum_dict`.
- Changed: pg_sift: a top-N scan for each key an enclosing query sets (`CROSS JOIN LATERAL (.. WHERE device = d.device ORDER BY ts DESC LIMIT 3)`, a correlated scalar sub-select with `ORDER BY .. LIMIT`) is the `SiftAgg` top-N of the rows of the key, started over for each value, and no longer a read of the whole table for each: 20 devices over 1,000,000 readings with no btree on the key take 1.1 ms where the normal plan takes 739 ms; a top-N below a few superblocks reads its units for each value (100,000 rows: 2.4 ms for 20 devices, 60 ms before). The scan has no joins, partitions or window functions then, and a text key keeps the normal plan. No format change.
- Fixed: pg_sift: `date_trunc(unit, tstz, 'UTC')` and `tstz AT TIME ZONE 'UTC'` (and `GMT`) grouped by days of the UTC clock in a session whose time zone defines that name itself (the POSIX zone `UTC+3` has the abbreviation `UTC` at -03, which PostgreSQL reads first on 18 and later), so the scan and the calendar cells gave other groups than the normal plan there; the planner and the executor now check the name against the session zone (a plan made in another zone included) and evaluate the key for each value where it is not the UTC. The names with a slash (`Etc/UTC`) and every other session are unchanged.
- Changed: pg_sift: a `sift_table` reads only the rows its filters kept of a unit that keeps few (`read_sel`), and goes on with those rows alone when they are fewer than an eighth of it (`narrow`), as `sift_col` does: over a million rows the single-value equality that kept one row in a thousand takes 5.5 million instructions, from 13.4 (`sift_col`: 6.5). `sift_col` is unchanged.
- Changed: pg_sift: a `sift_table` decodes the columns of a unit straight into the executor's buffers in their own vector type, with the code `sift_col` uses (`sift_aggsel_read_lane`), instead of building a vector of keys, converting it and copying it: over a million rows an unfiltered group-by goes from 91 to 72 million instructions (`sift_col`: 73), and a filtered one with three conjuncts from 72.5 to 53 (`sift_col`: 54). `sift_col` is unchanged.
- Changed: pg_sift: the dictionary of a text column that a segment stores in blocks (column format minor 14, dictionaries of 1,024 strings or more that are not stored whole) is not inflated whole by the first statement of a backend that groups by the column: the blocks of the strings of the groups the request keeps are decompressed one at a time, and the dictionary is loaded whole, and kept for the backend, once those blocks add up to an eighth of the dictionary's (`count(DISTINCT text)`, the key of a group by or a lookup of the entries are read this way; a predicate on the text, a top-N, `ROLLUP` or any use that reads the dictionary whole load it when it reads). On a fixture of 1,000,000 rows with two text dictionaries of about 97,000 strings of 25 bytes each, a request over 82 rows grouped by one of them, forced scan, new backend: the first statement runs 1.0 ms where it ran 5.6 ms with the default `dict_stored` and 2.6 ms with `dict_stored = 17` (51.3M instructions to 7.1M, and 9.7M to 6.5M), so the stored form no longer pays on it. Test `agg_dict_blocks_lazy`.
- Changed: pg_sift: a `uuid` column of a `sift_table` is stored as a dictionary of its values, as in a `sift_col` index, and its top-N (`ORDER BY <uuid column> LIMIT n`) reads the codes of the stored dictionary: over 200,000 rows 8.5 ms become 2.7 ms (1,024 distinct values: 6.9 ms become 0.9 ms); the table grows by 0.2% with unique values and shrinks by a third with 1,024 random ones, and writing the groups costs 1% to 3% more. Groups written before stay readable and become dictionaries when rewritten.
- Changed: pg_sift: `p ->> 'k' IS [NOT] NULL` and `count(p ->> 'k')` of a member of the elements of `jsonb_array_elements(doc -> 'list')` (the way `jsonb_to_recordset` was already read), declared typed by the index (`list[]{k float8}`), are the null test and the count of the typed leaf of the member, as the null test of a member of the document is: `->>` is NULL for a missing key and a JSON null, the leaf too. A member that holds a value the leaf cannot store keeps the normal plan, as every read of it does. 1,000,000 events with `count(p ->> 'score')`, `count(*)` and `avg((p ->> 'score')::float8)` of the parts that have a score take 11.3 ms where PostgreSQL's plan takes 583 ms (`elem_member_null`); 200,000 documents of 1 to 9 elements, `WHERE p ->> 'score' IS NOT NULL` 3.6 ms against 404 ms, `count(p ->> 'score')` 0.5 ms against 381 ms. No format change. Test `agg_elem_member_null`.
- Changed: pg_sift: the aggregate scan reads a leaf for the surviving rows only when fewer than a thirty-second of the unit's rows still count (it was an eighth): over a million rows, with a filter that keeps 6% of the rows, a group-by takes 18.5 million instructions instead of 30.6, and one that keeps 0.1% still takes 6.4; between 3% and 12% the gather cost more than it saved.
- Improved: pg_sift: an `ORDER BY .. LIMIT n` scan with n above 32 over an integer, date or timestamp column of an index with `order_columns` answers a superblock (and a unit) from the head of its sorted run once the heap holds n rows and the last entry of the head is after the heap's bar: no row after the head can enter the result, so the unit is not read. `ORDER BY ts DESC LIMIT 1000` over 3,000,000 rows with `ts` in no order takes 2.6 ms, not 4.2 ms (3.0 ms, not 3.9 ms when the rows are written by `(tenant, ts)`), and the cost of the units read is no longer linear in the rows (slope 0.30, was 0.42); the index and the answers do not change.
- Changed: pg_sift: `date_trunc`, `extract` and `date()` (`::date`) of a `timestamptz` in the session's time zone are grouped by the step of the zone's clock (a second, minute, hour or day) the scan numbers from the rows, as it does for a `timestamp`, and the expression is evaluated once per step and not per row: the offsets of the zone over the span of the data are read when the executor starts (PostgreSQL's own zone rules, so daylight saving time, 30 and 45-minute offsets and the offsets of local mean time give the rows of the normal plan, and a cached plan follows `SET TIME ZONE`); `date_trunc` to a second, minute or hour keeps the offset of the instant, so the repeated hour of a daylight saving change gives two groups as in PostgreSQL. A day of a zone with daylight saving time over 3,000,000 rows takes 20.8 ms and 284 million instructions (95 instructions a row) where the scan evaluated the key per row, 478 ms and 5.8 billion instructions, and PostgreSQL's plan 533 ms (1,000,000 rows: 8.0 ms, 164 ms, 175 ms). A call or unit that is not a step of the clock (`epoch`, `second`, `julian`, a zone named in the call) keeps the normal evaluation. No setting, no format change.
- Improved: pg_sift: on arm64 the chunk codec runs the fused range filter over bit-packed values (`unpack_range32`), the expansion of runs and the float4 keys of a decimal-scaled chunk with NEON kernels, bit-identical to the scalar ones: 4096 values on an arm64 core take 1750 to 313 ns (the range filter at 3 bits a value), 1125 to 273 ns (run expansion) and 1375 to 719 ns (float4 keys); the kernels that did not beat the scalar loop on that CPU (`max32`, the prefix sums, `scale32/64`, the float8 and float4 conversions) stay scalar.
- Added: pg_sift: `CREATE INDEX` on a `sift_col` index says in a NOTICE which foreign key columns of the table are not columns of the index: a join on them keeps the normal plan (`join_condition`), and the scan cannot filter the table by the keys of the rows another table's filters keep. The TPC-H-like cases declare their foreign keys and index the join keys, and `tpch_q3`, `tpch_q10` and `tpch_q12` then run on the scan (1,000,000 lineitem rows: 69, 115 and 64 ms for PostgreSQL's plan, 18, 23 and 13 ms with the lookup tables kept from an earlier query); the first query after a write loads those tables, and `tpch_q3_cold`, `tpch_q10_cold` and `tpch_q12_cold` time that run (31, 23 and 38 ms against 73, 112 and 60 ms: the planner prices the load, and the scan takes the cheaper table as the scanned one). Test `col_key_columns_auto`.
- Fixed: pg_sift: the ordered output of an aggregate scan (the groups emitted in the order the parent sorts by) kept its description of the order in an array of the planner's stack that was gone when the plan was made, so a scan with a semi or anti join under an `ORDER BY` or a window read garbage ("operator 0 is not a valid ordering operator", a floating-point exception, AddressSanitizer stack-use-after-return); the array lives as long as the paths. It also builds on PostgreSQL 17, which has no `SortGroupClause.reverse_sort` (the direction is the sort operator's there).
- Changed: pg_sift: the planner prices the test of an IN list by the structure that makes it: a hash table of values spread over a wide key costs a row 5 to 10 ns (a byte table in the cache 1, a bit table 2, a byte table wider than the cache 10), where every list cost the same fraction of a nanosecond. A list of 300 to 1,000 values spread over a key of 1,000,000 rows took the scan, which ran in 7.6 to 11.3 ms (51M to 63M instructions) where the btree plan runs in 0.3 to 1.0 ms; it now keeps the btree plan. Test `agg_in_list_cost`.
- Changed: pg_sift: the dictionary of a text column that a segment stores in blocks (column format minor 14, dictionaries of 1,024 strings or more that are not stored whole) is not inflated whole by the first statement of a backend that groups by the column: the blocks of the strings of the groups the request keeps are decompressed one at a time, and the dictionary is loaded whole, and kept for the backend, once those blocks add up to an eighth of the dictionary's (`count(DISTINCT text)`, the key of a group by or a lookup of the entries are read this way; a predicate on the text, a top-N, `ROLLUP` or any use that reads the dictionary whole load it when it reads). On a fixture of 1,000,000 rows with two text dictionaries of about 97,000 strings of 25 bytes each, a request over 82 rows grouped by one of them, forced scan, new backend: the first statement runs 1.0 ms where it ran 5.6 ms (51.3M instructions to 7.1M). Test `agg_dict_blocks_lazy`.
- Changed: pg_sift: `sift.table_set_paths` says, in a NOTICE, when the column groups of the table written before the declaration may keep the column as one leaf; its paths are used once every group has their leaves (`VACUUM FULL` rewrites them), and until then the plan stays the normal one.
- Fixed: pg_sift: a sum or average of numerics that came to exactly 2^127 - 1 (170141183460469231731687303715884105727, at any scale) was returned as NaN: the integer of a sum, 128 bits wide, was read as the NaN the numeric programs mark with that value. A `numeric(38,0)` column with two values of 99999999999999999999999999999999999999 and 70141183460469231731687303715884105728, a `numeric` without a precision holding the value, and a sum of products that ends there gave NaN where PostgreSQL gives the number; the sums convert their integer without the NaN reading now (test `agg_sum_int128_max`). No format change.
- Changed: pg_sift: the fixed cost of opening a scan in the planner's price is 100,000 operator costs (0.25 ms), where it was 210,000: a lookup of 500 to 700 rows in a table of 20,000 to 1,000,000 was left to the btree plan (a scan of 50 to 100 us against 190 to 370 us) because the scan was charged 0.5 ms before it read a row. The cost bands of the plan-choice gate are re-recorded for the shapes whose scan price falls.
- Improved: pg_sift: a comparison of two columns of one row (`l_shipdate < l_commitdate`, `end_at > start_at`: int2, int4, int8, date, timestamp, float4 and float8 columns of one type) is a predicate of the scan, evaluated on the vectors of both columns by one kernel per SIMD level (AVX2, AVX-512, NEON) with PostgreSQL's float order and the NULL rule, not one expression resolved for each tuple: the five filters of the TPC-H-like Q12 on `lineitem` (`tpch_q12`, 1,000,000 rows, scan forced) 220 to 4.5 ms and 832 to 40 instructions a row, a single column comparison 64 to 4.2 ms (425 to 31 instructions a row; 76 with the scalar kernel); 4.9 ms on an arm64 core.
- Changed: pg_sift: the lookup table of a join (the rows of a joined table that the scan keeps in a hash by key) is loaded from the leaves of a `sift_col` index of the table when one holds the columns the join reads (the keys, the columns of the expressions, plain integer, date, timestamp, text and numeric(p <= 18) columns), where it was read from the heap: the units are decoded a unit at a time, the rows the index says are visible get a tuple of the columns, and the table's own conditions of the form `column <op> constant` are tested on the decoded leaves before a tuple is made, which drops a condition that tests nothing more. A table with no such index, a unit holding a value the leaf cannot store (a NaN in a numeric) or a load by an index scan keeps the heap. `EXPLAIN ANALYZE` shows `Lookup Column Loads`; `sift.agg_lookup_columns = off` turns it off. `lookup_columns` (1,000,000 orders joined to a table of 200,000 rows that the query loads) takes 23.3 ms where the heap load took 33.5 ms, against 152 ms for PostgreSQL's plan; the load alone of the fact-first TPC-H-like Q3 and Q10 joins (400,000 lineitem rows) is 102 M to 41 M instructions.
- Fixed: pg_sift: `sift.explain_why` for `ORDER BY .. LIMIT` told that the top-N scan was used when the plan was the b-tree's: the arm compared its path with the whole cost of the ordered paths, before the LIMIT was costed (an index that gives the order reads only the rows asked for). The row now waits for the final paths and, when the b-tree wins, names the reason `cost` with both costs (`the scan costs 3124.66, the normal plan with the LIMIT 19.11`).
- Changed: pg_sift: the build, promotion and forced rebuild of an index with `key_cut` sort the rows by the key copied next to each reference with an inlined comparison, and cut the units from the sorted references without copying the rows into a window first (two copies of every row less): `CREATE INDEX .. WITH (key_cut = 'ts')` of 200,000 rows with a scattered `ts` takes 1.53 billion instructions, not 2.18 billion (a page-cut index: 1.23 billion), and the build CPU is 1.4 to 2.0 times a page-cut index's at 800,000 rows, where it was 1.6 to 2.9. A merge of segments and the promotion of a write buffer copy a row without arrays leaf by leaf, not through a decoded row: a forced `sift.col_merge` after churn of 200,000 rows takes 0.58 billion instructions on a page-cut index, not 0.70, and 0.86 billion with `key_cut`, not 1.22. No format change.
- Changed: pg_sift: the plan in full that checks a kept choice between a shared CTE and its inline plan (or between the flattened form and the one as written) comes every 8th hit of an entry the first time and, each time it confirms the choice, after twice as many hits (8, 16, 32, 64), where it came every 8th hit always; a plan in full that changes the choice starts at 8 again. A statement of 27 KB repeated in one backend: the planning of 70 runs averaged 6.1 to 6.7 ms (median 5.6) where it averaged 8.3 to 9.2 ms (median 6.1 to 7.1), and the plans in full over 20 ms went from 8 (184 to 231 ms in all) to 1 to 3 (26 to 75 ms). The name of the shared table of the choices changed (`sift_agg_verdicts_2`).
- Changed: pg_sift: an `ORDER BY .. LIMIT n` scan over a `float4` or `float8` key hands the rows of a unit to the heap through the min/max and compare kernels as it does for an integer key (a float key was tested and pushed row by row): `ORDER BY value DESC LIMIT 10` over 1,000,000 readings no longer reads every row of the units whose best value cannot pass the n-th; the order is PostgreSQL's (`-0` equals `0`, `NaN` is the largest, the infinities), no option, no format change.
- Changed: pg_sift: the sum and average of a product of two decimal expressions whose digits together are 39 to 48 (`price * (1 - disc) * (1 + tax)` of three `decimal(15,2)` is 47) is added by the scan, as the sums of up to 38 digits are: each pair is multiplied in 256 bits and added to the 128-bit sum with the number of times it wrapped, where the product went through `numeric_mul` for every row (`sum_product_47`, 1,000,000 rows, six aggregates of a pricing summary: 466 ms in PostgreSQL's plan, 40 ms now; a scan that evaluated the product per row took 830 ms against 723). A product of more digits, a `DISTINCT` sum and a `min`, `max` or `count(DISTINCT)` of it keep the normal plan; a sum whose carries pass 2^63 (2^31 rows of the largest value of the 48 digits) raises an error. Test `agg_sum_product_exact`.
- Changed: pg_sift: the merge of segments of an index with `key_cut` copies each row once, from the unit it was decoded into to the unit it is written to, where it copied it three times; a forced merge of six segments of a 600,000-row table is 20% shorter (1.82 to 1.48 s and 2.49 to 1.96 s for the same table at two sizes). The build and the promotion use the same cut and are unchanged.
- Changed: pg_sift: a `sift_table` applies the integer filters of an aggregate scan (comparisons, ranges, `IN`, `NOT IN`, not null) on the encoded chunks of its groups, as `sift_col` does, with the same code (`agg/chunkfilter.c`): a filtered group-by over a million rows drops from 42-101 to 33-85 million instructions, and `sift_col` is unchanged.
- Changed: pg_sift: the aggregate scan of an inner join of three or more tables, or of two when the plan without it costs 1,000 or more, is priced with each table in turn as the scanned one and the others as its lookup tables, and the cheapest is planned, where the first table of the FROM list was the only one tried; a lookup table read whole is priced at the rows it loads (20 operators a row from the heap with the pages it reads, 8 from the leaves of a column index that holds its columns: `sift.agg_lookup_columns`), and a test between two columns of the scanned table is priced at the rows that survive the zones (it is evaluated once for each row). A join such as TPC-H Q3, Q10 or Q12 is answered by the scan of the fact table whichever way the FROM list is written, also cold, when the lookup tables are read from column indexes; with no covering index a lookup table that the heap makes dearer than the normal plan keeps the normal plan. No setting: `sift.enable_agg = off` turns the scan off.
- Changed: pg_sift: the small-set summaries of a unit (the key lists a scan prunes units with) are checked in one loop for each width of the list, where each key was a call, and a dictionary written uncompressed or in blocks is read into its own arrays, where it was read into a buffer and copied. On a fixture of one million rows (434 units, a text column of 100,000 values) a request over about 80 groups runs 0.5 ms warm where it ran 2.4 ms, a request without a text column 0.7 ms where it ran 2.5 ms, and the first statement of a new backend 2.9 ms where it ran 5.6 ms (17.2M to 11.7M instructions); the planning of a cold session is unchanged (38 ms and 6 ms). Test `col_set_summary`.
- Fixed: pg_sift: a `sift_table` hands the byte order of a text dictionary (`sift.table_dict_lookup`) to the aggregate scan, as `sift_col` does: the top-N of a text column orders its units by exact codes instead of reading the strings.
- Fixed: pg_sift: a `sift_table` is read by the rewrites that only looked for a table with a `sift_col` index: the decorrelation of a `LATERAL` sub-select that aggregates and of `EXISTS` / `NOT EXISTS`, the inline plan of a shared CTE and the cost of a statement that reads one. Such a statement on a table of this access method ran as the normal plan with a sub-plan for each outer row (test `tbl_agg_rewrites`).
- Changed: pg_sift: `sift.enable_col_scan` is on by default: a `sift_col` index with the `sift.col_*_ops` classes is a planner path (bitmap scan, Index Scan) priced against the b-trees and the sequential scan, so a table whose b-trees were dropped reads its point, range and `IN` lookups from the index and not by a sequential scan (1,000,000 rows: `id = x` 29.5 to 0.21 ms, `id IN (100 ids)` 43.4 to 0.62 ms, a range of 100 rows 32.8 to 0.31 ms, a range of 1,000 rows 40.6 to 2.9 ms); with the b-trees present the plans are the b-tree's (24 of 26 shapes identical at 100,000 and 1,000,000 rows), and a few `count(*)` shapes that SiftAgg answered take the bitmap scan by cost, up to 0.16 ms slower. `SET sift.enable_col_scan = off` restores the earlier planner. Plan gate lanes unchanged.
- Added: pg_sift: the reloption `order_array_columns` of a `sift_col` index (a subset of `order_columns`: ordered runs, optionally with a global order array for any LIMIT) keeps, for each integer, date or timestamp column it names and each segment written afterwards, the sorted array of every row's key and heap address. An `ORDER BY column LIMIT n` over one such key, with no filter, join or aggregate, merges the arrays of the segments and of the write buffer and fetches `n` heap rows in the order of the key, whatever the number of units, where the scan opened every unit; the heap confirms each entry as it does a b-tree's (a dead row or an address another row took is skipped); EXPLAIN shows `Global Order`. A segment without the array keeps the scan of the units; a 3.3 library refuses a minor 21 segment (`ALTER INDEX .. RESET (order_array_columns)` and `REINDEX` first); `sift.col_verify` checks the arrays.
- Added: pg_sift: the reloption `order_array_columns` of a `sift_col` index (a subset of `order_columns`: ordered runs, optionally with a global order array for any LIMIT) keeps, for each integer, date or timestamp column it names and each segment written afterwards, the sorted array of every row's key and heap address. An `ORDER BY column LIMIT n` over one such key, with no join or aggregate and no filter but comparisons of the key with constants (a range, `ts >= x`: the scan searches each segment's array for the first entry of the range and ends at its other end), merges the arrays of the segments and of the write buffer and fetches `n` heap rows in the order of the key, whatever the number of units, where the scan opened every unit; the heap confirms each entry as it does a b-tree's (a dead row or an address another row took is skipped); EXPLAIN shows `Global Order`. A segment without the array keeps the scan of the units; a 3.3 library refuses a minor 21 segment (`ALTER INDEX .. RESET (order_array_columns)` and `REINDEX` first); `sift.col_verify` checks the arrays.
- Added: pg_sift: an `ORDER BY` of a key with a global order array (`order_array_columns`) and more integer, date or timestamp keys (`ORDER BY created_at, id LIMIT 50`), with or without the page after a key (`WHERE (created_at, id) > (x, y)`), reads the array from the key of the page: the scan takes the whole groups of rows that share a value of the first key, puts each in order by the heap row's other keys, and stops at the group that makes the rows wanted, where the units were read (1,000,000 and 3,000,000 rows, scattered timestamp, no b-tree: the page of 50 rows 17.1 and 17.3 ms for the units, 0.29 ms; descending 14.4 and 53.4 ms, 0.34 and 0.32 ms; a b-tree 0.13 to 0.21 ms); a first key that many rows share (more than 4,096 entries read, or 64 for each row wanted) reads the units. Test `col_topn_gorder`, fuzz family `gorder`.
- Added: pg_sift: the reloption `order_array_group` of a `sift_col` index (one integer, date or timestamp column) sorts the arrays of `order_array_columns` by that column and then the key, so `WHERE grp = x ORDER BY ts DESC LIMIT 10` (also with a range of the key or the page after a key) reads the entries of the group x in each segment and `n` heap rows, whatever the rows of the table, where the units read every unit of the group (1,024 groups, scattered timestamp, no b-tree, 100,000 / 1,000,000 / 3,000,000 rows: 0.25 / 0.32 / 0.22 ms against 0.46 / 1.6 / 3.7 ms for the units; a b-tree on (grp, ts) 0.08 / 0.10 / 0.05 ms); opt-in: about 6 bytes a row for each array column (+74% of a 4-column index with one), CREATE INDEX +65% over the runs; a query that does not pin the group reads the units; column format minor 21. Test `col_gorder_group`, fuzz family `gorder`.
- Fixed: pg_sift: a parallel worker of an aggregate scan that starts a sub-plan or a child plan did not make the snapshot of the statement its active snapshot, so a build with assertions stopped the worker (`GetActiveSnapshot() == queryDesc->snapshot`) and the sub-plan ran with the restored copy of the leader's active snapshot instead of the one it was started with; the worker now pushes it for the life of the executor states. No plan changes.
- Changed: pg_sift: a column chunk of plain 3, 5, 6 or 7-byte values (a timestamp with microseconds spread over years) is decoded with one 8-byte load per value and not a library copy per value. No format change.
- Fixed: pg_sift: a `VACUUM` that vacuumed the other indexes of the table with parallel workers failed with "cannot assign transaction IDs during a parallel operation" when a merge of a `sift_col` index in its leader decided `key_columns = 'auto'` again (the write of the option); the merge leaves the decision to the next one outside parallel mode (`col_vacuum_parallel_keys`).
- Fixed: pg_sift: an `ORDER BY` a text column `LIMIT n` over a `sift_table` whose column was added with `ALTER TABLE .. ADD COLUMN` after its column groups were written ended the backend with a segmentation fault: the column of those groups, all NULL, had no dictionary (test `tbl_topn_absent`).
- Improved: pg_sift: a group by of one integer column with millions of groups takes fewer instructions a row and its time grows with the rows, not faster: the groups of one key stay in the arrays past a million (they were rehashed into the hash table at 1,048,576, which cost 2.4 times the instructions for twice the rows), a key that outgrows its arrays no longer renumbers the groups (the arrays are copied whole), and the slots of the ids and groups of the next 16 rows are fetched ahead. At `work_mem = 1GB` `count(*), sum(v)` of 3,000,000 rows takes 583 to 325 ms with 1,500,000 groups (the normal plan 1,151 ms); the case `group_many_groups`, with the default workers, takes 246 ms against 1,047, and of 4,000,000 rows 893 to 325 ms with 4,000,000 groups and 349 to 232 ms with 1,000,000 (10,000 groups: 23 to 22 ms, 100,000: 78 to 74 ms); with about as many groups as rows, 1,000,000, 2,000,000 and 4,000,000 rows take 128 to 93, 450 to 165 and 920 to 340 ms. On an arm64 core the same query takes 191 to 114 ms (1,500,000 groups), 155 to 126 ms (1,000,000) and 360 to 181 ms (4,000,000). The memory limit that caps the arrays is the same; two or more keys keep the limit of 1,048,576 slots.
- Changed: pg_sift: an `ORDER BY .. LIMIT n` scan over an `int2`, `int4`, `int8`, date or timestamp key hands the rows of a unit to the heap through the range and min/max kernels, 256 rows at a time and the segments by the best value they hold, where it tested and pushed them one by one: a descending scan of a column that follows the table (the latest rows) no longer pushes every row of the unit (15.1M instructions at 1,000,000 rows, 8x the ascending scan, now 0.7M: 1.2 ms to 0.07 ms, flat to 3,000,000 rows) and a random key reads a unit with fewer instructions; no option, no format change; units under 512 rows and filters that leave under a sixteenth of a unit's rows keep the row loop; the kernels are the existing ones of x86-64 (AVX2, AVX-512) and arm64 (NEON).
- Improved: pg_sift: the slots of the (group, value) pairs of a joined `count(DISTINCT)` are fetched ahead of the adds, and so are those of the merge of the workers' sets: with a set past the caches (6,000,000 rows, 600,000 groups, two columns of a lookup table) the scan takes 661 to 428 ms (the normal plan 1,318 ms), the case `count_distinct_lookup_pairs`; a set that fits the caches does not change (1,000,000 rows, 190,000 groups: 60 and 71 ms, within noise).
- Improved: pg_sift: a short IN list of integers that has runs of consecutive members (`id IN (7, 8, 9)`) is one range test of the column engine, not one pass over the chunk a member, and the AVX2 range kernel compares in registers (`filter_few_rows`, 1,000,000 rows: 24.4 to 15.0 million instructions a query, 2.07 to 1.66 ms).
- Improved: pg_sift: a unit of a `sift_col` index with dead rows (a VACUUM cleared their liveness bits) is answered from its stored counts and integer and numeric sums minus its dead rows (`sift.agg_dead_delta`, default 15, 0 off) when every page it has rows on is all-visible and at most that percent of its rows are dead, where it was read by its rows: after 60 days of churn at 20% a month the grouped sum of 3,000,000 rows takes 84 to 721 of its 721 units from the pre-aggregates and runs 21.6 to 15.0 ms (336 to 160 million instructions), and of 10,000,000 rows 104 to 80 ms (1,214 to 821 million); no stored byte, no write, no format change; a minimum, a maximum, count(DISTINCT), a float sum and the write buffer still scan
- Improved: pg_sift: the overlays that churn leaves in a `sift_col` index are cut by TID range into units of `sift.col_unit_floor` rows (default 2,048) when a promotion reaches 16 superblocks or more with under half a unit's rows in each, and the tier merges of such overlays stay cut that way, so a table that has churned keeps hundreds of units where it had tens of thousands (3,000,000 rows, 20% of the rows changed a month: 10,552 to 721 units, a grouped sum 64.5 to 24.6 ms, a key lookup 35.0 to 7.5 ms; 10,000,000 rows: 35,107 to 2,415 units, 231 to 97 ms, 142 to 40 ms) with a smaller index while dirty, the same size after a merge and less WAL; the newest rows by `ORDER BY .. LIMIT` cost 0.1 to 0.4 ms more; a segment cut by TID range is column format minor 17 and `sift.col_unit_floor = 0` writes the superblock cut as before
- Added: pg_sift: `ORDER BY a, b LIMIT n` where only the first keys are columns of the `sift_col` index (an integer, date or timestamp column, then a column the index does not hold or an expression) is a top-N of the first keys under PostgreSQL's `Incremental Sort` of the whole `ORDER BY`: the scan gives the rows up to the last one that ties with the `n`-th on the first keys, starting again with a heap 16 times larger (twice, then every row) when the statistics did not know a large tie group, and the sort puts the ties in order (up to 8192 rows sharing a value by the statistics; 1,000,000-row table, `ORDER BY event_time, watch_id LIMIT 10`: 108 ms to 0.4 ms; 10,000,000 rows: 940 ms to 0.5 ms, the cost of the one-key top-N plus about 0.03 ms). No format change, no setting; `EXPLAIN ANALYZE` shows `Top-N Candidates` and `Top-N Restarts`. Test `col_topn_trail`, fuzz family `trail`, plan-gate shapes `b07` to `b09`.
- Fixed: pg_sift: an aggregate scan of a key-cut index (`key_cut`) vetted every row of a unit with dead rows through the heap even when every page of its block range was all-visible (a table VACUUMed after its updates): such a unit is now counted from its liveness alone and the scan fetches no heap page (a group-by over a table with 9% of its rows dead in the oldest units fetched all 29,628 pages of the heap).
- Changed: pg_sift: the automatic `key_cut` also cuts by a date or timestamp column when no key qualifies: a range holding the rows of one unit reads at least 8 times the units those rows fill and at least 64 units (a scattered `ts`: a day 9.05 to 0.23 ms and 30 days 11.1 to 0.38 ms at 3,000,000 rows, the rows of one key 12.6 to 28.4 ms), under the same guards and the 3% growth, and never advised (a time leaf is cut by or left alone); the decision ends at the first look of 8 superblocks when a leaf follows the heap order or a time cut would grow the index by more than 7%, and its sample is a quarter of the heap at most; the `NOTICE` and `sift.key_cut_decision` say "a range" for it (`col_key_cut_auto`).
- Improved: pg_sift: sums and averages of a decimal expression run on int64 vectors in blocks of 512 tuples when the data proves it safe (every multiplied operand below 2^31, no value near 2^62), with the vector kernels of each CPU (AVX2, NEON) and the int128 interpreter for the other blocks: `tpch_q1` (1,000,000 rows, scan forced, one worker) 44 to 27 ms and 609 to 347 instructions a row.
- Fixed: pg_sift: the build of a `key_cut` index on a table that was never analyzed aimed its units at 1024 rows, where a superblock held 7,000: 1,400,000 rows made 654 units a segment instead of 93, the pre-aggregates grew 1.1 to 6.9 MB and the index 25.1 to 33.0 MB (+32%, against +6% analyzed). A unit now aims at the rows a superblock holds in the run, and the automatic decision reads them from the middle superblock of the heap (`col_key_cut_auto`).
- Fixed: pg_sift: the size the automatic `key_cut` predicts counts the row directories as the index writes them, from the addresses of the sampled rows, not 0.65 bytes a row: a scattered key's addresses take 1.2 to 1.5, so an index of 10 columns on 700,000 rows that grew 3.3% was predicted at +0.2% and is now predicted at +3.9% (and advised), and 1,400,000 rows +3.1% measured against +0.7% before and +5.6% now (`col_key_cut_auto`).
- Changed: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index cut its units by a key (`key_cut`, column format minor 17) by default when a sample of the heap shows that zones cannot skip the units of the key: the first key-shaped column whose equality reads at least 8 times the units its rows fill and at least 64 units, when no other leaf follows the heap order, the index has no `order_columns` and its predicted size grows by at most 3%; otherwise the `NOTICE` gives the units read, the predicted growth and the command; `sift.key_cut_decision(index)` says what a `REINDEX` would decide, `key_cut = 'off'` switches it, indexes built before keep their units (`col_key_cut_auto`). The decision first reads 8 superblocks and stops there for a table whose keys sit in few units (11 to 16 ms on a 353,029-row table, where the full sample took 225 ms of a 0.5 s build).
- Improved: sift_table: COPY, CREATE TABLE AS and REFRESH MATERIALIZED VIEW into a `sift_table` spend 12% fewer instructions (`bulk_copy_wal` 1,583 to 1,377 ms against 853 ms for a heap table, 1,000,000 rows): a numeric(p <= 18, s) value becomes its integer key from the digits of the numeric (70 instructions instead of 1,400 through the multiply, round and int8 conversions) and a row goes from the slot to the column batch without being formed and deformed as a tuple.
- Changed: pg_sift: the conditions of the outer aggregates of a two-level scan read the value of each cell once: the inner average, sum or count of a cell is asked by the first outer aggregate that needs it, not by every condition and every value (twenty times a cell on a score distribution over customer-and-month cells), and the comparison of a cell's float value with a constant is inline (NaN above every number, equal to itself, as `float8_cmp_internal`). On the statements with two-level cell conditions (eleven outer aggregates, five of them filtered on the cell's average) 5.0% to 7.2% fewer instructions (3.10G to 2.92G for the largest, 2.37G to 2.25G for a monthly one); over a set of 156 statements of a reporting workload -2.7% in all, none more than 2% above. Tests `agg_cell_conditions`.
- Improved: pg_sift: a grouped scan whose group table spills (`work_mem` below the groups) is 33% faster (`count_distinct_spill`, 193,000 groups at `work_mem` 1MB: 423 to 300 ms, the normal plan 306 ms): the groups it returns are no longer kept for a rescan the plan never asks for (and a plan that does ask appends them without reading each back, 170,000 write and 170,000 read system calls less), the first spill counts the groups its table holds, not the room it has, to choose the partitions (8 became 16 here: a partition that did not fit hash memory was spilled a second time), and the key tables of a table after a spill start with the room of the one written.
- Changed: pg_sift: planning a statement on a table with a column index no longer reads the index's statistics for every plan (they are kept by the backend while the index, its table and the catalog are unchanged) and no longer serializes the spec of a path that loses, which most are; plans are unchanged.
- Added: bench: the generic suites of the cases, generated locally and measured against PostgreSQL with the losses in the tables: TPC-H-like (`tpch_q1`, `tpch_q6`, `tpch_q3`, `tpch_q10`, `tpch_q12`: customer, orders and lineitem with the columns and key relations of TPC-H, not its distributions or its results), ClickBench-like (`hits_count`, `hits_filter_count`, `hits_sum_avg`, `hits_distinct_users`, `hits_distinct_phrases`, `hits_region_users`, `hits_top_phrases`, `hits_like_count`, `hits_like_group`, `hits_like_order`, `hits_ilike_count`, `hits_order_time`: a generated wide table, not ClickBench's data), time series (`ts_range_by_day`, `ts_hour_metric`, `ts_top_devices`, `ts_latest_device`) and the write overhead of a `sift_col` index (`oltp_insert_index`, `oltp_update_index`).
- Changed: pg_sift: the inline plan of a shared CTE no longer repeats an aggregating `IN` sub-select of the CTE body in every arm: it is a `MATERIALIZED` CTE of the same `WITH` that the arms read, so the ranking runs once and the plan is smaller to build; the plans of the load-replay statements are unchanged (numbers in docs/COL-PLAN.md).
- Improved: sift_table: UPDATEs of keys that are updated many times in one transaction are 3x faster (`tbl_update_keys` 3,076 to 950 ms against 705 ms for a heap table): the uniqueness check and the scan skip the old versions the transaction deleted without reading their rows, and the fetch scan keeps the TIDs of up to 32 superblocks (8 MB per table at most) instead of 4.
- Changed: pg_sift: a lookup table that a later inner or semi join keys by one of its columns alone (the ids of a request, a CTE or sub-select of them, joined to the table of their rows), or that a CTE of ids joins on the same leaf of the scan, is loaded by a btree probe for each of those keys while the descents and the pages of the rows they find cost less than the pages of the table at the planner's page cost, where the whole table was read for every request (`Lookup Index Scans`, `Lookup Rows Read`); the plan price follows the same rule, and `sift.agg_lookup_pushdown = off` reads the table whole (`agg_lookup_keyed`).
- Changed: pg_sift: a range on a leaf with a sorted run (`order_columns`), or an equality on it, in a unit of 1,024 rows or more is answered from the run by binary search when it holds at most an eighth of the unit, and the chunk is decoded only at those rows, where every row was tested; an aggregate scan tests such a range before its other filters and EXPLAIN ANALYZE shows `Units Filtered By Sorted Run`. On a timestamp with microseconds the time falls 1.1x to 1.5x (1,000,000 to 10,000,000 rows), on whole-second keys only the instructions of the range test fall and the time changes by -12% to +13%. No format change and no option; an index without `order_columns`, a float or text leaf, a chunk with overflowed rows and a range with one bound are tested as before.
- Added: pg_sift: every option, setting and automatic behavior of the options record (`auto(name)` rows for the heuristics that have no setting) states what it does, its measured gain, its impact (index size, build, merge and write cost), when it applies, why its default is what it is and how to turn it off; `docs/OPTIONS.md` lists the NOTICE texts the library raises.
- Changed: pg_sift: a scan whose keys or conditions hold a hashed sub-plan (`x IN (SELECT ..)`) can plan workers: each worker starts the sub-plan in an executor state of the statement and hashes its table once. A sub-plan that holds a function unsafe in a worker and the output of an ARRAY sub-select keep one process.
- Changed: pg_sift: a scan with a spine (the rows of a plan the leader runs, joined to the cells of the scan) and a scan whose join is a correlated sub-select (run for each distinct combination of the leaves it reads) can plan workers: each worker reads the spine from the shared memory and starts the sub-select's plan itself, and the rows the sub-select returned for a set of parameter values are kept in the shared memory, so that the processes of the scan run it once for each set between them, not once each.
- Changed: pg_sift: a grouped scan under an `ORDER BY` or a window on its group keys (also over a sub-select or an inlined CTE that the window reads, whose columns may be constants or `COALESCE(key, constant)`) emits its groups in that order, and no Sort is put above it (EXPLAIN shows `Group Order Keys`): the distinct values of each key are sorted once with the comparator of the pathkey and the groups by their ranks. The rows and their order are those of the normal plan; a grouping set, `DISTINCT ON`, a scan that spills and an `ORDER BY` an aggregate keep the Sort. No setting, no format change. Replay statement with a window over 40,586 groups at 1,000,000 rows: 514 ms against 645 ms with 4 workers and 705 against 800 ms in one process (the Sort above the scan was 117 ms of it).
- Changed: pg_sift: a parallel scan with float sums keeps the sums of each lane a process scanned apart in the table it scans into and sends one table for all its lanes, where it started a table, a blob and a merge for each lane; the leader adds the lanes in order. The float sums of every scan that could have workers before are the same bits as before, with or without workers. The scans that gained workers (a spine, a correlated sub-select) add in the order of the rows when their plan has no workers, as they did, and in lane order when it has, so with workers their last digits can differ from the earlier release and from the plan without (at most 2 (n - 1) 2^-53 of the sum of the absolute values, docs/COL-PLAN.md; 4e-13 relative to the result on the replay statements at 1,000,000 rows); integers, numeric, counts, min, max and distinct are exact in any plan. `max_parallel_workers_per_gather = 0` or the table's `parallel_workers = 0` keeps their earlier sums.
- Changed: pg_sift: the ids of a dictionary's entries (the key of a group by a text column, the values of `count(DISTINCT text)`) are made when a tuple first reads the entry, where the statement made them for every entry of the dictionary at its start (the hash of each string): a request over 82 rows with a 100,000-entry category dictionary ran 9.7 ms warm, against 1.8 ms with the entries made on demand (the control without a text leaf 1.8 ms); the groups of a statement are numbered in the order the scan meets them. Tests `agg_dict_lazy_map`.
- Fixed: pg_sift: a parallel scan with lanes (a float sum or average, added in the order of the scan) that reads a column of a lookup in an expression (`avg(x) FILTER (WHERE d.tag IS NOT NULL)` after a `LEFT JOIN`) could crash the backend with a segmentation fault from the second lane on: the lookup's output rows start again with each lane, and the ids that an expression combo had given to the values of those rows did not, so a row that matched nothing was read as one of the earlier lane's rows. Test `agg_lookup_lanes`.
- Fixed: pg_sift: a `COPY` into a `sift_table` next to other writers that keep its directory at the limit (`sift.table_max_segments`) settles the directory again, up to 8 times, before it fails with `has N segments`: the room one settle made was taken by the writers waiting for it before the load held the lock. It still fails when the directory stays full (rows of running transactions in the way). 3 of 30 runs of 8 writers and 2 `COPY`s of 30,000 rows of 1 kB at a limit of 16 failed; 0 of 60 do.
- Changed: pg_sift: a scan whose lookup joins read a CTE, a sub-select or a function (rows the leader loads and shares with the workers) can plan workers when the functions of the expressions the scan evaluates are parallel safe; the relations of the joins no longer have to be, as PostgreSQL asks of a parallel scan, so a query with a CTE scan in it no longer keeps one process. The workers are priced as for any other scan (`parallel_setup_cost` and the rows of the lookup tables shared), so a small scan stays in one process. A request that reads every row of a 1,000,000-row index through a CTE of 12,000 keys takes 42 ms in the aggregate node with 4 workers where it took 80 ms with one process (best of 3).
- Fixed: pg_sift: a parallel scan with float sums (merged in lane order) and a lookup keyed by a column of an earlier lookup (a second join on a column that the first one outputs) kept, for the second lane that a worker scanned, the ids that the lookup had given to the values of the lane before: the tuples of the lane were counted in the groups of another lane's rows (totals kept, the split between groups wrong). The ids start again with each lane.
- Changed: pg_sift: an `ORDER BY .. LIMIT n` scan whose output columns are all integer, date or timestamp columns of the index takes the rows of the result from the units that gave them, where it fetched each from the heap (`Heap Fetches: 0` in EXPLAIN): a sort key is taken from the entry, the other columns are decoded from the unit when the first key follows the heap order (`pg_stats` correlation of 0.9 or more) and the unit gives at least half of the rows the scan keeps. Rows from the write buffer and from units that give fewer are still fetched. No format change; a column outside the index, a text column or a numeric one keeps the heap fetch.
- Changed: pg_sift: an aggregate scan reads the point directory of a leaf (`point_columns`, `point_tid_columns`) for an `IN` list or an equality only while the keys are expected to have fewer rows than 4,096 or a 64th of the segment's rows (the keys of the leaf have the segment's rows divided by its distinct keys, each): a list of keys with many rows each lists a row in most units and costs more to read than the units cost to scan, so the zones and summaries answer it; the plan price follows the same rule. A scan that hands addresses to the heap is as before (`col_point_dense`).
- Changed: pg_sift: `count(DISTINCT x)` of a `bigint` column is kept as the distinct values of a decimal expression (the path `count(DISTINCT x + 1)` already took), not as the combinations of a generic aggregate: the number is the same, the set is exact, and the aggregate runs in the workers of a parallel scan (the leader merges their sets) where it ran in the leader alone.
- Fixed: pg_sift: the scan of a grouped `count(DISTINCT x)` of an integer column is no longer priced for bitmap chunks its sets do not allocate: a group of few rows (under 16 chunks of the range of the column, and the directory) starts as a hash set, so 100,000 groups of 10 rows over 1,000,000 rows stay on the scan (69 ms; the normal plan picked at a 4GB `work_mem` took 493 ms) as the 30,000 groups before them did (bench case `count_distinct_highcard`: a hundred thousand groups of a few rows).
- Fixed: pg_sift: the heap check of the aggregate scan (the pages of the kept units that are not all-visible) is never priced above the random fetch of the same pages: with `seq_page_cost` above `random_page_cost` (100 and 4, the tuned profile of `scripts/plangate.sh --lane tuned`) the check cost 100 a page, which was most of the price of a scan of a table of 15 million pages, and the scan lost to a normal plan that runs many times slower for 105 of the 237 statements of the corpus planned at that size. Configurations with `seq_page_cost` at most `random_page_cost` are priced as before.
- Fixed: pg_sift: the load of a lookup table that the scan reads whole (no index usable, no plan of its own) is priced with the pages PostgreSQL's own sequential scan of it reads, besides its rows: a join to a table of 1.5 million rows cost the scan about half of what the normal plan pays to read the same table, and the scan was taken for outer sides of up to 200,000 rows where the normal plan ran faster.
- Changed: pg_sift: `count(DISTINCT x)` of a column of a joined table (a lookup join, the nullable side of a join from the left) runs in parallel: a worker sends the pairs (group, value) of its set with the values, the leader interns them and a value two workers saw counts once, `count(x)` of such a column adds up. The start of workers is priced at 3 ms and 0.35 ms each in estimates of 500 units a millisecond (it was 2,750, which charged the median scan of a replay of 156 statements 215 ms and kept the workers off every one of them), a set of a joined column is charged its pairs (at most the distinct values of the argument for each group), so a scan of 20-valued names takes workers from about 800,000 rows (28.6 to 16.7 ms; 94.5 to 38.3 ms at 3,200,000) and one of almost one value a row stays serial at 800,000 rows. Of the replayed statements 60 scans that kept one process for this reason are allowed workers, 4 plan them (the sum of the 156 at two workers 17.20 s against 17.16 s serial), and the rows are the same at 0 and 4 workers.
- Changed: pg_sift: a scan whose query joins a table from the left to what it scans (a calendar `LEFT JOIN` the facts) runs in parallel: the preserved side is shared read-only, each participant marks the rows its tuples matched, a worker sends the marks with its tables and the leader adds the rows nobody matched once every worker has told, and the counts and float sums of the nullable side add up in lane order. Aggregates that still keep the scan in one process: `min`, `max` and numeric sums of the nullable side, arms and a spine.
- Added: pg_sift: `docs/OPTIONS.md` has a table "Upgrading from 3.3": the options that are new, have another default or write a newer column format minor than 3.3 reads, and those whose behavior this release changes, with what a 3.3 user gets after `ALTER EXTENSION pg_sift UPDATE` and whether it needs a `REINDEX`. No default changed since 3.3.
- `col_index_advice`.
- Changed: pg_sift: a lookup join on one column of the scanned table (a text, integer, date, int8 or float8 column) resolves its rows in a loop of the column's kind with the cache of matches probed inline, and reads a row's value once for all of its elements; with the key ids shared by grouping sets, reports with grouping sets over such joins run in 9.5% fewer cycles (10.3% fewer instructions; x0.1 replay statements, one backend).
- Changed: pg_sift: under `GROUPING SETS` whose groups are kept in a hash table (keys of many values), the aggregate scan prepares the key ids of a chunk once for all the sets that read its rows, where it prepared every key again for each set.
- Changed: pg_sift: the plan-choice gate keeps the expected plan and cost band of each shape in the `--lane tuned` and `--table` lanes (`test/plan/bands.tsv`; `scripts/plangate.sh --update` in a lane rewrites its rows), as the planner costs of those lanes differ from the stock lane's by a regime (the tuned profile prices a sequential page at 100) and not by a drift: a plan or ratio that leaves the lane's own record now fails on PostgreSQL 18 (a warning on another major), and the CI tiers run the table lane with the tuned profile too (`plan-gate-table-tuned`).
- Added: pg_sift: `scripts/plangate.sh --lane tuned` runs the plan-choice gate with the planner settings of a tuned profile (`seq_page_cost` 100, `random_page_cost` 4, 8 workers per gather, `work_mem` 4GB): the results must agree and a lose fails, the expected plans and cost bands stay those of the stock lane (a different plan is printed, a band drift is a warning); the CI tiers run it as `plan-gate-tuned`.
- Added: pg_sift: `scripts/plangate.sh --table` runs the 96 plan-choice shapes on the tables as `sift_table` (`test/plan/table.sql`): the results agree with the normal plan on every shape; the plan and cost band are those of the heap with a `sift_col` index (a different plan is printed, a drift is a warning); the CI tiers run it as `plan-gate-table`.
- Changed: pg_sift: an aggregate's argument or `FILTER` over a column of a query joined from the left (a spine `LEFT JOIN` the rows of a table with a `sift_col` index, under grouping sets: PostgreSQL hands the column as a placeholder) is read as the expression it stands for when that is a strict expression of the scanned table's columns, which is NULL where the join finds no row: the cast of a map value is the float leaf, so `avg((value #>> '{}')::float8)` is the leaf aggregate and `FILTER (WHERE value >= 8)` compares the decoded values, where both went through the lookup cache once per element and per aggregate. A constant, a non-strict expression (`coalesce`, `CASE`) or a volatile one keeps the placeholder. Reports of that shape run in 1.5-2.1x fewer cycles (2.4x fewer instructions in the scan, x0.1 replay statements, one backend).
- Changed: pg_sift: a merge of every segment of an index whose `key_columns` is `auto` decides the key lists again from the segments it rewrites: after churn, a key-shaped column that the cube holds in under 40% of the units gets its key list at the merge, as a `REINDEX` would have chosen (the option reads `auto: set_id`; the report after the merge drops what does not fit or what the cube holds in half of the units, so the choice does not alternate); an explicit `key_columns`, a partial merge and `sift.col_format_minor` under 15 are unchanged, and a merge that cannot take the table's maintenance lock at once (a VACUUM holds it) leaves the choice to the next merge instead of waiting for it with the index metapage held, which deadlocked with the VACUUM (`col_auto_merge_concurrent`).
- Added: pg_sift: `CREATE INDEX` on a `sift_col` index names, in one NOTICE, the covered columns that look rewritten often (a name ending in `updated_at`, `modified_at`, `_count`, `_version`, or `likes`, `views`, `version`): an `UPDATE` of them is not a HOT update (0% HOT updates against 28% on the 13-column index of the web API, measured). The column stays covered.
- The figures for `compression`, `table_compression`, `key_columns` and `table_dict_lookup` that had no source are replaced by measured ones, and the GPU section of `docs/GPU.md` reads the stored table (the device ties the CPU).
- Changed: pg_sift: the refusal `lateral_limit` of a `LATERAL (.. LIMIT 1)` child names the columns of its key and ORDER BY that no unique index holds and says what makes the key unique: end the ORDER BY with the primary key, or create a unique index on them (a partial one, `WHERE col IS NOT NULL`, for a sub-select with that condition; an index on an expression does not count), and that `CLUSTER` on it and `VACUUM` keep the lookups on adjacent pages.
- Changed: pg_sift: a lookup of a key or an IN list of the point directory of heap addresses (`point_tid_columns`) on a query the index covers is planned on the addresses and the heap, as it is on a query it does not cover, instead of through the index-only scan that reads every kept unit; and a key with a range on another column hands the heap only the rows of the range when a unit lists many of the key's addresses (the unit is read and filtered), where it handed all of them to be rechecked away.
- Added: pg_sift: `CREATE INDEX` on a `sift_col` index says in a NOTICE that the primary key is not a column of the index when a foreign key of another table references it, since a join of two sift tables on a key that is not a column of both indexes keeps the normal plan (`join_condition`). The key is not added by default: it grows an index of the web API by 19.6% to 131% (about 3 bytes a row for ids with gaps; nothing for consecutive ids), more than the 3% a default may cost.
- Changed: pg_sift: `EXISTS (SELECT .. FROM f WHERE f.k = x.k AND ..)` and `NOT EXISTS` where PostgreSQL leaves a sub-plan for each outer row (the select list, under an `OR`, in a `CASE`) are `f` grouped by `k` left joined to `x`, EXISTS being a group that is there: kept when the planner costs it below the lookups of the plain plan (`found IS NOT NULL`: never NULL, a NULL key has no group, an empty table none, a repeated key answers each time). An EXISTS the WHERE ANDs stays the semi join (a NOT EXISTS the anti join) PostgreSQL makes of it; `IN`, `NOT IN` (their NULLs differ), a condition other than equalities of the table's column and an outer one, a `LIMIT`, a join, a sub-select inside and an outer query that groups leave it as written.
- Changed: pg_sift: `sift.index_advice` names `preagg = 'auto'` when more dimensions are asked for than the cube's 16, and `rollup_dims = 'auto'` with the advice of a pair (`col_index_advice`).
- Changed: pg_sift: the page mask and the run head of a unit that a top-N scan reads for the head bypass are kept by the backend for the next query (in the cache of `dict_cache_kb`), so the superblocks of an `ORDER BY .. LIMIT` read them from memory.
- Added: pg_sift: `docs/OPTIONS.md` records the case of every option and setting: its class (default, opt-in, diagnostic, safety, deprecated), for an opt-in the limit of the default rule it failed, the measured gain, index growth and write cost with their conditions, the pull request, the benchmark case and the advice line. README.md and the tables of `docs/USAGE.md` gain a class column.
- Changed: pg_sift: an `ORDER BY .. LIMIT` over a `sift_col` index probes the zones of six superblocks from the segment headers before the walk, and no longer reads a unit of each.
- Changed: pg_sift: the largest value of a float key of each row averaged by group is one two-level scan: `max` and `min` of a `float8` column or of a float8 key of a jsonb document are inner aggregates of the arms, so `SELECT .., avg(x) FROM (SELECT id, .., max((doc ->> 'k')::float8) AS x .. GROUP BY id, ..) GROUP BY ..` (the per-row scores of a table, averaged by group) no longer returns a row per inner group to a PostgreSQL aggregate. A cell with no value gives NULL, which the outer avg and sum skip; a NaN is the largest value. The answers are those of the normal plan (differential regress `agg_two_level_inner_minmax`: NULL cells, a NaN and infinities, a WHERE and a HAVING on the inner aggregate, two arms). No format change.
- Changed: pg_sift: the `sum` and `avg` of a decimal expression of up to 38 digits (a price times one minus a discount times one plus a tax, `tpch_q1`) are computed by the scan: the sum is kept in 128 bits and the number of times it wrapped, so it is exact for any number of rows, where an argument of more than 26 digits was left to PostgreSQL's own numeric arithmetic row by row (the pre-aggregates still take only the 26-digit programs); the one aggregate was 92% of the time of the query, which now takes 19 ms where it took 200 ms at 400,000 rows.
- Changed: pg_sift: a `numeric` column without a precision (or past 38 digits) is read through a dictionary of the unit's values whose entries are built from the digits of the stored image, not from its text through `numeric_in` (the numerics are the same bit for bit), and a `sum` or `avg` of such a column adds the entries as an integer and a scale decoded once for each entry of a dictionary, where the combo made a datum of every entry and `numeric_add` added it: 1,000,000 rows of 100,000 distinct values in 6 groups take 68 ms where PostgreSQL's plan takes 155 ms (the scan took 530 ms before, a 3x loss that the planner kept out of). The sum stays on one process, so the planner keeps a parallel plan of PostgreSQL where that is cheaper (a table of 1,000,000 rows read by 2 workers: 61 ms against 93). The sum of a `numeric(30,16)` column of 100,000 rows takes 7.3 ms against 16.8 ms. Test `agg_wide_sum_dict`.
- Changed: pg_sift: the groups that cannot be among the first n are dropped before they become tuples: an `ORDER BY <aggregate> [DESC] LIMIT n [OFFSET m]` over a plain `GROUP BY` (the first sort key a `count`, `sum` of an integer, decimal or float8, or `avg` of a float8 column; no HAVING, window, DISTINCT or grouping sets) ranks the groups by the raw value of the aggregate and emits only those that tie with or beat the (n + m)-th, where PostgreSQL sorted every group; the sort and the limit above stay in the plan and rank the ties by the other keys (EXPLAIN shows `Groups Dropped By Top-N`). `GROUP BY id ORDER BY sum(amount * quantity) DESC LIMIT 10` over 400,000 groups takes 14.6 ms against 96.6 ms (over 1,000,000 orders 61 ms against 174 ms, normal plan 504 ms), the top 10 customers by `sum` 15.1 ms against 32.2 ms at 77,000 groups, by `count` 14.4 ms against 20.0 ms; at 100,000 groups 4.4 ms against 23.2 ms. The rows are those of the normal plan, ties at the n-th aggregate value included (differential regress `agg_group_topn`: ASC and DESC, NULLS FIRST and LAST, an OFFSET, NaN and infinities, NULL sums, a second sort key, ties, parallel workers). No format change.
- Changed: pg_sift: the groups that cannot be among the first n are dropped before they become tuples: an `ORDER BY <aggregate> [DESC] LIMIT n [OFFSET m]` over a plain `GROUP BY` (the first sort key a `count`, `sum` of an integer, decimal or float8, or `avg` of a float8 column; no HAVING, window, DISTINCT or grouping sets) ranks the groups by the raw value of the aggregate and emits only those that tie with or beat the (n + m)-th, where PostgreSQL sorted every group; the sort and the limit above stay in the plan and rank the ties by the other keys (EXPLAIN shows `Groups Dropped By Top-N`). `GROUP BY id ORDER BY sum(amount * quantity) DESC LIMIT 10` over 400,000 groups takes 14.6 ms against 96.6 ms (over 1,000,000 orders 61 ms against 174 ms, normal plan 504 ms), the top 10 customers by `sum` 15.1 ms against 32.2 ms at 77,000 groups, by `count` 14.4 ms against 20.0 ms; at 100,000 groups 4.4 ms against 23.2 ms; when no group ranks after the n-th (400,000 groups of 2 or 3 rows ordered by `count`) the scan finds that in the first pass and keeps all: 47.3 ms against 46.9 ms with no drop at all (48.5 ms when a second pass keeps them). The rows are those of the normal plan, ties at the n-th aggregate value included (differential regress `agg_group_topn`: ASC and DESC, NULLS FIRST and LAST, an OFFSET, NaN and infinities, NULL sums, a second sort key, ties, parallel workers). No format change.
- Changed: pg_sift: the aggregate scan finds `mode() WITHIN GROUP (ORDER BY x)` on the histogram of the values of a group, as it does `percentile_disc` (a float, integer, date, timestamp or decimal of up to 18 digits): the value of most rows, the first in the order of the sort on a tie (the smallest, the largest for DESC), where it went through the generic aggregate. Over 400,000 rows in 40 groups the mode of a decimal takes 26 ms against 175 ms and that of a smallint 7 ms against 117 ms, and both together over 1,000,000 rows 84 ms against 468 ms (the normal plan: 199 and 170 ms); the answers are the final function's (differential regress `agg_ordered_set`: ties, NULLs, NaN, DESC, a histogram that spills, parallel workers, shared with percentiles of the same argument). The mode of a text column keeps the generic path. No format change.
- Changed: pg_sift: the aggregate scan of `percentile_disc` finds the percentile_disc of a decimal, timestamp or bigint on the 64-bit values of its histogram: a `numeric` of up to 18 digits (a column or an expression of columns), a `timestamp`, a `timestamptz` or a `bigint` column keeps its values as integers (a double holds 53 bits), picks the rank by the same quickselect and returns the value in its own type (a decimal with its scale, a NaN last), where it went through the generic aggregate. Over 400,000 rows in 40 groups the median of a decimal takes 70 ms against 168 ms and that of a timestamp 66 ms against 126 ms (the normal plan: 332 and 273 ms); at 1,000,000 rows, a decimal and a timestamp median together take 205 ms against 610 ms. The answers are the final functions' (differential regress `agg_ordered_set`: NULLs, `infinity`, the extremes of `bigint`, NaN, DESC, several fractions on one argument, a histogram that spills, parallel workers). No format change.
- Changed: pg_sift: the aggregate scan of `percentile_cont` and `percentile_disc` (an `ORDER BY` of float8, float4, int2, int4 or date values) finds the percentiles of a group by a quickselect over its histogram: the positions they pick are found together, where the histogram was sorted whole (aggregates on one argument share the passes). With 1M values in 40 groups the median takes 76 ms against 193 ms and two percentiles 88 ms against 194 ms (the statement runs 0.3 G instructions against 1.0 G; the normal plan takes 592 ms). The answers are the final functions' (differential regress `agg_ordered_set`, 28 random fractions on 1M rows); a histogram that spills still goes through the tuplesort. No format change.
- Fixed: pg_sift: the rows of the scan of a CTE are lifted toward the rows its body reads from a table with a column index only through nodes that keep their rows: a body that aggregates, groups, limits or combines sets (`SELECT kind_id, count(*) .. GROUP BY kind_id`, 20 groups) was lifted toward the rows of the table below it (20 groups to 94 rows with 20,000 read), and the plans above it were priced for them. Its rows are now its own estimate. The CTEs of the 156 and 237 statements of the replay corpora are lifted as before (no line of the lift changes).
- Changed: pg_sift: `x JOIN LATERAL (SELECT g, sum(f.a) FROM f WHERE f.k = x.k GROUP BY g [HAVING ..]) ON true` (also `LEFT JOIN`, and the comma form) is decorrelated as well: `f` grouped by `k` and `g`, joined to `x` by `k`, kept when the planner costs it below the lookups of the plain plan. A row for each group stays what it was: an inner join gives none for a `k` without rows, a left join a row of NULLs (a count included), and the columns of the select list are evaluated in the grouped query, not above the join. A `HAVING` without `GROUP BY`, grouping sets, an `ORDER BY`, a `LIMIT` and an outer column in a group or the `HAVING` leave the sub-select as written.
- Changed: pg_sift: `sift.suggest_paths` (and the `paths` that `CREATE INDEX` takes from it) takes a wildcard for a map of more than 4 keys of one type (it was 32; a map of ten keys read with `jsonb_each` was not served by ten named keys), leaves out a text key whose values average over 64 bytes, reads a document that is an array as a group of the column's own path and leaves out arrays inside the elements of a group (the index of a group with a list of its own was refused): the `jsonb` columns of the web API, 8,830,976 bytes by hand, 12,058,624 with the named keys and 28 SiftAgg nodes of 237, now 9,314,304 (+5.5%) and 138 nodes; the 40 statements that read a key no document holds stay on the heap.
- Changed: pg_sift: `FROM x, LATERAL (SELECT sum(f.a) FROM f WHERE f.k = x.k)` (the comma form: an inner join of the sub-select to the item of the list that holds the columns it reads) is decorrelated like `x JOIN LATERAL (..) ON true`: the table grouped by `k` and joined to `x`, kept when the planner costs it below the lookups of the plain plan. A sub-select that reads the columns of another lateral sub-select is left as written (the rewrite of one replaces the columns the other reads).
- Changed: pg_sift: a table with several `sift_col` indexes is planned through the cheapest index that gives a plan, not the newest one: a lookup of a key reads the index cut by that key (`key_cut`) and a range of dates the index cut by block range, whatever the creation order; the cost of a scan that answers from the pre-aggregates follows the units the zones keep, as the cost of the other scans does.
- Added: pg_sift: `sift.approx_count_distinct(x)`, an aggregate that returns a HyperLogLog estimate of the distinct values of `x` (16,384 registers, relative standard error 0.81%; `NULL`s are not counted, no row gives 0). It is the same number in every plan, with workers and by partition, and nothing turns `count(DISTINCT)` into it: that stays exact whatever the settings and options say. The `sift_col` reloption `hll_columns` (default none; `int2`, `int4`, `int8`, `date`, `timestamp`, `timestamptz`, `text`, `varchar` and `uuid` columns) stores the sketch of each unit written afterwards, and the aggregate scan merges the sketches of the units it reads whole and hashes the rows of the others; an index without the option writes the bytes it wrote before. The sketches of two columns over 1,000,000 orders answer in 6.4 ms where the normal plan takes 109 ms. `EXPLAIN` shows `Approximate Distinct` and `Units Without The Sketch`, `sift.explain_why` names `approx_no_sketch`, `sift.upgrade_commands()` and `sift.upgrade_check()` know the option, and `scripts/check-approx.sh` is the lint that keeps the estimate opt-in. Usage in [docs/USAGE.md](docs/USAGE.md) "Approximate distinct counts".
- Changed: pg_sift: before a page of a secondary index splits after `UPDATE`s that keep the index keys, the b-tree asks the table which of the page's versions are dead (bottom-up deletion; a simple deletion asks about the entries marked dead), and a `sift_table` answered by reading every row of the page (several hundred, in key order, each from another superblock of the column groups, which the fetch scan reloaded for nearly every one). It now answers from its tombstone log alone (a TID is never reused, so the log of its key is the row's whole history), reads a row only to learn that a simple deletion's row is gone, in TID order, and stops a bottom-up pass once the entries found free the space the index asked for: 40,000 updates by key of 2,000 rows of a 600,000-row table with 100,000 rows in the write buffer went from 9.0 s to 3.1 s (a heap: 0.7 s), 2,000 updates with 22,000 buffered rows from 3.5 s to 0.24 s, and the cost of an update no longer grows with the rows the write buffer holds (flat from 1,000 to 100,000).
- Changed: pg_sift: the extension SQL is version 3.4 (`pg_sift--3.4.sql`, `pg_sift--3.3--3.4.sql`, `default_version = '3.4'`): run `ALTER EXTENSION pg_sift UPDATE` in every database. The scripts of 3.3 are the released ones again (an install at 3.3.0 never ran the change that an edit of them made); `sift.upgrade_commands()` of 3.4 also lists an index whose `key_cut` came after its units.
- Changed: pg_sift: a column segment whose units are cut by a declared key (`key_cut`) carries a flag in its directory entry, and the readers and the merge tell it by the flag, not by the column format minor being 17: a later minor for another record no longer makes a segment keyed or not. Segments already written as minor 17 are read as keyed.
- Changed: pg_sift: a unit of a `sift_col` index in which one integer leaf is NULL in at least 99% of the rows (a soft-delete stamp) also keeps the pre-aggregates of the rows where it is not NULL (column format minor 19, no option; a merge or `REINDEX` writes them, +1.4% of the index on a table with 0.2% stamped rows), and `WHERE stamp IS NULL` is answered from the pre-aggregates as the unit's aggregate minus those rows (counts, integer, numeric and standard-deviation measures exactly, a float sum guarded against cancellation, `min` and `max` scanned): a count and an average by a key over 353,029 rows 6.1 to 2.5 ms, 171 of 171 units from the pre-aggregates instead of 5 (`Units Net Of Exceptions` in `EXPLAIN ANALYZE`; docs/COL-PREAGG.md section 11).
- Changed: pg_sift: the sorted runs of an index with `order_columns` also keep a head (column format minor 18): the first and last `n / 128` keys, at most 32, of a leaf with no NULL, each with the heap address of its row, and an `ORDER BY .. LIMIT n` of at most that many rows with one sort key, no condition and no join reads no unit that is clean and all-visible: the heap gets the addresses straight from the head (`Units Answered By Run Head` in EXPLAIN). `ORDER BY rnd LIMIT 10` over 800,000 rows takes 0.85 million instructions a statement where it took 1.20 (the scan of the units 0.25 million where it was 0.65), the index grows 0.60% (three order columns) or 2.4% (a narrow index of 24 bytes a row) and a build, a promotion and a merge cost the same. Segments of minor 13 and 18 live in one index; a key-cut segment has no head. A 3.3 library cannot read minor 18: `sift.col_format_minor` 14 or less writes the runs without heads.
- Added: pg_sift: `sift.cluster_advice()` names `key_cut` (and `key_columns` for a key of over 256 values) when the key a workload looks rows up by reads many units and no other leaf is ordered by the table: the key is the first leaf of `key =>` or the leaf of `point_columns`, `point_tid_columns` or `key_columns` (not unique) that would save the most units, and with no such leaf the reason says how to choose one; a table whose id or date follows the arrival order still answers `competing leaves`, an ordered one `already ordered`. `CREATE INDEX` and `REINDEX` print the same advice as a `NOTICE` (silenced by `sift.cluster_advice = off` and the index option `cluster_advice`). No format change; the test is `col_key_cut_advice`.
- Changed: pg_sift: planning an `ORDER BY .. LIMIT` over a table with a `sift_col` index no longer scans the operator class catalog for each sort key: the type cache gives the default btree family, 58,000 fewer instructions of 406,000 per statement (a table of 200,000 rows, callgrind, one sort key), and the same plan.
- Changed: pg_sift: the aggregate scan of `Custom Scan (SiftAgg)` asks the point directory (`point_columns`, `point_tid_columns`) for the rows of a short `IN` list, an equality, a short range, and the keys of an inner or semi lookup join, and reads only the units that hold one of them, each as the rows the directory lists (`Units From Point Directory` and `Rows From Point Directory` of `EXPLAIN ANALYZE`); a 51-key request over 16,384 units read every unit (23.7 ms, 28.6 ms with the join) and now reads 51 (3.6 ms, 2.7 ms; log-log slope 0.80 and 0.74 before, 0.20 and 0.08 after); the same request on a million rows with no b-tree on the key takes 2.1 ms against 68.7 ms for the normal plan. The planner prices the units the directory leaves; the directory is unchanged, so no format change and no index growth or write cost.
- Changed: pg_sift: the access-method scan of a `sift_col` index with `point_tid_columns` merges the address lists of its segments by runs (they are ascending) and sorts nothing when the segments do not interleave, does not sort a list that one key gave, and finds the unit of an address once per superblock: `SELECT * WHERE grp = 7 LIMIT 10` over 800,000 rows (8,092 addresses of the key, four segments) takes 7.29 million instructions instead of 2.70 million, the forced index scan 0.53-0.63 ms instead of 0.14-0.25 ms (the planner still prefers the heap scan of a LIMIT over a key that is spread evenly, as the b-tree row of the same query costs 1.0 million).
- Changed: pg_sift: a `LIKE` or `^@` with a literal prefix on a column that has a point directory (`point_columns`, `point_tid_columns`) is priced by the planner as the lookups of the keys the dictionary names, and the scan turns the dictionary's table of codes into those keys by words: `count(*) WHERE ref LIKE '<4 characters>%'` over 1,000,000 rows of a 64-character key is the access method's lookup (measured in `bench/cases.tsv`, point_prefix_count) where an aggregate scan read every unit the zones kept.
- Changed: pg_sift: the access-method scan of a `sift_col` index tests the zones and summaries of the other predicates before it asks the point directory (`point_tid_columns`), so a key that is in every unit (a group, a tenant) with a range on another column (a date) hands the heap the rows of the range, not every row of the key: `grp = k AND ts BETWEEN ..` over 800,000 rows reads 806 heap pages instead of 7,456 and takes 0.45 ms instead of 3.1 ms (the b-tree on `(grp, ts)`: 0.43 ms).
- Fixed: pg_sift: an index with both `key_cut` and `point_tid_columns` wrote, for its key-cut segments, point directories of heap addresses, which name no unit of such a segment: an index-only scan of a key equality or `IN` list returned no rows, a lookup of a row in a late unit failed with "segment is corrupted", and `sift.col_verify` reported the segment. A key-cut segment now lists the rows of the segment (a lookup reads the unit of each row, as with `point_columns`), `CREATE INDEX` says so, and a key-cut segment with a table of addresses is refused with the `REINDEX` hint.
- Changed: pg_sift: `count(DISTINCT x)` of the primary key over the elements of a list under a grouping set and a spine joined from the left (`periods LEFT JOIN parsed ON ... GROUP BY GROUPING SETS (...)`, the shape of a report whose buckets are filtered distinct counts) counts the rows of each group with a marker, as the same aggregate did without the grouping set, where it kept a set of ids per group and per FILTER: PostgreSQL hands the column as a placeholder there, which the proof of its uniqueness now sees through. A report of two periods, four grouping sets and five distinct counts over 353,000 rows with a map of two keys each: 620 -> 260 ms (one backend, work_mem 4GB); the distinct rows of a grouping set over a preserved join are the bench case `distinct_rows_sets` (200,000 rows). A `count(DISTINCT x)` that keeps the set says why in `sift.explain_why` (`distinct_set`).
- Changed: pg_sift: the choice between a shared CTE and its inline plan (several planner runs of the statement) is kept in the backend, under the statement without its constants, the size class of each table it reads and the settings that move a plan, and a statement seen again is planned as the chosen variant only: up to 4,096 statements, under 1 MB, dropped when a table it reads is invalidated (DDL, an index, `ANALYZE`, `VACUUM`) and checked by a plan in full at every eighth hit. A stale verdict is the plan of the other variant, never other rows. On 156 statements of a replay planned in one session, then again with three other key values and a shorter date window, the plans of all 624 statements are the same as without the cache and 248 of the 324 choices came from it; `sift.explain_why` shows a `verdict_cached` row and `sift.stats()` counts `verdict_cache_hits`, `verdict_cache_misses` and `verdict_cache_flips`.
- Changed: pg_sift: the shared plan of a CTE that several aggregates read is made with `enable_memoize` off (restored after, also when planning fails), and the choice between it and the scans compares the scans with that plan. PostgreSQL prices a Memoize node over a lateral lookup of the CTE's rows on the cache hits it expects, and for a key that is almost unique per row (a jsonb document) there are none, so the shared plan cost less than it did and kept the CTE where the scans were faster. On a corpus of 258 statements 22 take the scans now, and each of the 18 timed runs faster with them; on a replay of 156 statements 3 change and the plans of the others are the same. A plan the shared CTE keeps has no Memoize node over its lookups.
- Changed: pg_sift: a CTE that several aggregates read, built from a table with a `sift_col` index joined to filtered tables (a dimension whose few rows own all of the table's, a list of ranges by containment), was estimated at a row or two, and the shared plan priced from it cost the same at ten times the rows as at one: the choice between the shared CTE and the scans kept the CTE (`cte_cost` in `sift.explain_why`). The rows of the CTE scan in the shared plan are now the geometric mean of that estimate and the largest result its body builds from the table, never fewer than the planner's own, and the extra charge for the aggregates that unnest the CTE stays what it was (it is computed on the planner's rows). The inline plan is not touched. A floor, not an estimate: a skewed join still needs its filter evaluated. `DEBUG1` shows the rows before and after.
- Changed: pg_sift: in a statement that reads a table with a `sift_col` index, a sub-select or CTE scan whose filters have no statistics (a comparison or `IS NULL` on a column of a window, an aggregate or a grouping sets output) expects the geometric mean of the planner's default guess and the rows it filters, not the guess alone, never fewer rows than the planner's own. Two such filters in a chain multiplied to one row, and every join above chose a nested loop that ran an aggregate once per row that came; the plan hashes now. A statement over tables without a column index keeps its plan. `DEBUG1` shows the rows before and after. In a replay of 156 statements at 4 GB and 4 MB of `work_mem`, 36 plans changed from a nested loop to a hash join with equal rows, the statements that use the scans stayed the same, and none was persistently slower.
- Changed: pg_sift: a top-N over a table with `order_columns` (`ORDER BY x LIMIT n`, the sorted run of each unit) reads the sort key of a unit's rows only for the rows it keeps, not for every row of the unit's window, and sizes its candidate buffers by the unit instead of the table's largest. 40 scans of `ORDER BY customer_id LIMIT 10..49` in one statement over 1,000,000 orders go from 48 ms to 24.8 ms; `ORDER BY rnd LIMIT 10` over 800,000 rows from 113 to 82 microseconds a scan, `LIMIT 1000` from 1.23 to 1.05 ms. The segment format does not change.
- Changed: pg_sift: a statement with a lateral sub-select the planner flattens (`LEFT JOIN LATERAL (aggregates) ON true`) plans the flat query first and, when it reads every table that has a column index in a scan, stops there: the query as written is not planned. Planning one such statement took 48 ms, now 25 ms (the same statement written with a grouping, 24 ms); the plans are the same on 657 statements at `random_page_cost` 4 and 1.1 and on 39 statements on two restored dumps.
- Fixed: pg_sift: a dictionary that a backend tests often (more than 32 equalities) is parsed and kept, but one the cache cannot hold (over half of `sift.dict_cache_kb`, 32 MB: 800,000 uuids, a million 64-character keys) was parsed again for every later test, 10 to 13 ms each. A dictionary the cache cannot hold is no longer parsed again for every test: it is searched in the stream, as before the 33rd test. A lookup of a uuid key over 800,000 rows stays at 0.036 ms (it was 13.9 ms from the 33rd), and 40 lookups of 64-character keys in one statement over 1,000,000 orders go from 782 ms to 3.3 ms.
- Changed: pg_sift: an `IN` list of up to 65,536 keys is looked up in the point directory (`point_columns`, `point_tid_columns`) by the access-method scan, where a list of more than 64 keys was answered by reading the units (the lookup cost 36 microseconds a key past the 64th and 1.2 to 2 below it): a list of more than 64 keys is still left to the units when it names a sixteenth of the rows or more. `SELECT id, amount, title FROM orders WHERE id IN (<100 ids>)` over 1,000,000 orders goes from 5.4 ms to 0.25 ms and a list of 1,000 ids from 10.6 ms to 1.75 ms. The segment format does not change.
- Changed: pg_sift: `x IS NULL` and `x IS NOT NULL` are tests of the NULL count of a unit's zone (a scalar column of the rows, in the aggregate scan of a `sift_col` index): a unit with no NULL (or only NULLs) is not read, and a unit whose rows all pass is answered from its pre-aggregates, as for a comparison its zone lies inside. `count(*), sum(amount) WHERE deleted_at IS NULL GROUP BY country` over 1,000,000 orders with 1% of them soft-deleted in one batch goes from 6.4 ms to 0.58 ms, the list of the deleted rows from 1.6 ms to 0.34 ms. The segment format does not change.
- Changed: docs: [docs/USAGE.md](docs/USAGE.md) says when to set `sift.agg_force` for a role (the cost model keeps the normal plan of a shared CTE that PostgreSQL prices for a row it expects and the scan would run better) and when not to, and `enable_nestloop = off` for a nested loop that reruns an aggregate once for each real row of an outer side estimated at one.
- Changed: pg_sift: the `DEBUG1` line `sift: shared CTE plain cost ...` of the choice between a shared CTE and its inline plan also gives the rows the scans read, the aggregates of the shared plan that unnest the CTE and what they were charged.
- Fixed: pg_sift: an `EXPLAIN` or `EXPLAIN ANALYZE` that was the first statement of a backend that had not preloaded the library showed (and `EXPLAIN ANALYZE` ran) the plan made before the library loaded, without the inline CTEs and flattened laterals. The executor start now plans the query of an `EXPLAIN` or a `CREATE TABLE AS` again through the hook, as it does for a plain `SELECT` ([docs/USAGE.md](docs/USAGE.md) section 1); a cursor and a query inside a function keep the plan they were given.
- Fixed: pg_sift: the charge of the plan that keeps a shared CTE for the rows its aggregates unnest took the rows the scans read from PostgreSQL's estimate of the table's filter, which for a filter on a jsonb path or an expression (no statistics) is a few rows of a table that keeps thousands: the rows are now the geometric mean of that estimate and the table's rows, as the group estimates have them. A statement whose scans won at the larger sizes of `sift.agg_scale_horizon` and lost the comparison at its own now takes them.
- Fixed: pg_sift: the plan that keeps a shared CTE was priced for the rows PostgreSQL guesses for what its aggregates unnest (a row for a CTE, 100 for a function) where the scans of the inline plan read tens of thousands, so the CTE was kept while inlining it let the scans win: when the scans replace most of the aggregates that unnest the CTE, those aggregates of the shared plan are charged the square root of the excess rows, in the shared work and in the total the plans are compared by. An aggregate that does not unnest, and a plan whose scans replace half of them or fewer, keep their estimate.
- Changed: pg_sift: `x LEFT JOIN LATERAL (SELECT avg(e) FROM f(x.doc) WHERE ..) s ON true` (an aggregate with no GROUP BY over a lateral function of the row, one row for each row of `x`) is read by the scan as the grouping of the elements by the key of `x`, as `GROUP BY x.id` over the lateral function would be, where `x` holds each of its keys once (a table, or a sub-select, CTE or DISTINCT whose tables are joined by their keys) and the aggregates (`avg`, `sum`, `min`, `max`, `count` of an expression) have strict arguments, so that the row an outer join makes up gives what an aggregate of no rows gives. The lateral pull-up looks inside sublinks too. Anything else stays as written.
- Changed: pg_sift: a statement whose first plan still reads a table that has a column index in an ordinary plan takes its lateral-flattened plan when that plan reads every such table in a scan, whatever the plan above the scans is estimated to cost: that part is priced from the rows it takes for one, and the flattened plan was dropped for a higher total when the scans ran the work.
- Changed: pg_sift: the arms of a two-level aggregate that read the columns of a lateral `LIMIT 1` lookup (`rm.derived_cat AS category`, no `CASE` around it) are one scan when some test the lookup for NOT NULL and a third does not, also with a sub-select in front of the lookup that has a WHERE of its own (`WHERE score IS NOT NULL`): PostgreSQL leaves that condition at the join of the arms where the outer join became inner and in the WHERE of the third, and the arms were refused as `two_arms` and ran as rows. The conditions of the inner joins are compared as one WHERE.
- Added: pg_sift: `key_cut`: the units of a build, a promotion, a merge and a `REINDEX` are cut by the range of the key, whatever the heap pages of their rows, and list their heap addresses, so a lookup or a range of the key reads the units of its range only (one key value of 600: 1 of 100 units, against 8 of 8 by superblock) and keeps doing so after churn and a repack. The units hold up to 32,768 rows and have the summaries of groups of units (superunits) and the cubes of the page cut, so that a scan that does not use the key reads about the units of the page cut (1,200,000 rows of four integer columns: 37 units against 26; whole-table `sum` 0.09 ms against 0.09; `GROUP BY` of a cube column 0.17 ms against 0.45) and a key value reads 1 unit (0.11 ms against 0.64). The index is 17% smaller to 16% larger for narrow rows and 0.2% larger for wide ones, and the build takes 3.4 to 4.1 times the CPU, a promotion and a merge 3.5 times. The memory of a build and of a merge is bounded by `maintenance_work_mem`, not by the number of rows: a build writes a sorted segment each time the rows it holds pass half of it, and a merge is a k-way merge that holds one unit of each input and a window of the output.  3.2, 3.1 and 3.0 refuse minor 17 (`sift.upgrade_check` lists it; `ALTER INDEX x RESET (key_cut)` and a `REINDEX` roll it back, [docs/UPGRADING.md](docs/UPGRADING.md) section 4); the default of `sift.col_format_minor` stays 15.
- Changed: pg_sift: the plan of a statement with its laterals pulled up, when its shared and inline CTE plans cost the same as those of the first plan, is decided as the first was and is not planned again at the larger sizes (`sift.agg_scale_horizon`): the pulled-up form was the one PostgreSQL plans for the first anyway.
- Changed: pg_sift: the `cte_inline` row of `sift.explain_why` carries `scan_cost` and `normal_cost` (the totals of the plan with the shared CTEs inlined and of the plan that keeps them), for a choice either way; a `cte_cost` refusal could not be ranked by its costs before, only by the text of its reason.
- Added: pg_sift: `sift.agg_force` (off; a diagnostic like `sift.plan_scale`): takes the aggregate scan wherever the planner can build it, whatever the plans cost, so the plan the cost model refused (`cost`, `cte_cost`) can be timed: the `enable_*` settings do not change the choice of a shared CTE.
- Fixed: pg_sift: the normal plan of an aggregate over the elements of a jsonb array was priced for the rows PostgreSQL guesses for a function (100 per call) and a join on a range, a few rows where the table's own filters keep tens of thousands, so the scan lost the comparison it wins: the paths that are not the scan are charged the square root of the excess.
- Fixed: pg_sift: the private memory of a `sift_col` build with many small units (a segment of thousands of units and a cube over several columns) is bounded by the segment, not by the units it holds: the merge of each group of units into its superunit summary kept what it allocated until the segment was finished.
- Changed: pg_sift: an `ORDER BY .. LIMIT` over a sort column that does not follow the table (a top-N whose walk visits the units by the best value of their zones) keeps the order of that walk in the backend, for the same index file and directory version and the same sort keys, up to four orders of 16 MB each: the zones of every superblock are read and the superblocks sorted once, not at every query, and the superblocks that hold rows of a write buffer are visited first, as before. The rows, and the units read and skipped, are those of the walk without the kept order.
- Changed: pg_sift: a statement with laterals the scan can read is planned a second time with them pulled up only when its first plan still reads a table that has a column index outside a SiftAgg node; where every such table is read by the scan already, the second plan could not run more in it, and it was planned (and its inline CTE variants with it) for nothing.
- Fixed: pg_sift: an average or sum over a `numeric` column of a precision past 18 digits (`numeric(30, 16)`) whose units hold many distinct values was planned as the scan, which is slower than the normal plan there (each unit builds an entry for each new value); the planner now prices an entry as measured and keeps the normal plan for such a column, and the scan for the column of few values. The answers are the same in both.
- Tests: `upgrade_settings` (the newest 3.2 release writes a database with `ALTER DATABASE` and `ALTER ROLE ... SET sift.*` stored, one of them a setting the new library does not know): after the update the session starts with a warning ("removing it"), the stored settings apply, `sift.col_auto_merge` is on and `sift.col_auto_merge_wal_pct` 5 where nothing set them, the update prints nothing about the auto merge, `CREATE INDEX` prints its NOTICE once, and `pg_dump -C` brings the settings and verifying indexes back; `pg_upgrade` carries the stored settings of a database and of a role in it.
- Tests: `pg_upgrade` runs with a released old side (`pg_upgrade_32`, `pg_upgrade_31`: the newest 3.2 and 3.1 tag, built for the old major, `SIFT_PG_OLD_REF` in `pg_upgrade.sh`): the release writes BM25, `sift_col` (without the options of 3.3) and `sift_table` objects, `pg_upgrade --check` and the upgrade carry them, answers are equal before the extension is updated and after `ALTER EXTENSION UPDATE`, the preflight blocks nothing, and writes, VACUUM and merges pass. Across majors it needs the previous major's binaries (`SIFT_PG_OLD_BIN` or the PGDG directory).
- Fixed: pg_sift: a scan that fuses the arms of a two-level aggregate reported to the plan above the sum of the groups of its arms, though each arm has no more groups than the rows it reads: the inline plan of a shared CTE looked several times dearer than it is and the shared plan, which keeps an arm off the scan, was kept. The node reports at most the rows its arms aggregate.
- Fixed: pg_sift: `sift.upgrade_check('3.2')` (and `scripts/upgrade-check.sh --to 3.2`) did not list an index with `point_tid_columns` (column format 16) as a blocker for a rollback to 3.2, 3.1 or 3.0, which refuse it; `SET sift.col_format_minor` and `REINDEX` is the advice it gives, and `docs/UPGRADING.md` section 4 tells how to drop the option first.
- Fixed: pg_sift: the private memory of a `sift_col` build is bounded by `maintenance_work_mem`, independent of the number of rows: the writer of a finished segment was never freed, so the memory of `CREATE INDEX` grew with the number of segments it had written. The index bytes are the same.
- Changed: pg_sift: a run of a correlated `LIMIT 1` child per distinct input is priced at half of PostgreSQL's cost of the child (`AGG_CHILD_RUN`; it was all of it, from a sweep of tables of 100 to 122,000 rows and a mapping child), and a row of a first-row lookup table at a tenth of a unit (`AGG_FIRST_ROW`; was one). The two-level scan that runs the child once for the tuples of its arms is now chosen over the normal plan above the three separate arms it replaces, and the first-row table is still read where the child would run for more inputs than the table has rows. A statement whose estimated inputs fall between the old and the new price changes plan: of the 258 corpus statements at `work_mem` 4GB and the default, 49 change (the arms of the monthly shapes merge into one scan) and 115 read a first-row table (117 before; two that it made slower leave it, two gain it).
- Changed: pg_sift: the arms of a two-level scan may differ in the type of the join of a lateral `LIMIT 1` child (a lookup in a mapping table): a NOT NULL test of what the child returns makes PostgreSQL join it by an inner join in the arms that have the test, and by an outer join in an arm without it, and the arms were refused as `two_arms`. The scan keeps the outer join (a row the child has no row for is a tuple of NULLs, which the arm without the test counts) and the arms with the test reject that tuple, which is why PostgreSQL made their join an inner one; a join of anything else (a table, a child without `LIMIT`) that differs keeps the refusal. Test `agg_two_level_arm_where`, fuzz family `twolevel`.
- Changed: pg_sift: the first N rows of each group asked with a window function (`WHERE rn <= N`, `rn < N + 1` over a sub-select or a CTE read once with the one window function `row_number() OVER (PARTITION BY .. ORDER BY ..)`, over a table with a column index) are also planned as the group's arrays of each column, ordered, cut at N elements and unnested with their ordinal as `rn` (`array_agg(c ORDER BY ..)`, `unnest(a[1:N]) WITH ORDINALITY`), and that plan is kept when it runs in the scan and costs less than the window. The arrays hold the rows of each group: a query whose groups do not fit the memory of a hash aggregate keeps the window. `sift.explain_why` names `window_first` for a window it leaves as written.
- Changed: pg_sift: VACUUM leaves the large segments of a `sift_col` index that is due for the automatic rewrite where they are: it moved them into the free blocks below, a copy of the bytes of the index that also made the segment new, which started the WAL budget of the rewrite (`sift.col_auto_merge_wal_pct`) over, so the rewrite was refused every second cycle and the units stayed dirty. Churn of 3% of the rows in 10 cycles, 50,000 to 800,000 rows, the server's own worker: the units that answer from their pre-aggregates after the maintenance, 50 to 75% of the cycles before, are every cycle with 400-byte rows (50% with light rows, where the budget refuses every second rewrite), for the same WAL (6.2 against 6.2 MB a cycle at 400,000 rows). The peak size of the index file is unchanged (2.5 to 2.7 times the live bytes at 200,000 rows and more), the average size while the churn lasts is higher (2.7 against 2.0 times, 100,000 rows, 40 cycles) because the large segment is no longer compacted at every VACUUM; when the rewrite is not due VACUUM moves the segments as before
- Fixed: pg_sift: the VACUUM that the `sift.col_auto_merge` worker runs flushes the WAL first: with `synchronous_commit = off` the commit record of the churn was not on disk, PostgreSQL sets no hint bit for such a transaction and the pages that held only its new row versions stayed out of the visibility map (92 of 1,910 pages after a change of 5% of the rows, 0 now), so the scans that followed checked their rows in the heap until the next VACUUM; with `synchronous_commit = on` (the default) nothing changes
- Fixed: pg_sift: the worker that rewrites a `sift_col` index by partitions (`sift.col_auto_merge_partition_kb`, and every index over `sift.vacuum_rewrite_limit`) slept less than `sift.maintenance_cost_delay` asks for: the pages written were counted for each partition, and the pages under 16 of each (the rest, never carried to the next) earned no delay; a rewrite of 89 pages with a delay of 100 ms took 272 ms, 468 ms now
- Fixed: pg_sift: the aggregate scan of a `sift_table` skips the units whose zones exclude the filters of the statement (a range, an equality, an `IN` list, `IS NULL` tests, an `OR`), as it does for a `sift_col` index: it began every scan with no zone test, so a range on an ordered column read every unit (2,000,000 rows, a range of 10,000: 27 units and 4.3 ms, now 1 unit and 1.6 ms). The test of the filters moved to one place both sources use (`agg/zoneprune.c`) (test `tbl_zone_prune`).
- Added: tests: `test/fuzz/diff_fuzz.py --table` runs the random queries of the differential fuzz on `sift_table` fact tables (no `sift_col` index, jsonb paths declared with `sift.table_set_paths`) across DML, rolled back writes, VACUUM [FULL] and `sift.table_maintain`, against the normal plan of the same table; the families that need an option of the column index (`key_cut`, `order_columns`, the point directory) are skipped, and `scripts/ci.sh` runs it as the job `difftest-fuzz-tbl`.

## 3.3.0 - 2026-10-24

- Fixed: pg_sift: `UNION ALL` arms that share one pass over the index (`Arms In One Pass`) and include an aggregate of a lookup's column (`count(d.label)` over a joined table) failed with `sift: aggregate kind 14 has no partial table` whenever a parallel worker took a share of the scan; the shared pass now runs in the leader alone when its aggregates are not ones the leader can merge. It showed on a small table only when the workers happened to get a morsel (an arm64 runner); `agg_fuse_aggs` now runs with `parallel_leader_participation = off` so that the workers always scan.
- Added: bench: twelve cases of realistic query shapes in `bench/cases.tsv` (grouping_sets_report, ranked_top_per_group, window_json_report, two_level_spend, group_by_id_top, count_distinct_unique, semi_exists_self, anti_not_in_self, keyset_page, point_agg_fk, point_agg_id, jsonb_each_member): reports with grouping sets, ranks and JSON documents over the groups, aggregates of aggregates, one group per row, semi and anti joins on the same table, keyset pages, lookups by key and one member of a JSON object read through `jsonb_each`; `bench/bench cases --work-mem 4GB` times every lane at a large work_mem.
- Fixed: pg_sift: a UNION ALL whose arms run in one pass could stop the backend (SIGSEGV) when a group key held a hashed sub-select (`CASE WHEN kind IN (SELECT id FROM kinds) THEN lookup.name END`) next to a column of a lookup join and other sub-selects of the statement came before it: the pass renumbered the sub-select's own output parameter as a column of the join and read past the end of the join's columns; the parameter keeps its number.
- Tests: `compat_matrix` runs every release tag of the 3.0, 3.1 and 3.2 series (v3.0.0 to the newest, picked from `git tag`, so a new release joins with its tag; `SIFT_COMPAT_EXTRA_REFS` adds a release in preparation): ALTER EXTENSION UPDATE with indexes of every kind, answers equal, `col_verify` clean, and the release reading what this tree writes or refusing it. It ran three releases before.
- Changed: pg_sift: the set of a `count(DISTINCT)` over an integer column of a wide range starts as a hash set, not a bitmap, when the groups hold few values (the planner's rows over its groups are fewer than 16 times the chunks of 4096 values the zones span): a bitmap costs a directory and 512 bytes for each chunk a group has a value in, 880 bytes a group for a column of 100,000 values with a row or two of each in a group. 1 million rows over 100,000 groups of a column of 100,000 values: 293 to 48 ms and 484 to 63 MB of peak memory; over 1 million groups, 444 to 116 ms and 876 to 148 MB (`work_mem = 4GB`). A group of many values (1,000 groups) keeps the bitmap. The answers are exact in both.
- Fixed: pg_sift: a parallel scan whose worker failed could report `sift: a worker of a parallel aggregate stopped without its result` (08006) instead of the error of the worker (`integer out of range`, 22003) when the leader saw the worker leave before it had handled the signal that carries the error; the leader now reads the error queues first.
- Fixed: pg_sift: `ORDER BY` a `bytea` column with `OFFSET` and `LIMIT` in the top-N scan could return other rows than the normal plan (rows of the empty string, `'\\x'`, were skipped) after the backend had run several scans over the same dictionary.
- Fixed: pg_sift: a grouped scan of a partitioned table failed with `relation N has no parent because it's being detached` when a `DETACH PARTITION .. CONCURRENTLY` of one of its partitions was in progress while the statement was planned; it now plans as it did before the group-by-partition-key finish.
- Changed: pg_sift: a row comparison in the WHERE of an aggregate or top-N scan, `(a, b) > ($1, $2)` (the page of a keyset pagination), is planned as the comparisons it means, `a > $1 OR (a = $1 AND b > $2)`, which the zones, the summaries and the kernels test: `ORDER BY c, id LIMIT 10` from a key 60,000 of 128,000 rows on reads 5 of 500 units (0.3 ms), 239 before (6.4 ms); the rows are those of the normal plan for each operator, NULLs, three columns and parameters
- Changed: pg_sift: an aggregate scan that begins for every morsel of units (SiftAgg) finds the superblocks the zones and summaries leave a unit in once for the scan, and does not begin the others: a request that prunes nearly every unit of a table of 1,000 units costs 381 instructions a unit (callgrind, 500 and 1,000 units), 1,676 before, of which 126 are the planner's sample (at most 2,048 units) and 56 the count of all-visible pages (now with the popcnt instruction where the CPU has it); the units read and pruned are the same. Finding the unit of a superblock in a segment gallops from the unit at the same share of the directory, not from its middle
- Changed: pg_sift: an aggregate scan that begins for every morsel of units (SiftAgg) finds the superblocks the zones and summaries leave a unit in once for the scan, and does not begin the others: a request that prunes nearly every unit of a table of 1,000 units costs 268 instructions a unit between 600 and 1,800 units (callgrind), 1,676 before, of which 210 are the planner's sample (at most 2,048 units, so it stops growing) and 88 the count of all-visible pages (now with the popcnt instruction where the CPU has it); the zones of each word of 64 units clear the word when none of its units can meet a range or list, so a request on a clustered key tests the words, not the units, and the scan visits only the superblocks that hold a unit (0.03 to 0.08 ms at 1,800 units, 0.22 ms at 1,000 before); the units read and pruned are the same. Finding the unit of a superblock in a segment gallops from the unit at the same share of the directory, not from its middle
- Changed: pg_sift: a group key of a two-level scan that holds a hashed sub-select (`CASE WHEN kind IN (SELECT id FROM kinds) THEN .. END`) is one every arm groups by when the arms' sub-selects are the same query: each arm plans a sub-plan of its own for it, and the keys now compare equal when they differ in nothing but that planning's numbers, so the scan compiles the key once and asks the sub-select once, where the statement was refused as `two_keys`. Arms with different sub-selects keep the refusal. Test `agg_two_level_subplan_key`, fixture `s2_per_order_dimension_subselect`, fuzz family `twolevel`.
- Changed: pg_sift: `sift.col_auto_merge` is on by default (3.2: off; it needs `sift.async_maintenance`, on), so that a `sift_col` index keeps answering from its pre-aggregates after churn without a setting: a worker vacuums the table when a thousandth of its rows changed and rewrites the index after the `VACUUM`, within `sift.col_auto_merge_wal_pct` (5) of the WAL the server writes. The rewrite writes the bytes of the index, so it waits until the server has written twenty times that since the index was written (its build or its last rewrite, read from the page LSN of its oldest large segment: nothing is stored), whatever the width of the rows, the full-page images or the other indexes make of the WAL; `0` rewrites at every `VACUUM` that finds the index due, which cost 12% to 28% more WAL without full-page images and 0% to 3% with them (1M rows, 3% of the rows changed per round, 20 rounds). With the default: +0.9% to +1.9% WAL with full-page images and a checkpoint per round, the units answerable from their pre-aggregates 32% to 51% (wide rows) and 33% to 83% (scraper mix), and nothing changes for a hot set or for new rows. `sift.col_auto_merge_wal_pct` replaces `sift.col_auto_merge_min_churn_pct` (never released), `sift.col_maintenance_status` names the wait in `blocked_by`, `CREATE INDEX` and `REINDEX` of a `sift_col` index say what the worker does and how to stop it in a NOTICE, and it applies to the indexes that exist when the library is loaded after the update (`sift.col_auto_merge = off` for the server or the database keeps 3.2). The test clusters of `scripts/pgtest.sh` run with it off.
- Changed: pg_sift: a group table that spills writes each `count(DISTINCT)` set, and the next table of the scan starts the words of its narrow sets with room for as many groups as the table before it held, where each table grew them from 64 groups by doubling and copying: the scan of 38,646 groups with two such sets under `work_mem = 1MB` executes 2% fewer instructions (454 to 445 million; the time moves less than the noise of a busy host). The words count in the hash memory as they grew before.
- Changed: pg_sift: the scan is priced for the units its join key filters keep as the data has them, so the plan no longer leaves the scan when the table grows and the request stays the same. The keys of a lookup whose rows PostgreSQL only guesses (the output of a plan) are counted as `lookup_set_ratio` counts them (the geometric mean of its estimate and the leaf's distinct values); the share of units whose zone holds one of them comes from the sample of the real zones (a few wide zones among narrow ones made a mean width that kept every unit), cut by the small-set summaries of the units; and the normal plan that reads the table by the probes of a join below a merge or hash join (or a Materialize) is charged those keys, as the probes of a nested loop alone were. At 316 and 3,160 distinct keys the scan cost 3,208 and 8,880 and the normal plan 389,929 and 738,908 for a request of 32; the scan cost 4,160 and 11,315 and the normal plan 18,989 and 6,841 before, and was refused at the larger size. Of the 237 corpus statements, 102 take the scan at 3,160 keys (none did) and 46 more at 316, and none lost it.
- Tests: `agg_scope_cost` adds the facts joined to a third table; `agg_semi_lut` runs its workers case without the join key filters, which now keep too few units for workers to pay.
- Changed: pg_sift: the price of a `LIMIT 1` child per distinct input, against its table read once as a first-row lookup, counts the inputs of an unnested list by the distinct values the index holds for each field of its elements (the widest field of an element, as a scoped lookup already did) and by the tuples the scan reads, not by PostgreSQL's default of 200 values a field and the rows it expects of the join, which a filter it has no statistics for leaves at a few. A child over a thousand elements with a key of their own now reads its table once (`agg_first_row_lookup` adds the statement and its three-document twin, which keeps the child).
- Tests: `agg_first_row_lookup` adds a list of elements whose filter PostgreSQL has no statistics for.
- Changed: pg_sift: a scan with workers is priced for starting them, 3 ms and 0.35 ms for each (the larger of that and `AGG_SETUP_FACTOR` times `parallel_setup_cost`), so a low `parallel_setup_cost` no longer sends a scan of a few milliseconds to workers; and a `count(DISTINCT)` is priced for the leader merging the (group, value) pairs of the workers' sets into its own one by one, so its workers do not scale and a grouped one stays serial where they do not pay. A `parallel_setup_cost` of 0 asks for the workers and prices neither. The sum, count and average scans keep their workers at the default and benchmark costs.
- Tests: `agg_distinct_cost` adds the workers planned for a sum and for a `count(DISTINCT)` at setup costs of 0, 1 and 100.
- Changed: pg_sift: the items that the participants of a scan with workers take from the counter are a quarter of the span of a superunit (were the whole span), so that a table of a few hundred superblocks gives them more than two items. The superunits an item holds whole are still answered from their summary; a serial scan keeps the span.
- Changed: pg_sift: a group table that spills writes each `count(DISTINCT)` set that is narrow (a column of up to 512 values) as the words of its window, one row for each group that has a bit, and a parallel worker sends it the same way; the receiver ORs the words into its own and counts the new bits by popcount, where it was one (group, value) pair for each value, walked once for the partitions and added one by one: the scan of 38,646 groups with two such sets under `work_mem = 1MB` executes 1% fewer instructions and takes 5% less time, and a dictionary column's words go with the values of the ids that have a bit. The blobs are the temporary state between a scan and its merge (spill files and worker messages); nothing stored changes.
- Changed: pg_sift: a backend keeps the checked head of a segment (its schema and unit directory) and the zone arrays of each leaf it has read (`dict_cache_kb`) for the next query, where each query read and validated both again: a hit on a table of 782 units costs 25 to 60 instructions a unit less (uuid, integer and text columns; callgrind), and a segment that is rewritten, a damaged header or a cache with no room reads them as before. `sift._test_col_dict_cache` counts the loads and hits
- Changed: pg_sift: counting the all-visible pages of the heap (each plan of a column index and each aggregate scan does) reads the visibility map a word at a time, not through a copy of each word: 185 instructions a unit of 256 pages less (callgrind, tables of 391 and 782 units), the count is the same
- Changed: pg_sift: a group table that spilled starts the next table of the scan with room for as many groups as the one it wrote (the keys, the hash slots and the per-group arrays), where each table grew from 1,024 groups by doubling and copying its arrays again: the aggregate scan of 38,646 groups under `work_mem = 1MB` executes 8% fewer instructions (496 to 459 million), 5% less time. The room counts in the hash memory as the empty table does; the answers do not change.
- Changed: pg_sift: the scan is no longer priced for the bitmap chunks of a `count(DISTINCT)` of a column whose values fall in a window of 512 (`dist_narrow_words`, 8 words) or less: the range of the column's zones, or the dictionary of a text column. Those sets keep a few words for each group, since the narrow form. `count(DISTINCT quantity), count(DISTINCT status)` of 38,646 groups was priced 13,953 at `work_mem` 16 MB against 2,552 at 4 MB (the normal plan 22,475): the price no longer rises with the memory. The plans of the 237 corpus statements at `sift.plan_scale` 1, 10 and 100 are the same.
- Tests: `leak_path_vets` (TAP): a session that recorded the heap check of an empty visibility map ends normally and another is terminated in a statement; the leak gate finds nothing left over. The context of the heap-check list is named `sift cache path vets`: a backend keeps it on purpose.
- Changed: pg_sift: `sift.explain_why` says what its `scan_cost` and `normal_cost` are: the totals of the path that was built and of the cheapest normal path at the grouping stage, before an `ORDER BY` or a `LIMIT` above it (PostgreSQL chooses by the cost of the top node, a `LIMIT` included); the numbers are unchanged. The test `agg_refusals` checks that a `LIMIT` leaves them as they are.
- Fixed: pg_sift: a partitioned table whose partitions keep different key lists (`key_columns = auto` follows the data of each partition: one keeps the list of an integer column, its neighbour does not) is one source of the scan again, counting on the lists every partition keeps, instead of keeping the normal plan with `partition_source` (a table of 128 partitions by hash of a column with a few values in each did). A unit without a list is read by its rows. Test: `agg_part_presence`.
- Changed: pg_sift: the backend's cache of parsed dictionaries and list lengths (`dict_cache_kb`) finds its least recently used entry in a list, not by walking the whole table for each entry it drops: a query that touches more units than the cache holds (about 12,800 at the default) drops one entry for each it adds, and the walk was as long as the cache.
- Fixed: pg_sift: the histogram of a date or timestamp column asked of a superunit summary (a table of more than one unit; `count(DISTINCT)` of the date) read the column array of the schema past its end, a heap overflow that an address sanitizer reports: the calendar cells of a date are not merged into a superunit, so it now says it has none, as it did in the runs that did not fail.
- Changed: pg_sift: a scan that begins for each morsel of units (SiftAgg) keeps the units its zones and summaries cleared for the next begin while the predicates stay the same (not for an OR, a point directory answer or a wide integer), where each begin tested its units again. 782 units, `count(*)` and `avg` of a column with every unit pruned: 4.5 M instructions before, 2.6 M after (`count(*), sum(col)`: 3.5 M, 2.1 M).
- Tests: `agg_visibility_refusal` plans a refused shared-CTE statement and another aggregate statement in one transaction with `sift.explain_refusals` on, and runs in the assert lane (`scripts/assert.sh`): the build before the fix of the heap-check list (3648712) fails it with an assertion failure in `lappend`.
- Changed: pg_sift: the list lengths a backend keeps for the next query of a list leaf (`dict_cache_kb`) are 16 bit and the running ends are summed again from them, 5.1 kB a unit of 2,076 rows, 18.5 kB before: the 64 MB of the cache hold the lengths of about 12,800 units, not 3,500 (a unit with a list of more than 65,535 elements is not kept). Results are the same.
- Changed: pg_sift: the scan is priced for the jsonb values that the test of a lookup join without an equality key builds: `LEFT JOIN t ON jsonb_build_array(a) = jsonb_build_array(b::date)` (a cast of the column of the table is not a hash key) tests every row of the table for every distinct value of `a`, and each test builds two jsonb values in PostgreSQL's executor, 70 operators each, at the executor's speed instead of the scan's. A 701-day series over 200,000 events was priced at 22,983 for the scan and 51,857 for the merge join, and ran slower in the scan; it keeps the normal plan now. The join on the column itself is a hash key and keeps the scan. Test: `agg_unkeyed_json_cost`.
- Changed: pg_sift: the set of a `count(DISTINCT)` over a column whose values fall in a window of 512 (a status, a month, a dictionary of few values, a small code) keeps up to 8 words of bits for each group, 64 bytes at most, where a bitmap keeps a directory and a chunk of 1,024 bytes: the state of two such sets over 38,646 groups goes from 77 MB to 3 MB, and over 100,000 groups of a 5-value column from 102 MB to 1 MB. A value outside the window widens it, and past 512 values the set becomes a bitmap; an integer column whose zones span more than the window starts as one. The limit `dist_narrow_words` (docs/LIMITS.md) sets the words; 0 turns the form off.
- Fixed: pg_sift: `sift.explain_why` and `sift.explain_refusals` could crash the backend (a segmentation fault) on a statement whose plan keeps an aggregate scan while the visibility map of its table is empty: the heap check of each path the arms priced was kept in a memory context that the choice between the plans of a statement had deleted before it was read. It is kept in a context of its own now.
- Changed: pg_sift: a scan that begins for each morsel of units (SiftAgg) tests the zones of the units of that morsel only, and stops at the first predicate that leaves none, where each begin tested every unit of the segment for every predicate: the zone pass of a scan grew with the square of its units and with each aggregated column. 782 units, `count(*)` and `avg` of a column with every unit pruned: 14.0 M instructions before, 4.5 M after (`count(*), sum(col)`: 7.8 M, 3.5 M).
- Added: pg_sift: a serial scan of a partitioned table grouped by its partition key (the column itself), with a `count(DISTINCT)`, finishes the groups of a partition when its units end and starts the group table again, so the table holds the groups of one partition, not of all, and does not spill at `work_mem` (`EXPLAIN` shows `Groups Finished: per partition`; the cost is that of one partition's table). Not for a scan with workers, a float sum or average, grouping sets, a partition of a partition or an expression as the partition key. Test `agg_part_final`.
- Tests: `agg_group_spill_fuzz`: 64 seeded `count(DISTINCT)` mixes under the smallest hash memory, the scan forced, against the normal plan's rows; the build before the fix of short counts in a merged partition fails it on 3 seeds.
- Changed: pg_sift: a generic plan of `col = ANY($1)` on an integer column prices the scan for the table of the list (a byte table over the whole range of the column, for the `sift.agg_param_set_size` elements it assumes), as it does for a list it can see: it was left out, and the plan cache, which compares a generic plan with the average of the custom ones, adopted the scan for lists of 256 keys over a range of a million where the index plan is several times faster.
- Changed: pg_sift: the first plan of a backend on a `sift_col` index counts the pre-aggregate sections of up to 256 of its units (spread over the segments, in proportion to their units), not of every unit: it read 2.5 buffers of each, 2,921 buffers of 1,000 units of an index of seven leaves and a cube, 1,064 now (7.2 ms of planning to 2.0 ms), and an index of 15,000 units 42,000. The planner uses the share of the units that have a section, which the units counted give; an index of 256 units or fewer is counted whole.
- Changed: pg_sift: the scan is priced for the table of an `IN` list on an integer column (a byte table over the range of its values, a bit table for a long list over a wider range), which it allocates and clears when it starts whatever the rows it reads: a list of 256 keys over a range of a million values added about 2 ms to a scan that read two units, and the planner took the scan where the index plan was several times faster. A hash table (a range past 4,194,304 values) is not charged.
- Fixed: pg_sift: a `count(DISTINCT)` aggregate scan over a column source that gives no zone of a range of units (the test source `sift_colstub`) crashed the backend with a null function call, because the streaming `count(DISTINCT)` (`sift.agg_dist_stream`) asked for that zone; it now takes the streaming path only when the source has it, and the other sources are unchanged.
- Docs: docs/USAGE.md, memory of a backend: with `MALLOC_MMAP_THRESHOLD_=131072` for the server a backend that replays the mix of large aggregate statements holds 70 MB instead of 104 MB, for the same CPU per statement within the noise of the runs; the library does not call `mallopt` or `malloc_trim`.
- Fixed: pg_sift: a `count(DISTINCT)` of a scan whose group table spilled could return counts that missed values: when the set of a group outgrew hash memory while a partition was merged, its values went through the sort, and the groups of the partition were returned before the counts were taken from it (`agg_group_spill`, the sets of 800 groups of 94 rows at 64kB). The scan that spills and the scan that does not give the same counts.
- Changed: pg_sift: the groups of an aggregate scan go to the hash table, not to the direct arrays, once the arrays would take more than twice hash memory (`work_mem` times `hash_mem_multiplier`): the arrays hold every aggregate of every slot of the product of the keys' rooms, a million slots for ten keys, so a statement of 28 aggregates held 218 MB for 1,200 groups. The statements of a mixed workload with the largest scans peak at 117 to 159 MB (264 to 310 MB before); the rows are the same.
- Fixed: pg_sift: a scan that groups by seven or more keys cleared group arrays of 4^7 slots or more before it read a row, even when the query found a handful of groups or none: when the planner expects far fewer groups than the arrays have slots, the groups start in the hash table.
- Fixed: pg_sift: `sift.col_verify(index, true)` failed with `invalid memory alloc request size 1342177280` on a `sift_col` index of more than 16.7 million rows: the hashes of the rows it compares with the heap (40 bytes each, in one array that doubles) outgrew the 1 GB limit of an ordinary allocation. The array may be larger now, so the check runs on any size the memory allows (about 80 bytes a row).
- Changed: pg_sift: a statement whose CTEs the planner hook weighs (shared or inlined, as written or with its lateral sub-selects pulled up) keeps the memory of the plan it uses, not of the up to four plans it made: each plan is made in a context of its own and the other ones are deleted (planner memory of a CTE statement with many aggregate arms, 134 MB at most before, 44 MB; the plans are the same). Planning takes up to a quarter longer when the C library returns the freed blocks to the system at once (with `MALLOC_MMAP_THRESHOLD_` and `MALLOC_TRIM_THRESHOLD_` raised it takes as long as before).
- Changed: pg_sift: a group table that spills writes each `count(DISTINCT)` set (and each distinct `sum` and `avg`) in one walk for all its partitions instead of one walk for each, and a `count(DISTINCT)` whose groups hold few values starts the tables after the first spill as hash sets instead of building and moving bitmaps again; the answers are the same. The key ids of each partition are renumbered through a map the scan keeps clear between two partitions, where a map of the ids of the whole table was made and cleared for each of the 128 partitions of a spill, and the columns of the groups are copied with a load and a store where a call to memcpy copied each value: `count(DISTINCT quantity), count(DISTINCT status), sum(amount)` by 193,409 customers of 1,000,000 rows at `work_mem` 1MB (20 spills) takes 2,133 million instructions where it took 2,424, 272 ms of CPU where it took 310 (the normal plan: 296).
- Changed: pg_sift: the planner prices the access-method scan of a `sift_col` index (`sift.enable_col_scan`) by what it does: a start-up for opening the index and pruning the units (which a `LIMIT` does not wait for), then the units kept, the values filtered and the rows handed over, in about microseconds; an equality or short `IN` on a column with a point directory (`point_columns`) is priced from the rows of its keys (text, uuid and a parameter of a generic plan included, which cost 4,000 and now cost as the constant does), and the zones of a long `IN` list are bisected instead of tested against every key (a list of 50,000 values adds about 3 ms to the planning). Against the measured time, the median error of the price falls from 2.3x to 1.9x (6.7x to 2.1x for the point lookups).
- Changed: pg_sift: the VACUUM of a `sift_col` index leaves alone a tier of clean segments that share at most the superblock at their edge (the segments of a parallel build, of rows appended and promoted) when they total more than `sift.vacuum_rewrite_limit` and the directory has room: the merge gave a reader nothing and wrote all their bytes, so the first VACUUM after a parallel build rewrote the index (nine segments of 900,000 rows: 1.3 MB of WAL for the VACUUM against 8.8 MB). A dead row in one of them, or a full directory, merges them as before.
- Fixed: pg_sift: before PostgreSQL 18 the aggregate scan returned a constant group key as the constant in the sets that leave it out (`SELECT 'cur' AS label, status, count(*) .. GROUP BY GROUPING SETS ((1, 2), ())` gave `'cur'`, not NULL, in the grand-total row): PostgreSQL's setrefs.c never replaces a bare constant of a custom scan's output with a column of its scan tuple, so the output now reads the constant through a relabel to its own type. Test `agg_grouping_set_keys`, fixture `k_group_const_sets` (its `ORDER BY 1` on the constant is no order before 18: it orders by the other columns).
- Fixed: pg_sift: `CREATE INDEX` and `REINDEX` of a `sift_col` index with parallel workers showed no heap blocks in `pg_stat_progress_create_index` (`blocks_total` and `blocks_done` stayed 0) and counted rows only while the leader scanned; the workers now report the blocks and rows they did through the leader, as the BM25 build does (`test/isolation/specs/col_build_progress.spec`).
- Changed: pg_sift: the access-method scan of a `sift_col` index probes the point directory before it walks the zones and set summaries of the units (when the directory answers, the walk would only narrow what it already narrowed), and skips the pruned units a word of the keep bitmap at a time, so an equality or short `IN` list costs less for each unit of the index; the rows and the units read do not change (a regress counts them over a segment of 65 units).
- Added: pg_sift: `sift.col_index_status`, one row per `sift_col` index of a table the caller can read: its size and that of its segments, the free and reusable blocks, the segments against the directory's 128, the dead rows, the write buffer and the state of the worker, joined with `sift.col_maintenance_status`, so the state of an index on a table of hundreds of millions of rows is one `SELECT`.
- Fixed: pg_sift: a scan that groups by several keys paid a fixed 10 ms before it read a row: its group arrays are indexed by the keys' ids together, so their room is the product of the keys' rooms (a date key of 60 days, a category, a sub-category, a period: 32,768 slots for 62 groups), and each doubling of a key's room copied and zeroed the arrays of every aggregate in all of it. When the planner expects fewer than a quarter of the slots to be groups and fewer than a sixteenth hold one, the groups go to the hash table. A daily series of an average (a calendar of 60 days left-joined to the rows of 51 parents, unnested json, 7 aggregates of which 5 `count(DISTINCT)`), 195,000 rows, warm, minimum of 5: 47.5 ms -> 5.5 ms (the normal plan 19.1), one parent 11.8 -> 1.4 ms (normal 0.9); the 258 shapes of a query family of six endpoints, natural plans with a full map: 27.8 s -> 24.9 s. Test `agg_group_sparse`.
- Changed: pg_sift: a top-N scan (`ORDER BY <interval column> LIMIT n`, `sift_col` and `sift_table`) takes an interval column as a sort key, as it took a numeric of more than 18 digits: the entries of the dictionary of its datums are compared with `interval_cmp`, so `1 mon`, `30 days` and `720:00:00` tie and each row shows its own spelling. At 200,000 rows `LIMIT 10`: 28.4 ms to 12.7 ms for distinct values (`LIMIT 100` 32.3 to 14.8, `LIMIT 1000` 27.6 to 17.7), 26.1 ms to 4.8 ms for 300 distinct values, 11.3 ms to 0.38 ms with a `WHERE` that leaves 1% of the rows.
- Changed: pg_sift: a top-N scan (`ORDER BY <enum column> LIMIT n`, `sift_col` and `sift_table`) takes an enum column as a sort key, in the order of its type: the column units hold the OIDs of the values, which are not in that order (`ALTER TYPE ... ADD VALUE BEFORE/AFTER` changes it without touching a row), so the scan reads the order from `pg_enum` when it opens and ranks each OID; the zones of the column (of OIDs) do not bound the scan. At 200,000 rows `LIMIT 10`: 21.4 ms to 3.2 ms for a type of 5 values, 20.4 ms to 6.8 ms for a type of 301 values, 8.4 ms to 0.14 ms with a `WHERE` that leaves 1% of the rows.
- Changed: pg_sift: a top-N scan (`ORDER BY <numeric column of more than 18 digits, or without a precision> LIMIT n`, `sift_col` and `sift_table`) takes the column as a sort key, as it took a numeric of up to 18 digits: the entries of the dictionary of its datums (a NaN among them, which sorts last, the lanes keep it as NULL plus a flag) are compared with `numeric_cmp`, so a query that sorted the whole table and kept `n` rows reads the column units. At 200,000 rows of `numeric(30,4)`, `LIMIT 10`: 24.5 ms to 15.6 ms for distinct values (`LIMIT 100` 25.0 to 15.9, `LIMIT 1000` 30.4 to 20.4), 27.0 ms to 2.8 ms for 300 distinct values, 8.9 ms to 0.25 ms with a `WHERE` that leaves 1% of the rows.
- Changed: pg_sift: a top-N scan (`ORDER BY <text under C, uuid or bytea column> LIMIT n`, `sift_col` and `sift_table`) compares the rows of a unit with the bar by the order code of the dictionary of their segment, two integers, not by their bytes: the position of each entry in byte order when the dictionary has it (stored, or sorted after the scans of a backend compared four times its entries) or the first 8 bytes of each entry (made once the rows compared reach half of its entries, and dropped for a dictionary where a quarter of the comparisons tie), for dictionaries of up to 1,048,576 entries; a `bytea` has the prefix code only. At 200,000 rows `LIMIT 10`/`100`: a text column of 20,000 distinct values 3.7 ms to 1.2 ms, 200,000 distinct 2.8 ms to 2.2 ms (2.5x when the query repeats on a backend), six values 4.6 ms to 3.0 ms, strings that share their first 8 bytes 2.9 ms to 2.4 ms, `bytea` 6.6 ms to 2.5 ms; with a `WHERE` that leaves a few hundred to 20,000 rows no change. EXPLAIN ANALYZE shows `Top-N Units By Order Code`.
- Changed: pg_sift: a top-N scan (`ORDER BY <uuid or bytea column> LIMIT n` over a `sift_col` index) reads a uuid or bytea sort key as the codes of the dictionary the segment stores, not through a dictionary of aligned datums built for every unit, when no other part of the scan reads the leaf (a `WHERE` or an expression on it keeps the old read; so does a `sift_table`): about 2x faster at 200,000 rows (`ORDER BY ref LIMIT 10` 5.2 ms to 2.4 ms, `LIMIT 100` 6.5 ms to 3.3 ms, `bytea` `LIMIT 10` 6.6 ms to 2.9 ms; with a `WHERE` that leaves a few hundred rows no change). EXPLAIN shows `stored dictionary` in the `Top-N` line.
- Changed: pg_sift: a partitioned table whose partitions cost the scan more to open than the whole normal plan costs keeps the normal plan before any partition is looked at: the opening of the partitions is priced from their number alone (the planner already charges it), and was compared with the normal plan only after the layout, statistics and visibility map of every partition's index had been read. Hash partitions of 62 rows, `GROUP BY country`, planning only: 3,200 takes it too, planning in the time it takes with `sift.enable_agg = off` (43.5 ms against 44.8, 94.9 ms before), and 4,000 the same (48.9 against 52.2, 112.6 before), and the first plan of a new backend takes 168 ms where it took 4,480 ms (it read the facts of every partition); 400 partitions keep the scan and plan as before. `sift.explain_why` names the opening and the normal plan's cost. Test `agg_partitions_many`.
- Changed: `scripts/pg-assert.sh` builds the assertion server with `-DCATCACHE_FORCE_RELEASE`, so the `assert` lane finds a catalog tuple or a slot used after its release (a full `make pg-assert` regress of 18 tests takes about 6 minutes).
- Fixed: pg_sift: an aggregate scan whose lookup table, combination or join key holds a sub-select that PostgreSQL hashes (`NOT IN (SELECT ..)`) in an expression it evaluates for each distinct value kept the slots of the sub-plan in memory that the scan frees when it ends, before the executor frees the slots: a freed pointer in the tuple table, an assertion failure or a crash at the end of the statement. The state of such an expression is built in the query's memory, as the lookup tables were.
- Fixed: pg_sift: the first statement of a session that has not preloaded the library (`session_preload_libraries`, `shared_preload_libraries`) was planned without the planner hook, because the library is loaded while the planner opens the index: a shared CTE was not planned inline and a lateral sub-select was not pulled up, so the aggregates that every later statement runs in the scan ran their ordinary plan. A plain `SELECT` without parameters is now planned again through the hook when it starts, and the plan cache is reset; other first statements keep their plan (docs/USAGE.md).
- Changed: pg_sift: planning a query with a long `IN` list or `= ANY (array)` no longer repeats the work of the planner: the selectivity of the list is the one the planner kept for the condition (it was estimated again for every element, and again for each path), and `EXPLAIN` shows a constant array of more than 1000 elements as its first 8 elements and the count.
- Added: pg_sift: a long `IN` list or `= ANY (array)` over a range of values wider than the 4 million of a byte table (1024 values or more, up to 2^28 wide) is tested by a bit table instead of a hash table, and the zone test of a set travels to the column layer up to twice the units of the scan (it was a fixed 256 values), once per scan instead of once per range of units.
- Changed: pg_sift: the rows a lookup join loads (`IN (SELECT ..)`, `EXISTS`, a join) are read by the heap scan, not through an index, when an equality of the joined table keeps a third of it or more, as an `IN` list of the table already was.
- Changed: pg_sift: the planner takes the span of a timestamp key's buckets (`date_trunc`, `::date`, `extract` of a `timestamp`) from the smallest and largest value the index holds, where it needed the histogram of the column in `pg_statistic`: a table not analyzed since it was loaded, or whose histogram is missing its extremes, was planned with PostgreSQL's 200 groups (24 months now). The plans of the 258 corpus API shapes are the same. No format change. Test `agg_group_estimates`.
- Changed: pg_sift: the planner expects the groups of a calendar key on a date column (`date_trunc`, `extract`, `date_part` of a `date`) from the days the column takes, over the days one value of the key covers, where it took PostgreSQL's count of distinct dates (a month key of 700 days was planned as 700 groups, now 24). The scan's estimated rows, and the memory it is charged for the group table, follow. The plans of the 258 corpus API shapes are the same. No format change. Test `agg_group_estimates`.
- Added: pg_sift: the `neon` level of `sift.simd` for AArch64 (Linux arm64 and Apple silicon; NEON is part of the architecture, so nothing is detected): NEON kernels of the column filters, selections, zone maps and unpacking, the aggregation operators, the shredder output, the BM25 scoring, the bitmap population count and the unpacking of the chunk codec; the other kernels are the scalar ones, where NEON was not faster. Every level returns the same results bit for bit. `sift.simd` takes the level names of both architectures and the names of the other architecture cap nothing, so one configuration file serves a mixed fleet; `auto` is the best level of the CPU and the build needs no `-march` flag on arm64. SVE is not used.
- Fixed: `make check` and the test tools on macOS arm64 and on other CPUs than x86-64: one checksum helper (`scripts/sha256.sh`; the BSD `sha256sum -c --quiet` of macOS fails without a file operand, which stopped the 20 Perl TAP tests), `-march=x86-64` and the x86 tables of the standalone tools and the fuzz targets only on x86-64, the pinned digests of `kernels_simd` and `aggkern_simd` no longer depend on the compiler (the order of the arguments of a call) or the CPU (the sign of a NaN), `shred_kernels` states its level against the CPU, the harness keeps its Unix sockets under the 103 bytes of macOS, and what a host cannot run is skipped with a message. The CI workflow runs the PostgreSQL 18 tiers on Linux arm64 as well.
- Changed: pg_sift: an inline shared CTE whose scans cost less than the work of the shared plan they replace, but not by the margin, is planned again at the square root of `sift.agg_scale_horizon` (default 100) times the rows and at that many times (with `sift.plan_scale`), and the inline plan is kept if it wins at both: a close call at the size the table has no longer flips to the shared plan at a size in between where the group tables of the scans outgrow `work_mem` and the plan of the shared CTE still grows faster. `sift.agg_scale_horizon = 1` keeps the choice at the size it is.
- Fixed: pg_sift: `sift.workers` failed with `could not open file` in a database that has a `sift_col` or `sift` index on a partitioned table, because the view listed the index of the partitioned table itself, which has no storage; it lists the indexes of the partitions only.
- Changed: pg_sift: the planner charges the heap check of a scan on a table whose visibility map is not full (after `CLUSTER`, `VACUUM FULL`, `pg_repack`, or churn before the autovacuum) by the pages it reads, not by the rows: the pages of the kept units that hold a selected row, each at the price of a page read from shared buffers (about 1.3 us measured), from the cache of the operating system (3 us) or from the disk (`seq_page_cost` to `random_page_cost`, by the density of the pages the scan skips), weighted by `shared_buffers`, `effective_cache_size` and the size of the table, and 4 ns a row; it charged one cost unit for each selected row. 258 query shapes of six families on a 195,000-row table of wide rows, clustered and not vacuumed: 108 of them plan on the scan before (254 after, as with a full map), and the natural plans take 113.8 s in all before and 34.9 s after (33.0 s with a full map). A full map prices as before. The refusal text and `sift.explain_why` quote the new charge. Test `agg_visibility_refusal`.
- Added: pg_sift: `sift.plan_scale` (default 1, PostgreSQL up to 18): plans as if the tables with a `sift_col` index had that many times the pages and rows, with the dimension tables and the statistics of the columns as they are, so that `EXPLAIN` and `sift.explain_why` show the choice of the planner and the cost of the scan and of the normal plan at a larger size without the data. Only the plan changes; the rows are those of the table.
- Fixed: pg_sift: a row that the aggregate scan reads again from the heap (a value its index cannot store, a recheck) could crash the backend when the index had a toasted column and an invalidation of the index (another session's `VACUUM` or `ANALYZE`) was pending: building the index row detoasts the value, which locks the toast table and reloads the relcache entry, and the loop read the schema and the column arrays from the cached state it had freed. They are read before the first value. Test `cache_discard`. The same code is in 3.2.2; the repro that reaches it is the 3.3 re-read of a `(doc ->> 'k')::float8` path.
- Fixed: pg_sift: planning an index scan of a `sift_col` index (`sift.enable_col_scan`; off by default in 3.3) with two or more conditions on its columns could crash the backend when an invalidation of the index (another session's `VACUUM` or `ANALYZE`) was pending: the cost estimate of the units the zones and the small-set summaries leave kept the index's cached state across the statistics lookups of the conditions, and read it again after the entry had been reloaded. It asks for the state where it uses it. Test `cache_discard` (a server built with assertions reloads the entry at every lock and catalog read). Not in 3.2.x.
- Added: pg_sift: `count(DISTINCT x)` of an integer column whose superblocks hold ranges of `x` that overlap little visits the superblocks in the order of the zone minimum and drops from its sets the values below the minimum of the superblocks to come, so the sets stay within `hash_mem` where the plain scan would spill or sort them; the answer is the same, the sweep is given up for the plain scan when the table of the groups would spill, and `sift.agg_dist_stream = off` restores the plain scan. `EXPLAIN ANALYZE` shows `Distinct Stream Superblocks`.
- Added: pg_sift: the aggregate scan answers `LEFT JOIN LATERAL jsonb_to_recordset(..)` / `jsonb_array_elements(..)` of a list of the scanned row, `ON true` or with conditions of its own: a row none of whose elements passes (no list, an empty one, all failing) is one tuple with NULL element columns, so a per-row group keeps its row (a per-row `avg(x)`, `max(y)` and `count(*)`). A `WHERE` above the join that sees the NULL tuple, or a lateral condition that may raise, keeps the normal plan; `sift.agg_left_lateral` (on) turns it off. Test `agg_left_lateral`, fixtures `j_left_lateral*`, fuzz family `leftlateral`.
- Added: pg_sift: the two-level scan computes `count(col)` over the per-row arms when the column is the row id of the arms (a unique NOT NULL column): it counts the inner groups, as `count(*)` does. A per-row aggregate with a sum of its own and `count(row_id)` above it was refused as `two_aggs`. Test `agg_arm_cells`, fuzzer family `twolevel`.
- Fixed: pg_sift: a text group key is read as dictionary codes, which gives an id to every entry of the dictionary of each unit the scan reads, whether a row that passes the filters holds it or not, so a near-unique text key under a filter made a table of ids as large as the rows read for the few groups it returned (it outgrew `work_mem` and spilled over and over, walking the whole table each time) and the planner priced none of it. The scan is now priced for the entries it maps, which cost more as the table of ids outgrows the caches, and for the ids in the group table and the spill passes over them; the normal plan is kept for such a statement, and a key without statistics, or with few values, is priced as it was.
- Fixed: pg_sift: `sift.col_verify(index, true)` on a table with toasted values could crash the backend when an invalidation of the index (another session's `VACUUM` or `ANALYZE`, `ALTER INDEX`) was pending while the check read a toasted value: the first fetch from the toast table locks it and reloads the relcache entry, which frees the index's cached state, and the check read the schema from it afterwards. The check now works on a private copy of that state for the whole call. Test `verify_relcache` (toasted values under `debug_discard_caches` on a server built with assertions). Present in 3.2.2.
- Fixed: pg_sift: a scan with two sets of values on one leaf (an `IN` list or the dimension filter of a table, and the keys of a lookup join or a second list: `k IN (51 values)` and `k IN (SELECT .. LIMIT 5)`) read the units the first set's zones left, narrowed only to the range of both; a unit is now read when its zone holds a value of both sets (a filter over 93 units read 39, as against 9 for the same five values as a list). Test `agg_zone_sets`.
- Changed: `bench/bench cases` keeps its scratch cluster in `/dev/shm/sift-cases` (tmpfs, `CASES_TMP` for another directory) instead of `/var/tmp/sift-cases`: the cluster is disposable and a 200,000-row run no longer writes to the disk.
- Added: three more cases in `bench/cases.tsv` (a short `IN` list over a range a byte table covers, a near-unique text group key under a filter, and `count(DISTINCT)` by group at `work_mem = 1MB`, where the group table spills and pg_sift is slower than PostgreSQL), and a `-- set: a = 1; b = 2` line in a case file for the settings of every lane.
- Added: six cases in `bench/cases.tsv` on a second shared dataset, `keys` (`bench/sql/keys/gen.sql`): `IN` lists of 50,000 and 500,000 values over a 50-million-wide key and over a narrow one, and the `IN (SELECT ..)`, `EXISTS` and join forms of a 500,000-key lookup. A case SQL file takes `{ids:TABLE:N}` for the comma-separated ids of the rows of a table with that `n`.
- Added: `bench/bench cases --discover` (`bench/steps/cases_discover.py`): measures the statements of a corpus (`test/plan/shapes.sql`, the query families of the differential fuzzer, or results measured elsewhere) on PostgreSQL and on pg_sift and proposes, for a person to accept, the shapes the scan answers at least twice as quickly that no case of `bench/cases.tsv` covers, ranked by weight times gain; a different answer is listed first and fails the run.
- Changed: `CREATE INDEX`, `REINDEX` and `sift.col_refresh` of a `sift_col` index tell, as they did for a cube dim over its cap or a set summary that is a bloom filter, the options that wrote nothing in a share of the units: a `key_columns` leaf whose keys are one value or fill their range, or whose key list or bitmap would take over 3% of the unit, a `rollup_dims` rollup with more cells than a unit may keep, and the date columns that `calendar_columns` (default `all`) leaves out because the cube holds at most 16 dimensions.
- Changed: pg_sift: a shared CTE is planned inline when the scans cost 1.15 times less than the work of the shared plan they replace (it was 1.6), and when an ordinary plan still reads the table the inline plan may total up to 1.25 times the shared one (it had to total less than the shared plan by more than the scans cost). With the visibility map empty, 27 statements of a 195,000-row slice (14 of one SQL form, 13 of the other) take the scans instead of the shared CTE.
- Added: pg_sift: a semi or anti join (`IN (SELECT ..)`, `EXISTS`, `NOT EXISTS`) whose right side is a tree of joined tables (an inner join of small tables, or `IN (SELECT coalesce(a.k, b.k) FROM a LEFT JOIN b ..)` behind a table of its own, the shape PostgreSQL makes of nested sub-selects) is no longer refused as `from_shape`: the plan PostgreSQL made of the tree runs once when the scan starts and gives the key set, as the plan of a sub-select does (NULL and duplicate keys, an empty set and the anti join as in the normal plan; a tree that reads the query's other tables, or whose keys are an expression over two of its tables straight under the scan, keeps the normal plan). Test `agg_semi_tree`, fixtures `j_semi_tree*`, coverage row J35.

- Added: eleven more cases in `bench/cases.tsv` (a text sort key, sorted runs with and without a btree, the zones of the keys after the first, a long offset, five sort keys, the histogram of `count(DISTINCT)`, a numeric group key, `rollup_dims`, a dictionary equality and a prefix), `default+col` in the `index` field for a column the default index lacks, `{cols}` in a `setup` section, and the block names the variants (`runs_off`, `zone_off`, `forced`), the cases where the planner kept the normal plan ("same plan", no ratio) and the settings that bring no gain.
- Added: two cases in `bench/cases.tsv` for the scan of a partitioned table (hash partitions: 400, where the scan is kept, and 3,200, where the planner keeps the normal plan), and the block shows the time of pg_sift with the same plan as PostgreSQL's as a ratio when it is outside the 5% tie, instead of hiding it in "same plan".
- Added: ten more cases in `bench/cases.tsv` (a numeric sum, the whole-table total, a top-N through sorted runs with a filter, and on a second dataset, the events of the plan-choice gate (`test/plan/gen.sql`, built once per run in the schema `c_events`, `dataset` = `events` in the registry): a top-N through a join that only filters, a series joined on `jsonb_build_array`, the first-row lookups of a `LIMIT 1` child, a shared common table expression, a filter that keeps a few rows of every unit, and the elements of a list), and the block lists the losses first, then the ties and the plans kept, then the wins.
- The stage `scale` of `bench/bench refresh --stages scale --budget 0` measures it (opt-in, never in the default pass: 20 million orders at 100x of 200,000).
- Added: five more cases in `bench/cases.tsv` (six sort keys, 5,000 partitions of six rows, the dirty-document share at 3% and 50%, and three `UNION ALL` arms over the elements of a list above a spine, the shape the scan does not take yet), `bench/bench cases --plan` (the points, builds, disk and time of a run, nothing started; `bench/bench refresh --stages scale --dry-run` prints it), and the run drops the schema of each orders case after its measure so that a scale run does not keep a copy of the table per case. The events dataset of the plan gate is generated for its own schema, and the plan of the sift lane is taken after its first statement (a backend that has not loaded the library plans its first statement without the scan; the note says so).
- Fixed: `ALTER INDEX ... SET` of an option that names columns (`calendar_columns`, `key_columns`, `order_columns`, `rollup_dims`, `return_columns`) was not checked against the index in the first statement of a session that had not used sift yet, so a bad name, a column of the wrong type or a calendar list on an index with `paths` was stored and then failed every query, insert and `VACUUM` of the table until `RESET`. The check, and the analyzer warning, now run from an object access hook that PostgreSQL reads when the catalog row is updated, in every session.
- Added: pg_sift: `point_columns`: every segment written afterwards lists the rows of those columns by value (a table with a slot for each distinct key, a 32-bit fingerprint over the key's only row or the offset of the list of its rows, at the end of the dictionary section), and the access method finds the rows of an equality, an `IN` list or a short range with a probe and has the heap check them again, where the units that can hold the value are each read and tested. 200,000 rows, `SELECT id FROM t WHERE rnd = X` by a random integer, warm, median of runs 11 to 20: 0.119 ms without the option, 0.040 with `key_columns`, 0.020 with `point_columns`, 0.006 for a b-tree (an `IN` list of 5 keys 0.193, 0.059, 0.028, 0.009; the first query of a new backend 0.287, 0.142, 0.092, 0.037); a text equality once its dictionary is cached 0.032, 0.023, 0.016, 0.006. The price is size, 8 bytes a slot and 4 bytes a row of a key that repeats, so a key in very many rows never takes the leaf's table away: 200,000 rows, a directory on one leaf adds 4.21 MB for a unique key, 0.97 MB for 22 rows a key, 0.81 MB for 333 rows a key and 0.80 MB for 4,000 (a b-tree on the integer: 4.5, 1.6, 1.5, 1.4 MB). Keys with hundreds of rows gain little (333 rows a key, rows returned: 0.72 to 0.60 ms, b-tree 0.44) because the heap fetch of the rows is the cost; the access method scan runs only with `sift.enable_col_scan` on. `sift.col_verify` checks every row against its directory. An index without the option, or at `sift.col_format_minor` below 14, is read as before; `sift.upgrade_check` lists an index with the option and older segments as a gain from `REINDEX`. Tests `col_point_dir`.
- Added: pg_sift: `point_tid_columns`: the point directory of those columns lists the heap addresses of the rows instead of their numbers in the segment, so an equality or a short `IN` list of the access method returns the addresses without reading a unit, and the heap checks them again. A unit with a dead row is read instead (VACUUM frees an address to be used again), and so is every unit for an index-only scan, which wants the values. 1,000,000 rows, `SELECT * FROM t WHERE id = X`: 0.013 ms, and 0.011 to 0.013 ms at 26,700 and 100,000 rows (`point_columns`: 0.022, 0.015, 0.015; a btree 0.006), an `IN` list of 10 keys 0.022 ms (0.052; btree 0.013), a uuid 0.042 ms (0.078; btree 0.007). The size is that of `point_columns` within 3% (91.4 MB against 93.7 MB for five columns, four of them unique; a key of many rows takes 1 to 3 bytes a row, not 4), the build 65% longer, a promotion of the write buffer 5% longer and the inserts unchanged. The default of `sift.col_format_minor` stays 15 and only a segment with the option is minor 16; an index written at 14 or less lists rows, and an index written before the option is read as before until a `REINDEX` (`sift.upgrade_commands()` says so).
- Changed: pg_sift: a `sift_table` row fetch (a SELECT or UPDATE of a few rows, the heap fetch of an index scan) reads the text of a row from a segment whose dictionary is stored in blocks to be read in place (column format minor 14) one string at a time, where it loaded the whole dictionary of the segment first: the first statement of a new backend on a table of 200,000 rows with unique 104-byte text takes 1.4 ms, where it took 28.6 ms (median of 3 interleaved runs, 10 segments of 20,000 rows). After about a sixteenth of the dictionary has been read that way the dictionary is loaded and cached as before, so a scan that reads many strings is not slower.
- Added: pg_sift: a page of rows with the total of its set, `SELECT cols, count(*) OVER () FROM t WHERE .. ORDER BY k LIMIT n OFFSET m`, is a top-N of the column scan (`sift_col` and `sift_table`): the window functions that are aggregates over the whole set (`count`, `sum`, `avg`, `min`, `max` and the other aggregates the scan computes, with `FILTER`; no `PARTITION BY`, no `ORDER BY`, a frame of all the rows) are computed from every row that passes in the same pass as the heap of the n best, and every row of the page carries them (no unit is left out for the zone of the sort key, and the scan is serial). 200,000 rows, a filter that keeps 20,233 of them, `count(*) OVER (), sum(amount) OVER ()`, the planner's own choice, normal plan against the scan: `LIMIT 10` 47.1 ms against 2.7, `LIMIT 100` 53.3 against 3.9, `LIMIT 1000` 50.1 against 5.2; a filter that keeps a few hundred rows keeps the normal plan, which is priced cheaper. A window with `PARTITION BY`, `ORDER BY` or a frame that cuts the set, and a window function that is not an aggregate (`row_number`, `rank`, `first_value`), keep the normal plan under `topn_shape`, with the reason. Test: `test/sql/agg_topn_window.sql`.
- Added: pg_sift: `ORDER BY <expression of the columns> LIMIT n` is a top-N of the column scan of a `sift_col` or `sift_table` index, with any direction and NULLs clause, alone or among other keys. Until now any sort expression other than a column, `coalesce(column, constant)`, `lower(column)` and `upper(column)` kept the plan's sort of every matching row and lost the sift filtering with it. The scan runs PostgreSQL's own expression evaluator for every row that passes the `WHERE`, as the plan's sort does, so the operators, casts, functions, overflow, division by zero, NULLs, NaN and the collation of strings are the backend's, a query that raises raises the same SQLSTATE in both plans, and a `CASE` branch or a later `coalesce` argument that is not taken is not evaluated. The expressions are arithmetic (`a + 1`, `1000 - a`, `-a`, `a * b`, `total / 3`, `a % 7`), casts (`a::bigint`, `a::text`), `CASE` (searched or simple), `coalesce` of several columns, `NULLIF`, `GREATEST`/`LEAST`, boolean expressions (`AND`, `OR`, `NOT`, `IS NULL`, `IS TRUE`, `IS DISTINCT FROM`), `IN (..)` of constants, comparisons, and any immutable function or operator: concatenations (`lower(s || v)`), functions of functions (`upper(lower(s))`), `left`, `substr`, `initcap`, `md5`, `abs`, `mod`, `round`, `sqrt`, `date_trunc`, `extract`, `make_date`, `length(b)` of a bytea, `host(ip)` of an inet, a plpgsql function declared `IMMUTABLE`; the result is a number, a date, a time, a string, a uuid, a bytea or an inet. No zone bounds such a key, so the scan reads every unit that holds matching rows, and the planner prices the evaluation per row (the expression's cost, a charge per leaf, per numeric and per string) so that it keeps the plan's sort where the scan would be slower. A parameter of the statement (`ORDER BY a + $1` in a generic plan) and a function that is not `PARALLEL SAFE` keep the scan in the leader. A volatile or stable expression (`random()`, `now()`, `concat()`, `timestamptz + interval`, the text of a date), a sub-select, an interval, array or jsonb result and a numeric column of 19 or more digits keep the normal plan under `topn_shape` with the reason.
- Fixed: pg_sift: a common table expression that several aggregates read, and that selects from another CTE of the same `WITH` instead of from the indexed table, kept PostgreSQL's materialization: the check for a shared CTE over a column index looked at the tables of its own FROM list only. It follows the CTEs it reads now (a chain of up to four), so the inline plan is weighed against the shared one as for a CTE that reads the table (2 arms over 30,000 rows: the arms are scans, the same rows as the normal plan; an explicit `MATERIALIZED` still keeps the copy).
- Changed: pg_sift: `sift.table_auto_defrag_dead_pct` is 5 (was 20): the background pass of a `sift_table` starts when 5% of its rows are tombstones, not 20%, so a table that took 16% churn (the update, update and delete phases of the store benchmark) is cleaned before the next read instead of waiting for a `VACUUM`. 200,000 rows, the read after the churn 35.8 ms -> 7.6 ms (0.5 ms fresh); four clients of an update, insert and delete mix, 160,000 transactions, medians of 3, pass on: 3,206 -> 3,452 transactions a second, WAL 164 -> 157 MB, range read p99 45 -> 38 ms, all-client stalls of 20 ms or more 295 -> 187 (pass off: 2,217, 95 MB, 60 ms, 666).
- Changed: pg_sift: a backend reads the tombstone log of a `sift_table` at its first statement 2.2 times faster: a stable radix sort on the key replaces `qsort` with a comparison call (the batch is in log order, so it is the sort on key and position), the first batch goes into the main array and not through the recent one and a second copy, and the batch is allocated once from the count in the log header. A table with 32,400 tombstones (16% of 200,000 rows after the update, update and delete phases of the store benchmark): the sync takes 5.1 ms (0.96 read, 3.67 sort and merge) and then 2.3 ms (0.8, 1.5), measured inside the backend on a loaded host; it was 88% of the 5.8 ms a new session needed for a one-row `UPDATE`. The background pass asks the log header for the number of entries, and reads and sorts the log only when that number could cross `sift.table_auto_defrag_dead_pct`: a backend that ticks without reading the table no longer syncs it.
- Added: pg_sift: `sift.col_auto_merge` (off, 3.2) also vacuums the table, [docs/USAGE.md](docs/USAGE.md): the merge only helps after a `VACUUM`, so one setting keeps the table answerable. After churn spread over a table every heap page is not all-visible, so a scan of a `sift_col` index checks its rows in the heap one by one and answers none of its units from their pre-aggregates, until a `VACUUM` that autovacuum starts after 20% of the rows changed. A backend that inserts a row version, or whose scan checked 1,000 rows or more, now looks at the statistics of the table once per `sift.maintenance_naptime` and starts a worker when 0.1% of the rows changed (100,000 rows at most, 8.5 ms of rows a query checks in the heap, where autovacuum waits for 20%: 20 million of 100 million); it vacuums as autovacuum does, at most once per five naptimes (ten times the average `VACUUM` of the table on PostgreSQL 18), stays until the table is clean and leaves a table with `autovacuum_enabled = false` alone. 200,000 rows, the update, update and delete phases of the store benchmark, the interval scaled to one second, medians of 3: the monthly revenue read after the churn 18.7 ms -> 1.2 ms (0.6 ms with `sift.col_auto_merge`; 0.2 ms fresh), the second query 11.3 -> 1.9 ms; the rows a second of the three phases are the same, and the WAL of the churn rises from 11.7 to 16-17 MB by what the `VACUUM` after it wrote (16.8 MB in all before).
- Changed: pg_sift: the automatic rewrite of `sift.col_auto_merge` waits while a snapshot holds back the blocks the last rewrite freed (half the index or more): it would answer no more units and leave as much again behind. Under a snapshot held through 16 rounds of churn the index no longer outgrows the one without the setting.
- Changed: pg_sift: the calendar cells of the dates come first in the default cube of a `sift_col` index with more than 16 columns that can be dimensions: the plain columns filled its 16 dimensions in their order and left the cells out, so `GROUP BY date_trunc('month', d)` scanned every unit (21 columns, 60,000 rows: the month of a date, the month of another, a window of years and the day 60,000, 60,000, 29,280 and 60,000 rows scanned before, 0 after). The default dims give way, the dates' own histograms first and then the last columns, and a NOTICE names the columns left out; nothing changes when everything fits, or for a `cube(...)` written by the user. An index of more than 16 columns takes the new choice at its next `REINDEX`.
- Changed: pg_sift: a `sift_col` index with a `jsonb` column and no `paths` option takes the paths from a sample of the documents at `CREATE INDEX` (`sift.suggest_paths`, 2,000 documents from 16 evenly spaced stretches of blocks): the spec is written to the option as `auto: spec`, so a `REINDEX` and the workers of the build keep the same schema, and a NOTICE names it. It used to be an error ("the jsonb column needs paths"), and it still is for a table where no key qualifies; paths written by hand are untouched. `col_paths_auto`: the index verifies, the grouped and filtered queries on it equal the plain plan's.
- Fixed: pg_sift: `ALTER INDEX ... SET (paths = 'auto')` and `RESET (paths)` on a `sift_col` index with a `jsonb` column were refused ("needs paths"), so the paths chosen at `CREATE INDEX` could not be read again from the documents; the next `REINDEX` decides again now (`col_paths_auto`).
- Added: pg_sift: `calendar_columns` writes the calendar cells of the dates of an index with `jsonb` paths (3.3 refused them; not the default there: +9.3% to +12.1% of the main index of an application), and a unit whose date has too many days for a histogram answers a comparison, an AND or an OR of comparisons on it from the cells, each cell decided by its span, [docs/COL-PREAGG.md](docs/COL-PREAGG.md) (`agg_cal_paths`).
- Added: pg_sift: `sift.suggest_paths(table, column, sample)`, [docs/REFERENCE.md](docs/REFERENCE.md): the `paths` option that a `jsonb` column asks for, read from 2,000 of its documents (16 stretches of blocks evenly spaced over the table, so the same table gives the same answer; advice only, nothing is changed): a typed path for each key that is a scalar of one type in most of the documents that have it (`int4`, `int8` beyond 32 bits, `float8` for numbers with fractions, `text`, `bool`, `astext` for a mix), a wildcard for a map of more than 32 keys of one type, a group for an array of objects; rare keys and keys without a single shape are left out. `col_suggest_paths`: the spec is accepted by the option, the index built with it verifies and the grouped, filtered and unnested queries on it are pushed and equal the plain plan's.
- Added: pg_sift: `sift.index_advice(index)`, [docs/REFERENCE.md](docs/REFERENCE.md): the profile of the columns of a `sift_col` index and what its options could still use, advice only (nothing is changed): per column its kind, what the index holds of it and, over 32 evenly spaced units, the share that satisfy the rule of a dimension (at most 256 distinct values: a histogram of the cube); per pair of dimensions the share of units whose combinations fit a rollup of the rows, with `rollup_dims` advised when that is 90% of the units for within 3% more index. The same rows after a `REINDEX`.
- Changed: pg_sift: the automatic rewrite of `sift.col_auto_merge` no longer skips an index over `sift.vacuum_rewrite_limit` (128 MB, about 6 million rows, above which the merge did nothing): it rewrites it by partitions of that size, one step each with the lock let go between them (`sift.col_auto_merge_partition_kb` sets another size). 1,000,000 rows with the limit at 8 MB, a rewrite every round: the units answerable from their pre-aggregates 42% -> 100% (uniform churn, grouped sum 7.4 -> 0.10 ms) and 71% -> 100% (scraper pattern, 8.4 -> 0.16 ms), for the WAL of the whole-index rewrite.
- Added: pg_sift: `sift.col_maintenance_status(index)`, [docs/REFERENCE.md](docs/REFERENCE.md): where the chain that keeps a `sift_col` index answerable stands (the table, its last `VACUUM`, the rows changed since and the number that starts the next, why the rewrite is due, the last request of a worker) and `blocked_by`, what keeps a worker from running: the setting or `sift.async_maintenance` off, `autovacuum_enabled = false`, a standby, a snapshot holding back the last rewrite.
- Added: pg_sift: `order_columns` (`sift_col` reloption, default none; `int2`, `int4`, `int8`, `numeric`, `date`, `timestamp`, `timestamptz`, `float4` and `float8` columns) and `sift.topn_runs` (default on): every unit written afterwards keeps the ordinals of its rows in the order of the column (2 bytes a row for a column that does not ascend with the table, 16 bytes a unit for one that does: +50% on an index of ten columns, eight of them unordered), and `ORDER BY col LIMIT n` reads the first rows of each unit and not all of them (EXPLAIN shows `Units Read By Sorted Run`). 1,000,000 rows (230 units), `ORDER BY k LIMIT n` on a random `int4`: n = 10, 0.25 to 0.34 ms before, 0.17 to 0.22 after; n = 100, 1.7 to 1.9 before, 0.9 to 1.3 after; n = 1,000, 6.5 to 8.9 before, 1.9 to 4.9 after (a btree needs 0.35); n = 10,000, 29 to 32 before, 23 to 27 after. A scan with a `WHERE` that removes rows, and the rows of a write buffer, are read as before; the rows and their ties are the same as the plain plan's (240 shapes checked three ways). Column format minor 13 (11 is the write buffer of a `sift_table` with a wide row, 12 the superunits), written only for an index with the option under `sift.col_format_minor = 13` (the default; 3, 4, 6, 7, 8, 9, 10 and 12 keep the formats their libraries read): an index without the option writes the bytes it wrote before. A 3.2 library refuses a minor 13 segment with the REINDEX hint. `sift.upgrade_commands()` lists an index with the option and older units as a gain from `REINDEX`.
- Changed: pg_sift: a top-N scan with `order_columns` reads the sorted run of a unit also when a `WHERE` leaves few of its rows (it read them one by one before) and walks a start such as `k > $last` testing the row before it reads its key. 1,000,000 rows, `ORDER BY k LIMIT n`, before and after, median of 4 rounds of 15: a filter of 10% of the rows 1.50 to 1.32 ms at n = 100 and 2.53 to 2.16 ms at n = 1,000; `k > 500000` 5.18 to 1.96 ms at n = 100 and 5.71 to 2.54 ms at n = 1,000; no filter, 50% and a late start unchanged, 1% of the rows 1.44 to 1.58 ms at n = 1,000. No format change; the rows and their ties are the plain plan's (160 more shapes of `col_sorted_run`).
- Changed: pg_sift: the planner prices the bitmap chunks of a `count(DISTINCT)` per group (a chunk of 4096 values is allocated and zeroed when a value of the group first falls in it, about 0.7 microseconds), while they fit the memory of the set. A query with thousands of groups and a column of a few thousand values (200,000 rows, 4,907 groups, `LIMIT 100`: sift 9.4 ms, the btree plan 5.2 ms) took the scan, for the `count(DISTINCT)` was priced like `count(*)`; it keeps the normal plan now, and a query with a hundred groups, or a column whose sets leave the bitmap for a hash set, is priced as before.
- Changed: pg_sift: the planner charges the heap check of the rows on pages that are not all-visible for the pages of the units the zones leave, not for all the pages of the table: after `CLUSTER`, `VACUUM FULL` or `pg_repack` (an empty visibility map) the scan of a clustered table is no longer priced as if it read the whole heap. Tables whose map is full price the same as before. On a 195,000-row slice clustered by its key and not yet vacuumed, 87 of the 301 corpus queries plan on the scan (37 before; 238 after the `VACUUM`), and the 50 gained run in 7.8 s against 19.1 s for the normal plan. The heap check itself costs more per page than the model charges (15 to 20 us on a narrow-row table), so a narrow table with a btree on the clustered key can still choose a slower scan than its index plan until a VACUUM.
- Added: pg_sift: a `cost` refusal (`sift.explain_why`, the notice of `sift.explain_refusals`) that the heap check of the pages that are not all-visible decided says how many pages and what they add to the scan, and to run `VACUUM (ANALYZE)`: a rewrite by `CLUSTER`, `VACUUM FULL` or `pg_repack` leaves the visibility map empty, and the normal plan wins until a VACUUM. No plan changes.
- Added: pg_sift: a `cte_cost` refusal that the heap check of the pages that are not all-visible decided (the inlined scans would have won without it, and the shared plan does not pay it too) says how many pages and what they add to the scans, and to run `VACUUM (ANALYZE)`, as the `cost` refusal does. No plan changes.
- Fixed: pg_sift: docs: `CLUSTER`, `VACUUM FULL` and `pg_repack` leave the visibility map of the table empty; run `VACUUM (ANALYZE)` after them, or the scan of a `sift_col` index is priced for a heap check of every row and the normal plan wins (200,000 rows, a scan of 133,660 rows: 320 ms against 82 ms when forced; a table of wide rows: 180 ms against 7.7 ms for the scan, the normal plan 160 ms). The text said no `ANALYZE` was needed.
- Added: pg_sift: `ORDER BY coalesce(<int2, int4, int8, date, timestamp or timestamptz column>, <constant of that type>)`, `lower(<text or varchar column>)` and `upper(<text or varchar column>)` `LIMIT n` is a top-N of the column scan, with any direction and NULLs clause and in the sort of every row that passes: none of them can raise, so the scan that evaluates the key for every row that passes returns the rows the normal plan does, in its order (a NULL is the constant, with its zone widened to it so units are skipped as for the column; the string goes through PostgreSQL's own `lower()` / `upper()` under the call's collation once per distinct entry of a unit). 200,000 rows, `LIMIT 10`, the planner's own choice, normal plan against the scan: `coalesce(int, 0) DESC` 66.7 ms against 0.6, `coalesce(id, 0)` 69.1 against 0.6, `lower(text)` 67.5 against 34.2, `upper(text)` DESC 73.0 against 35.9, `lower(text)` with a filter 32.0 against 11.2; a locale collation (`lower(name COLLATE "en_US.utf8")`) is priced above the normal plan and stays with it. Arithmetic, casts, `CASE`, `coalesce` of two columns and other functions keep the normal plan under `topn_shape` (they can raise for a row far from the top, which only the normal plan reaches). A locale collation is now told from `C` by the collation's own flag, so a database created with `C` prices a text key at the byte order. Tests: `test/sql/col_topn_expr.sql`, the `topn_xf_*` coldiff fixtures, the `topn` family of the differential fuzzer.
- Changed: pg_sift: a `WHERE` over the columns of the nullable side of a preserved join (`calendar LEFT JOIN rows`, the filter PostgreSQL pushes into a grouped sub-select from the caller's `WHERE category = 'x'`) keeps the aggregate scan when the row of NULL columns passes it: the scan tests it on the matched tuples, so a calendar day whose matches it drops is no row and an empty day is. It was refused as `outer_join`, and a client had to fence the sub-select with `OFFSET 0`. A condition the NULL row fails keeps the normal plan. The `from_shape` refusal names the join or table, and the other orientation of a preserved join replaces the refusal of the first. Test `agg_preserved_where`.
- Changed: pg_sift: the two-level scan of arms joined to a spine takes `GROUPING SETS`, `ROLLUP` and `CUBE` (it kept the normal plan): each cell is added to the group of every spine row its keys equal in every set, and a `LEFT JOIN` spine row no cell matched is a group of NULL arm columns in every set. The `spine_shape`, `two_keys` and `two_aggs` refusals say which condition, key or aggregate. Test `agg_two_level_spine_sets`.
- Changed: pg_sift: the choice between a shared common table expression and the inline plan compares the scans with the plans of the CTEs and the aggregates that read them (a margin of 1.6), not the totals of the two plans: those differ by the plan above the scans, which each plan prices from the rows it expects of the arms (the shared CTE one row and 100 per unnest, the scans a group count, a join on an expression key the default selectivity). Three `UNION ALL` arms over the elements of a jsonb array, counted `DISTINCT` above a spine, 100,000 rows: 2.8 s on the shared CTE, 0.9 s in the scan, which the totals refused (PostgreSQL 18, 2026-10-22, loaded host). When a reader of the CTE stays an ordinary aggregate the totals still decide, and an inline plan priced 20 times the shared one is refused. No format change. Tests `agg_cte_inline_rows` (100,000 rows now), `agg_plan_invalidation`.
- Fixed: pg_sift: a sum over a `numeric` column without a precision was priced a tenth of the parallel sequential scan and ran four times slower in the scan (200,000 rows in 19 units: 137 ms against 34 ms): each unit builds a dictionary whose every new entry is spelled as text and read back by `numeric_in` (0.75 us), and the estimate charged a share of the rows equal to the column's distinct fraction. The scan is now priced for the entries each unit builds (300 operators each for such a numeric); a column of few values keeps the scan (10 ms against 22 ms). The same price made a clustered heap with a btree choose the scan for a key list. No format change. Test `agg_wide_numeric_cost`.
- Added: pg_sift: `sift.cluster_advice(rel, min_rows)` recommends the physical order (`CLUSTER`, `pg_repack --order-by`) that lets the scans of a `sift_col` index skip the most units, by a virtual sort of the zones: the key, the units read now and after, the rows saved, the churn, a verdict with the reason (small table, already ordered, a competing ordered leaf, rows changing too fast) and the `pg_repack` call with the `VACUUM (ANALYZE)` that follows a rewrite; the argument `key` asks about an order of your own (`'a, b'`). It is optional at two levels, the setting `sift.cluster_advice` (on) and the `sift_col` index option `cluster_advice` (on); neither is part of the format.
- Added: pg_sift: `sift.cluster_advice(rel, min_rows)` recommends the physical order (`CLUSTER`, `pg_repack --order-by`) that lets the scans of a `sift_col` index skip the most units, by a virtual sort of the zones: the key, the units read now and after, the rows saved, the churn, a verdict with the reason (small table, already ordered, a competing ordered leaf, rows changing too fast) and the `pg_repack` call. It is optional at two levels, the setting `sift.cluster_advice` (on) and the `sift_col` index option `cluster_advice` (on); neither is part of the format.
- Changed: pg_sift: the arms of a two-level scan may differ in their `WHERE`: a condition that every arm has is the scan's, one that only some arms have is a filter of those arms (a `NOT NULL` test of a lookup in two arms and none in a third), and a constant in the `GROUP BY` of the arms splits no group. The three-arm shape of an aggregate over a `LEFT JOIN LATERAL .. LIMIT 1` lookup, which the normal plan ran, is one scan of the index. The `two_arms` and `two_keys` refusals name the arm and the condition.
- Added: pg_sift: `ORDER BY` a text, varchar, uuid, bytea or inet column `LIMIT n` is a top-N of the column scan. The entries of the unit's dictionary are the keys, compared by the sort operator's own comparison (`varstr_cmp` under the query's collation, `uuid_cmp`, `byteacmp`, the inet comparison), so any deterministic collation, `C` or a locale of the C library or ICU, returns the rows of the normal plan; a nondeterministic collation, `char(n)`, `name`, `cidr` and `text_pattern_ops` keep the normal plan (`topn_shape`, with the reason). The keys have no zones, so every unit that holds matching rows is read; the heap keeps its own copy of the strings (a heap of keys wider than `work_mem`, or more rows than it holds, sorts every row in a tuplesort under the same operator and collation), and workers send their strings to the leader. The table source reads a wide leaf (uuid, bytea, inet) of a sort key after the first. 200,000 rows, `LIMIT 10`, serial, the planner's choice, median of 3 (PostgreSQL 18.6, a host at load 50), normal plan against the scan: text 52 against 17 ms, text with a filter 46 against 14, text first key of 4 values 61 against 8, `en_US.utf8` 79 against 25, uuid 59 against 24, inet DESC 71 against 30, bytea 66 against 29, integer then text 71 against 13. Tests `col_topn_text`, `topn_key_text*`, `topn_key_uuid*`, `topn_key_bytea*`, `topn_key_inet*`.
- Fixed: pg_sift: `pg_repack` of a `sift_table` destroyed the table (it swaps in a heap copy of the rows, and the next read failed with "invalid page layout"): the new event trigger `sift_table_repack_guard` refuses the `repack.repack_trigger` of `pg_repack` on a `sift_table` before anything is copied, with a hint to use `CLUSTER` or `VACUUM FULL`; `pg_repack` of a heap table with `sift` and `sift_col` indexes, and `--only-indexes` of a `sift_table`, are unchanged.
- Fixed: pg_sift: a dictionary of a string leaf is kept by the backend up to half of `dict_cache_kb` (64 MiB), where one over a quarter was parsed again by every statement that read a row of its segment: the dictionary of a free text leaf of 150,000 rows is 18 MB, and four clients of an update, insert and delete mix spent most of their time inflating it (200,000 rows, `sift_table`, 30 s: 1,073 transactions a second, 5,092 after).
- Changed: pg_sift: `sift.table_auto_defrag` is on by default (3.2: off; it needs `sift.async_maintenance`, on): a `sift_table` that deletes, updates or reads past `sift.table_auto_defrag_dead_pct` dead rows, or holds `sift.table_auto_defrag_free_pct` free blocks, is cleaned by a background pass, without `VACUUM`. Four clients of an update, insert and delete mix, 30 s, medians of 3, pass on against off: 25,000 rows 16,672 against 3,427 transactions a second, range read p99 3.4 against 10.7 ms; 200,000 rows 7,536 against 5,092, read p99 16.8 against 34.2 ms, file 10.7 against 18.1 MB; 1,000,000 rows no pass starts in the 30 s and the rate is the same. The pass adds WAL on that mix (200,000 rows, same transactions: about 60% more than without it, 237 MB against 142-154 MB). A table that only appends starts no pass (200,000 inserts a second, 25,000 rows, 3 rounds: 0.352 kB of WAL per transaction with the pass on, 0.353 off, no worker, 188,000-231,000 transactions a second against 169,000-220,000; the first trigger gave 0.41 kB and 8 to 9 workers in 30 s): the free blocks of a table that is being written are its promoted write buffers, about to be taken again, so the space is asked for only when no write buffer was opened since the backend last looked. A worker whose first pass found nothing leaves at once. `sift.table_auto_defrag = off` (server or database) restores the 3.2 behaviour; the test clusters of `scripts/pgtest.sh` run with it off, as without autovacuum.
- Compatibility: pg_sift: `sift.col_format_minor` defaults to 15 in 3.3 (3.2: 6). A database written by the 3.3 library with its defaults can no longer be read by a 3.2 library: a new `sift_col` index carries `cube_minmax` and `superunit` (minors 7 and 12) and the calendar cells of its date and timestamp columns (8), a string dictionary of 1,024 strings or more of a column named in `key_columns`, `point_columns` or `return_columns` (or of any column with `dict_lookup = on`) is stored for a lookup (14), a `sift_table` with a date or timestamp column holds calendar cells (8), and `rollup_dims` and `order_columns` add 9 and 13, `key_columns` 10 and 15, and `point_tid_columns` 16 (only for an index with the option). To go back, run the sequence of docs/UPGRADING.md section 4 (`sift.col_format_minor = 6`, REINDEX the indexes, VACUUM FULL the tables and DISABLE the two event triggers that `upgrade-check.sh --to 3.2` lists); to stay readable from the start, create the index with `cube_minmax = false, superunit = 0, calendar_columns = 'off'`, or set `sift.col_format_minor = 6` before it is written. `ALTER EXTENSION pg_sift UPDATE` warns of it, and `sift.upgrade_check('3.2')` lists what a rollback cannot read. `test/t/compat_matrix.sh` runs the refusals of 3.0.2, 3.1.0 and 3.2.0 for every minor and the downgrade to 3.2.0 end to end.
- Changed: pg_sift: `cube_minmax` and `superunit` are on by default for a `sift_col` index (`cube_minmax = true`, `superunit = 16`; `WITH (cube_minmax = false, superunit = 0)` keeps the 3.2 bytes). 200,000 rows of 13 columns, 53 units, minimum of 15 runs: `GROUP BY` with `min`/`max` of an integer 5.5 -> 0.34 ms (0.058 ms with the groups), a whole-table `count`/`sum` 0.26-0.37 -> 0.05-0.08 ms, index +1.5% (+0.17% and +1.0%), insert, promotion, merge and VACUUM CPU inside the noise; a range query on a table of fewer than 16 units costs about 40 microseconds more. A new index is column format 12, which a 3.2 library refuses (`SET sift.col_format_minor = 6` before it is created, or the two options off); an index built before keeps its units and `sift.upgrade_commands()` lists the REINDEX that gives them the sections. See docs/UPGRADING.md "Defaults changed in 3.3".
- Changed: pg_sift: `calendar_columns` of a `sift_col` index defaults to every date, timestamp and timestamptz column (`'off'` keeps the 3.2 bytes; not for an index with JSON paths, `preagg = 'off'` or a `cube(...)` given dim by dim), as a `sift_table` has always had them: 200,000 rows with a `date` and a `timestamptz`, `GROUP BY date_trunc('month', d)` 2.4 -> 0.54 ms, the UTC month of the `timestamptz` 2.6 -> 0.70 ms, the sidecar +1.8% of the index. A superunit no longer merges the calendar cells of its units (+1.2% of the index and no query faster). Column format 8.
- Changed: pg_sift: an index scan of `sift_col` that selects a few rows of a unit (a point lookup, an `IN` of a few values) decodes the TIDs of those rows only (the pages that hold them), not of all the 4,352 rows of the unit: a random-int equality on 1,000,000 rows takes 0.62 ms (was 1.10) and 0.21 ms with `key_columns` (was 0.39), the instructions of the scan half. No format change.
- Fixed: pg_sift: a delete or an update of a `sift_table` no longer waits for the background pass: the lock a writer takes to append a tombstone and the one the pass holds for its whole length were the same page lock (stalls of up to a quarter of a second at a 10% threshold under sustained churn).
- Added: pg_sift: `sift.table_auto_defrag_max_active` (0, never waits) and `sift.table_auto_defrag_wal_rate` (kB of WAL a second, 0 for no limit): the background pass of a `sift_table` waits for a quieter moment, between its merges as well, and keeps to a WAL rate; `sift.col_auto_merge_max_active` shares the wait. The pass is asked for only when tombstones older than every snapshot reach the threshold, where a long transaction made it read the table to remove nothing, and it merges the small groups next to the one it rewrites (`sift.table_auto_defrag_coalesce`).
- Changed: pg_sift: the table `VACUUM` merges until it finds nothing left to merge, where it stopped after 16 merges; a background pass still stops at `sift.table_auto_defrag_budget`.
- Added: pg_sift: `key_columns` takes uuid columns (it refused them: "not a plain integer or text column"): a unit keeps the set of the uuid's dictionary codes like a text column's, in the same minor 10 section (no format change; a library that knows minor 10 reads it). 30,000 rows in 24 units, a point lookup of a uuid scattered over the table reads 5 units, not 21.
- Changed: pg_sift: an equality or a `LIKE 'abc%'` / `^@` on a text or uuid column of a `sift_col` index finds its dictionary codes without testing every string: the first lookups in a dictionary of 1,024 strings or more scan it, the next ones use a hash table (equality) and a byte-order permutation of the dictionary (prefix, a bisection to the range of strings, the work proportional to the matches), both built lazily on the parsed dictionary the backend keeps. 1,000,000 rows, a prefix count: 15 to 20 ms before, 5.2 ms warm; the first query of a new backend (it parses the dictionary, still a cost proportional to the strings) uuid equality 192 to 223 ms before, 26 to 39 ms. The dictionary cache limit is `dict_cache_kb` in the registry (`sift.test_limits`, docs/LIMITS.md), replacing the testing GUC. No format change. `bench/col_scan.py` measures it, docs/COL-SCAN.md shows the numbers. Tests `col_dict_lookup`, `col_dict_cache`.
- Changed: pg_sift: a dictionary of 1,024 strings or more of a text, varchar or uuid column of a `sift_col` index or a `sift_table` is stored in blocks of 64 strings that a reader decompresses one at a time, where that adds at most 3% to the segment, and with the codes in byte order too where a lookup structure reads it. The row fetch of a `sift_table` and the strings an index-only scan returns no longer load the dictionary: the first fetch of a new backend on 200,000 rows with a unique 110-byte text takes 1.0 ms, where it took 47.4 ms (a page of 50 rows 1.2 ms, where 58.7 ms), and a backend whose cache cannot hold the parsed dictionary (23 MB; a segment of 700,000 such rows is over the 32 MB one entry may take) no longer parses it again for every unit it reads (89 to 96 ms each at a 16 MB cache); the dictionary costs 8 kB of the backend instead of 23 MB. The blocks tie with the compact form for ids, hashes, uuids and long strings (index of 200,000 uuids +0.8%, of unique 37-byte text +2.2%, of unique 110-byte text -1.9%) and lose to it for short strings that repeat words (titles and names of real text: +19% to +30% of the index of one column), which is why the writer estimates the extra from 16 blocks and the start of the dictionary and keeps the compact form for those (index growth 0.0% on every column measured above 3%). The order for an equality, an `IN` list, a prefix (`LIKE 'abc%'`, `^@`) or a range test costs 4 bytes a string (+10% to +70% of an index of one column): the option `dict_lookup` is `auto` by default and writes it for the columns of `point_columns`, `key_columns` and `return_columns`; `on` writes it for every column and `off` for none, and a `sift_table` writes it under `sift.table_dict_lookup` (default off). The first lookup of a new backend with the order: uuid equality on 300,000 rows 12.1 ms before, 0.43 ms; text equality 29.3 ms before, 0.54 ms; text prefix 36.7 ms before, 6.5 ms; a dictionary tested more than 32 times by a backend is parsed and cached as before. A 3.2 library refuses a segment written in the new forms (REINDEX hint) unless `sift.col_format_minor` was lowered to 6 first; the segments already written are read as before and are not rewritten. The zstd contexts of the column codec are reused between calls (the bytes are the same). The limit `dict_lookup_min` (registry, `sift.test_limits`) is the smallest dictionary that gets a form. Tests `col_dict_stored`, `tbl_fetch_dict`, `col_enc`.
- Added: pg_sift: the reloption `return_columns = 's, u'` of a `sift_col` index makes the named text, varchar and uuid columns returnable, so a query that needs only the index's columns, strings included, is an Index Only Scan (the strings of a few rows are read in place out of a dictionary stored for a lookup or plain, without parsing it; a value the index could not keep comes from the heap). Off by default: with the heap in memory the Index Scan is cheaper (300,000 rows, 100 rows of `id, u, s` 0.055 ms against 0.155 ms; a `SELECT id` 0.066 against 0.158 ms, because the executor asks for every returnable column), so it is meant for tables larger than memory. No format change. Test `col_index_return_str` (all three dictionary forms, overflow values, NULL, empty, long and non-ASCII strings, write buffer, DML, promotion, merge), `col_scan` runs with the option on.
- Changed: pg_sift: the plan of a top-N that sorts every row that passes (more than 4 keys, or more than 100,000 rows to skip) charges what the sort reads and spills: each key past the first is read at every row (4.8 operators a row, 11 ms for 200,000 rows) and a sort past `hash_mem` pays 4 operators a step, not 1 (172 ms more for 200,000 rows and 4 keys, 2.6 times what the plan said). Six keys on 200,000 rows ran 200 to 350 ms in the scan against 35 to 80 ms in the normal plan and were avoided only because the plan priced the sort at the normal plan's cost; a deep offset on 2 or 3 keys keeps the scan (59 / 65 ms against 124 / 184), on 4 or more keys with a spill the normal plan wins (212 to 264 ms against 192 to 210). The plan gate (96 shapes) is unchanged. Test `col_topn_deep` (`six_keys_wide`).
- Changed: pg_sift: the UNION ALL arms of a two-level or fused scan go from 4 to 16 and the virtual leaves of a scan (expressions on a jsonb path with a key) from 8 to 16; one more of either keeps the normal plan under the slug `shape_width`, and `sift.explain_why` says which limit it passed. The limits of the arms, virtual leaves, columns of a joined table read by expressions and aggregates of a rows set are read through the registry, so `sift.test_limits` lowers them. Tests `agg_shape_widths` (each width at N and N + 1, scan against the normal plan), `agg_lookup_budget`.
- Changed: pg_sift: the member of an element of a list that a `sift_col` index declares typed (`doc.parts[]{score float8}`), read as `(p ->> 'score')::float8` over `jsonb_array_elements(doc -> 'parts') p`, keeps the aggregate scan when some elements hold a value the leaf cannot store: the scan reads those rows again from the heap, converts a numeric string as the cast does, and raises the error of a value SQL cannot convert for a row that keeps the element, as the normal plan does. A condition on the elements, which could drop the element that raises and keep another (the scan keeps one error for a row), keeps the normal plan under `dirty_units`. Tests `agg_scalar_dirty_elems`, `agg_scalar_dirty_share`, `agg_scalar_overflow` and the `vd_scalar_elems_*` fixtures of the differential suite.
- Changed: pg_sift: the `FILTER` of an aggregate that casts a typed scalar path of a `jsonb` column (`count(*) FILTER (WHERE (doc ->> 'n')::int4 = 3)`) keeps the aggregate scan when some documents hold a value the leaf cannot store, as does a cast in the argument of an aggregate with a `FILTER` (`sum((doc ->> 'v')::float8) FILTER (WHERE grp = 1)`). The scan reads those rows again from the heap and raises the error of a value SQL cannot convert for the rows the aggregate evaluates the cast for: every row that reaches it for the cast in a `FILTER`, the rows the `FILTER` keeps for the cast in an argument, and none that the `WHERE` or the `FILTER` drops. Before, any such row kept the query on the normal plan (`dirty_units`). Tests `agg_scalar_dirty_filter`, `agg_scalar_dirty_share`, `agg_scalar_overflow` and the `vd_scalar_filter_*` fixtures of the differential suite.
- Changed: pg_sift: a condition of the `WHERE` that casts a typed scalar path of a `jsonb` column (`WHERE (doc ->> 'v')::float8 > 3`, a comparison with a constant, an IN list of integers or a NULL test, for `float8`, `int4`, `int8` and `bool` paths, or the cast of `doc -> 'v'` for `float8`) keeps the aggregate scan when some documents hold a value the leaf cannot store: the scan reads those rows again from the heap, converts a numeric string as the cast does, and raises the error of a value SQL cannot convert only for a row that every other condition keeps. PostgreSQL tests the conditions of a scan by cost and then in the written order, so the scan answers as it does exactly when each other condition is cheaper than the cast or as cheap and written before it; otherwise (a costlier condition, a condition that may raise, two casts of paths that hold bad values, an OR) the query keeps the normal plan, and `sift.explain_refusals` names the new slug `dirty_order`. Before, any such row kept the query on the normal plan (`dirty_units`). Tests `agg_scalar_dirty_where`, `agg_scalar_dirty_share`, `agg_scalar_overflow` and the `vd_scalar_where_*` fixtures of the differential suite.
- Changed: pg_sift: the planner prices the rows the aggregate scan reads again from the heap (a value a typed leaf cannot store) from a sample of the flagged units: the statistics of a `sift_col` index count the overflow slots of up to four flagged units of each segment (one chunk read each, once, when the statistics are made; no format change), and the share of the rows read again is that density times the share of flagged units, where it was a flat 2%. A row read again is priced at what it measured (about 1.6 microseconds, five times what the model charged). A table with a few percent of dirty documents keeps the scan (3%: 8.7 ms against 28.1 ms for the normal plan at 100,000 items); one with a third or more keeps the normal plan, which is cheaper there (50% dirty: 27 ms against 83 ms for the scan). Test `agg_scalar_dirty_share`.
- Added: pg_sift: `CREATE INDEX` of a `sift_col` index with a wildcard path (`paths = 'm.* int4'`) says when the keys passed the dictionary of 65,536 keys, with the share of the units that hold a key beyond it (`NOTICE: the keys of m.* passed the 65536 keys the index keeps in 3 of 40 units (7.5%)`, a hint to declare the keys queries read by name). The keys and values past the limit were already stored as overflow and read from the heap, and a query that reads the leaf keeps the normal plan (`dirty_units`); nothing said so. Testing builds: `sift.test_limits = 'shred_keys=16'`. Test `col_wildcard_keys`.
- Fixed: pg_sift: a scan of thousands of partitions took time quadratic in their number: each partition's source registered the scan's snapshot with the resource owner, which then held thousands of registrations of one snapshot (its hash adds and forgets them in time proportional to the equal keys it holds: about half of the instructions of a query over 5,000 partitions), and each begin of a range of units found its partition by walking the partitions from the first. The scan registers its snapshot once (`SIFT_COLO_SNAP_HELD`), finds the partition by bisection, and a source no longer zeroes the 10 kB of directory copy and test cache it has not filled. A query over 5,000 partitions of 6 rows each: 907 to 312 million instructions per run (the normal plan: 300 million); backend CPU time, with the host at a load of 30, 2.7 times the normal plan's before, 1.7 to 2.1 after (last-level cache misses, from the pages of the index of each partition, are three times those of the normal plan: the remaining difference). Tests `agg_partitions`, `agg_partitions_many`, `agg_par_partitions`.
- Changed: pg_sift: the spine of a two-level aggregate (a series or table of buckets joined to the arms) is matched through a hash of its join values, not by a pass over its rows for each distinct key of the cells, and it is no longer limited to 10,000 rows (slug `spine_rows`): it is bounded like the other tables the scan loads, by `sift.agg_lookup_max_kb` (new, 256 MB by default; rows times their width plus 64 bytes a row, as the planner estimates them; `sift.agg_lookup_max_rows` still limits the rows alone when it is set, and is -1 by default). Spine of 1,000 / 10,000 / 100,000 rows over 20,000 facts and 5,000 cell keys: 60 / 425 / 693 ms (the 100,000 rows were refused) before, 10.6 / 17 / 130 ms now. Tests `agg_two_level_spine`, `agg_refusals`.
- Changed: pg_sift: the rows of a joined table are shared with the other backends by default (`sift.agg_lookup_shared_kb` is -1: the smaller of `sift.agg_lookup_cache_kb` and 5% of `shared_buffers`; 0 turns it off, as it was before): sessions that run the same query at once wait for the one that loads the table (30 s at most) and read its rows in place; a system with no room for the segment (a small `/dev/shm`) gives a notice and each backend loads its own, as before. 32 sessions that join a 200,000-row table: 254 MB (private) and 88 MB (shared) in all, 7.9 MB and 2.5 MB for each session after the first. Test `020_lookup_shared_concurrent` (16 sessions, one load).
- Changed: pg_sift: a scan holds up to 32 lookup-joined tables (was 8), 128 value combinations (was 16), 16 columns of a joined table read by expressions (was 6) and 8 ORDER BY columns of a first-row lookup child (was 3); the arrays of the tuples of a chunk are sized by the query, so a narrow query costs what it did. One more of the first three keeps the normal plan and `sift.explain_why` names the limit and its value (slug `shape_width`); one more ORDER BY column runs the sub-select once per key as a lookup subquery.
- Changed: pg_sift: a numeric `IN` / `NOT IN` list of more than 64 constants is pushed into the scan as a hash set of the scaled values (it was evaluated by PostgreSQL for each distinct value, or refused), with NaN equal to NaN only and the constants no stored value can equal left out. Test `agg_numeric_in_list`.
- Added: pg_sift: `sift.table_wide_rows` (3.3, default on; `off` refuses the row again): a row of a `sift_table` that does not fit one write buffer (4 MB by default, 8 MB at most; it raised `row of sift_table is too large`) is written to tail pages beyond the page table of its buffer, up to PostgreSQL's 1 GB. The buffer that takes one is closed behind the row and marked directory format 11, which a build that does not know it refuses to write to; a row that fits keeps the layout of earlier versions, and a buffer without a wide row stays format 0. 5 / 50 / 200 MB rows insert in 45 / 400 / 1,400 ms; after `VACUUM` the pages are free again and the rows are in column units. The continuation pages are validated when read (a page that is not one, or holds no bytes, is data corruption). Tests `tbl_wide_rows`, `tbl_wide`, `errors_hints`.
- Added: pg_sift: every refusal of the aggregate pushdown has a slug that is a `semantic` shape or a `limit` (`sift.refusal_slugs()`, generated into the tables of USAGE and COL-PLAN), and a refusal by a fixed limit of docs/LIMITS.md names the limit and its value (`the limit group_keys is 32`) in `sift.explain_refusals` and `sift.explain_why`. The session counters `sift.stats()` count `refused.<slug>` and `limit.<id>`, with `sift.explain_refusals` or the new `sift.count_refusals` (no notice). New slugs `shape_width` (joined tables and value combinations) and `error_parity` (a cast of a JSON null before PostgreSQL 18, a `jsonb_typeof` guard around a read of its path, which used to report `other`).
- Added: docs: `docs/REFERENCE.md` and a README block list every user-facing object of schema `sift` (signature, result, release that added it, description), generated from the install scripts: the description is its `COMMENT ON`.
- Changed: pg_sift: a number or boolean path of a `jsonb` column that a `sift_col` index declares typed (`doc.v float8`) and a query reads as `(doc ->> 'v')::float8` or `(doc -> 'v')::float8` in an aggregate argument or a grouping expression keeps the aggregate scan when some documents hold a value the leaf cannot store (a numeric string, a boolean, a container): the scan reads those rows again from the heap and answers as the normal plan does, a numeric string converted and a value SQL cannot convert raising the same error for the same row, and nothing for a row a condition on another column drops. Before, any such row kept the whole query on the normal plan (`dirty_units`), or made a plan made before the value was written raise `0A000`. The path read in a `WHERE` condition, a join or a sub-select, or under a `CASE` or `FILTER`, and a query with an unnest keep the normal plan as before. Tests `agg_scalar_dirty`, `agg_scalar_dirty_share`, `agg_scalar_overflow` and the `vd_scalar_*` fixtures of the differential suite.
- Changed: pg_sift: `count(DISTINCT)`, `sum(DISTINCT)` and `avg(DISTINCT)` of a numeric column or numeric expression can run with workers (they ran on the leader alone): a worker sends the (group, number) pairs of its set with the numbers themselves, since the ids of its table of them are its own, and the leader counts or adds each number once per group. 200,000 rows, `count(DISTINCT amount) GROUP BY grp`: 8.5 ms in one process, 14.7 / 12.8 / 14.3 ms with 1 / 2 / 4 workers forced (the start of a worker is about 6 ms: the planner keeps the leader alone below the cost of that, as for every aggregate), `sum(DISTINCT amount), avg(DISTINCT amount)`: 72 ms alone, 79 / 74 / 61 ms. `EXPLAIN (VERBOSE)` shows `Serial Because` for a scan that cannot have workers whatever the cost. COVERAGE X02 is Yes, the aggregates that stay on the leader (a generic aggregate with DISTINCT, an integer expression through a combination of values) are X06. Test `agg_distinct_parallel`; the diff fuzz generates the aggregates.
- Changed: pg_sift: the errors say what to do and carry a code that tells a limit from a bug. Every error with SQLSTATE 54000, 0A000 or 55000 has a hint or a detail; no error is XX000 any more except `SIFT_INVARIANT` (an `Assert` in an assert build) for what the code rules out; a row or unit of a `sift_table` that a concurrent merge or defrag replaced is 40001 (retry); a numeric beyond 38 digits in a wide leaf is 22003; a lost parallel peer is 08006 with a hint; the format messages name the pg_sift version of the build. `scripts/check-errors.sh` (lint job, `scripts/ci.sh`) enforces it. `sift.score()` outside a sift scan says where a score exists. `CREATE PUBLICATION` and `ALTER PUBLICATION` of a `sift_table` without pg_sift in `shared_preload_libraries` now fail at once (3.3 scripts: the event trigger `sift_publication_check`) instead of at the first write. Tests `errors_hints`, `007_tbl_logical`.
- Changed: pg_sift: a scan that meets a value its index cannot store in a leaf its plan did not know of (a generic plan kept by a transaction, a value committed after the plan was made) invalidates the cached plans of the table in every session before it raises its `0A000` error, and the hint says to run the statement again: the next execution is planned with the leaf flagged (the normal plan, or a source that reads the rows again). PostgreSQL has no way to plan a statement again once it runs, so the statement that meets the value still fails. Test `agg_vleaf_overflow`.
- Changed: pg_sift: the planner prices the memory of the group table of a `Custom Scan (SiftAgg)`: the groups it expects times the bytes of a group (the group arrays of each aggregate, the ids of the keys; about 185 bytes for one key and five aggregates, measured) against the memory of a hash aggregate (`work_mem` times `hash_mem_multiplier`). Past it the scan is charged the spill of sorted runs of partial states, priced as `cost_agg` prices the spill of a hash aggregate, and when that makes it dearer than the normal plan the arm keeps the normal plan with the slug `group_memory` (`sift.explain_why`); a scan that fits is priced as before. Test `agg_group_memory`.
- Changed: pg_sift: the group table of a `Custom Scan (SiftAgg)` is bounded by `hash_mem` (`work_mem` times `hash_mem_multiplier`, 128 groups at least) for the aggregates whose partial states merge by key value (`count`, `sum`, `avg`, `min`, `max` over integers, floats and numerics, with any number of keys, no `DISTINCT`, no top-N): when the table passes the budget at the end of a unit, its groups are written to hash partitions on disk in the partial blob form of the parallel workers, the table is emptied, and the partitions are merged one at a time at the end (a partition that still does not fit is split again, up to `group_spill_depth` levels, then ERROR 54000 with the settings to raise). Results equal the unbounded table's (sums of floats within 1e-9 relative, their order depends on the merge). The planner asks for no workers when the table must spill (each would hold a table of its own), and `EXPLAIN ANALYZE` shows `Group Table Spills`. 200,000 groups of one integer key, `count/sum/avg/min/max`: peak backend memory (`VmHWM`, 16 to 19 MB of it the shared buffers the query touches) +44 MB unbounded, +20 MB at `work_mem` 4 MB; 115 to 145 ms unbounded, 313 to 407 ms spilling at 4 MB (the normal plan: 218 to 274 ms), unchanged when the table fits. An aggregate that cannot spill (`count(DISTINCT)`, a two-level arm, a set key, several grouping sets) keeps the scan and its whole table until the table is more than `group_nospill_factor` (32) times the larger of hash memory and 8 MB, and then the normal plan with the slug `group_memory` and the detail naming the factor. Registry rows `group_nospill_factor`, `group_spill_parts`, `group_spill_depth`. Tests `agg_group_spill`, `agg_group_spill_limits`.
- Changed: pg_sift: `count(DISTINCT x)` and the distinct `sum` and `avg` of numbers spill with the group table (they kept the scan and the whole table until it was 32 times hash memory): the set of a group goes to its partition as the (group, value) pairs and bitmap chunks the parallel workers send the leader, and the partitions united when they are merged. 150,000 rows, 20,000 groups, `count(DISTINCT val), count(DISTINCT tx)` at `work_mem` 64 kB: 6 spills, the rows of the normal plan. Still on the scan and its whole table until `group_nospill_factor`: the arms of a two-level aggregate, grouping sets, set keys, top-N, ordered-set and generic aggregates (their values are one sort or a state the blob does not hold per group) and `sum`, `min`, `max` over combos (numeric and text columns). Test `agg_group_spill`.
- Changed: pg_sift: `DISTINCT ON` in the aggregate scan accepts a unique index column that can be NULL in the `ORDER BY` when a strict condition of the `WHERE` clause (`col > 0`, `col IS NOT NULL`, `col = ANY (..)`) leaves no row with a NULL there, since no two rows can then tie; a table row of the nullable side of an outer join keeps the rule.
- Changed: pg_sift: an integer or date key of the joined table that is a cast of its columns (`dims.at::date = facts.ts`, a widened integer) is a hash key of the lookup join in the aggregate scan, evaluated for each row loaded, as a column is (a series of days joined to timestamps: 51 / 54 / 54 / 55 ms before and 2.3 / 2.6 / 4.7 / 7.7 ms after at 25k / 50k / 100k / 200k rows); an expression that can raise or is not immutable stays a condition of the table's rows.
- Changed: pg_sift: SiftAgg reads a partitioned table of up to 32,768 partitions after PostgreSQL's pruning (was 4,096, and a table above it kept the normal plan under the slug `partition_source`, the one of a partition without a usable index): above the limit `sift.explain_why` names `partition_count` with the number of partitions and the limit. The cost of opening the partitions a scan reads still decides against the normal plan. The remainders of a hash level are computed once per level for the values of the keys (was once per partition and combination of values, and none above 4,096 combinations: now up to 1,048,576), so `k = ANY (..)`, a join on a hash key and a parameter prune a hash table of thousands of partitions in a pass over the partitions. Documented: `max_locks_per_transaction` needs about 2 x the partitions of the plan whichever plan runs (the scan takes no more locks than the normal plan: 1,204 and 1,204 at 600 partitions). Tests `agg_partitions_many` (limit lowered to the edge of 100 partitions, lock count), `agg_partitions` (hash remainders, limits lowered), `agg_par_partitions`.
- Changed: pg_sift: a `sift_col` leaf whose strings in one unit (256 heap pages, up to 74,496 rows) pass 1 GB no longer stops `CREATE INDEX`, a merge or a promotion (`the variable-length values of a unit exceed 2 GB`, `54000`; 74,000 rows of 30 KB values reach it with the values detoasted). The values after the budget are stored as overflow (NULL and flagged, as a value the leaf cannot store is), a NOTICE names the leaf and the superblock, and a query that reads the leaf keeps the normal plan (`dirty_units`); the other leaves are scanned as before. `sift.col_verify` accepts such a string against the heap. A wide-numeric value that does not fit 128 bits is stored as overflow too, where it raised `XX000` (no column value reaches it: `numeric(p, s)` bounds the digits). Testing builds: `sift.test_limits = 'unit_bytes=65536'`. Test `col_wide_values`.
- Changed: pg_sift: an `ORDER BY .. LIMIT n OFFSET m` with `n + m` past 100,000, or with more than 4 sort columns, no longer keeps the normal plan by rule: the top-N scan keeps every row that passes as its keys and TID (in memory up to `hash_mem`, then a `Tuplesort` that writes runs to disk), sorts them, and the planner takes it by cost against PostgreSQL's sort (`EXPLAIN` shows `Top-N: .. (limit N, sorted)`). A heap of more than 1024 rows is priced against that sort and loses to it where it costs more. When the output columns are plain columns of the table, the first `m` rows that the `Limit` throws away are handed to it as rows of NULLs and are not read from the heap (`Heap Fetches` is `n`, not `n + m`). 200,000 rows, `ORDER BY ts, id LIMIT 10 OFFSET 150000` 52 ms against 113 ms for the normal plan (`OFFSET 50000`: 45 against 81 ms, `OFFSET 0` unchanged at 1.5 ms); the limits are the registry rows `topn_rows` and `topn_keys` (`sift.test_limits` lowers them in a testing build). Tests `col_topn_deep`, `col_topn_deep_limits`; the diff fuzz generates up to 6 sort keys and offsets past the table.
- Changed: pg_sift: the rows a `Custom Scan (SiftAgg)` reports to the plan above are the groups of the key columns' distinct values where a filter of the table has no statistics (a jsonb path, an expression), not the groups of the few rows PostgreSQL guesses that filter keeps; a join or window over the aggregates is planned for the groups the scan makes.
- Changed: pg_sift: three errors of the aggregate scan that the normal plan does not have are gone: the outer `sum` of an int8 aggregate of the arms is a 128-bit sum that comes out as numeric like `sum(int8)` (it raised "a sum of the aggregate scan exceeds bigint"), and a value-combination cache that reaches 16,777,216 entries (a lookup join or an expression over a column with that many distinct values) starts again instead of raising "too many distinct value combinations" (`EXPLAIN ANALYZE` shows `Combo Caches Restarted`; `sift.test_limits = 'combo_entries=N'` lowers the limit in a testing build).
- Fixed: pg_sift: two words of a text index that share their first 255 bytes were one term (a query for either returned both documents, not rechecked). A word over 255 bytes (folded) is now its first 236 bytes (cut at a character boundary), U+FFFF and a 64-bit hash of all of it, as the keyword analyzer already did, and is not stemmed; a prefix query longer than 236 bytes asks for the first 236. Indexes built before keep such words cut at 255 bytes (no format change, the analyzer version is unchanged so they stay readable): `REINDEX` an index whose documents hold words over 255 bytes (base64 blobs, hashes, long URLs); until then a query for such a word finds nothing in the old segments. Test `analyze_long_token`.
- Added: pg_sift: `sift.rls_scores` (superuser setting, off by default): a role under a row-level security policy gets the ranked scan and `sift.score()`, where it got an error. The policy's quals run on every row the ranker offers before any qual of the query (the order of a sequential scan: a function of the query that raises for a hidden row never sees one), the ranker is asked for more rows until `k` pass, and no hidden row is returned; policies that call functions or read other tables, `FORCE ROW LEVEL SECURITY`, `BYPASSRLS` roles and partitioned tables are tested against the full ranking filtered by the tenant. The scores are those of the whole index, so they depend on the hidden rows (the caveat is in docs/USAGE.md, "Row-level security"): the plan's estimates do not read the index, `EXPLAIN ANALYZE` prints none of the index's counters for such a scan, a keyed scan and `sift.max_score()` are not offered, and a role cannot switch the setting itself. Ranked top 10 at 100,000 documents, 50% / 5% / 0.5% of the rows visible, five queries from a term in 5% of the documents to a two-term query, median of 3: 0.15 to 0.9 ms (the owner without a policy 0.15 to 0.8 ms); the plan the role had before, the policy's bitmap and the match tested on each visible row, takes 5 to 31 ms to return the matches unranked and 4.9 to 28 ms for ten of them. Tests `rank_rls`, `rls_stats`.
- Fixed: pg_sift: with `sift.rls_scores` on, a policy that is itself a match (`USING (tenant = current_user AND body @@> 'open')`) was a term of the ranked scan's query, so the scores of a role under it counted that word and were not the table owner's for the same rows; it is a qual of the heap row now, as any other qual of a policy. Test `rank_rls`.
- Fixed: pg_sift: the row estimate of a `@@>` condition on a partition or an inheritance child of a table under a row-level security policy came from the index, so `EXPLAIN` (no setting needed) told a role whether a word is held by rows the policy hides (the child carries the parent's policy in its restrictions, not in its range table entry, which is what the check read); it is the planner's default now, as for a table without partitions. The score refusal names the table when a partition or child is under a policy (it said "needs a sift index on every partition"). Test `rls_stats`.
- Fixed: pg_sift: `sift.score()` in a query level that PostgreSQL moves: a score read only in `HAVING` (`HAVING max(sift.score(ctid)) > c`) and the source of an `INSERT ... SELECT` raised "must be computed by a sift scan" (the insert said "FOR UPDATE and FOR SHARE") after the planner pulled the subquery up; both are planned with the scan. `UPDATE`, `DELETE`, `MERGE` and `FOR UPDATE` re-check the rows they read, which a scan cannot: a subquery, view or `WITH` query of such a statement that reads the score is now planned apart (as with `OFFSET 0` or `MATERIALIZED`), so `UPDATE t SET s = q.s FROM (SELECT id, sift.score(ctid) AS s FROM docs WHERE body @@> 'x') q WHERE t.id = q.id` works; the same score read straight from a table the statement joins stays an error, and its hint says how to write it. The other `sift.score() must be computed by a sift scan` outputs of the tests (no `@@>` condition, a score of another relation, `sift.enable_scan = off`, a score in `WHERE` that is not a cutoff) are the documented errors. Test `rank_score_shapes`.
- Added: pg_sift: `docs/LIMITS.md`, the table of the fixed limits (value, unit, what happens at the limit, the setting that changes it), generated by `scripts/gen-limits.sh` from the registry in `src/core/limits.h`; `scripts/gen-limits.sh --check` runs in lint and fails when a `*_MAX_*` or `*_LIMIT*` macro has neither a row nor a reason in `scripts/limits.allow`. Testing builds: `sift.test_limits = 'id=value,...'` lowers a limit so that a test reaches it on a few rows, and `sift._test_limits()` lists the registry.
- Added: pg_sift: `ALTER EXTENSION pg_sift UPDATE` to 3.3 (any older version: it is the last step of the path) prints what is left to do, as warnings with the commands (the first 20 of each and the count of the rest) or a notice that nothing is: a `REINDEX` of every `sift` text index except `analyzer = keyword` (**queries miss rows** without it: a word over 255 bytes is stored as its start and a hash since 3.3, an older index holds it cut at 255 bytes, and the index cannot say whether it holds one, so it stays listed), a `REINDEX` of a `sift_col` index whose option came after its units (`cube_minmax`, JSON `paths`, `rollup_dims`, `key_columns`), and a `VACUUM FULL` of a `sift_table` written below the column format of its schema (calendar cells of date and timestamp columns, minor 8; complete JSON kinds, minor 6), which takes an ACCESS EXCLUSIVE lock. `sift.upgrade_commands()` (3.3 scripts) lists them again, one row per command (`kind`, `object`, `reason`, `command`, `required`); `sift.upgrade_check` takes its option and table rows from it. The check only reads, and never fails the update: when the library cannot answer it says to run the function after reconnecting. Tests `extension_lifecycle`, `compat_matrix`, `upgrade_check`.
- Added: pg_sift: `sift.explain_why(query)` (3.3 scripts), the public form of what the test function `sift._test_explain_why` returned: one row per arm of the aggregate pushdown (`arm`, `used`, `slug`, `reason`, `scan_cost`, `normal_cost`) for one `SELECT` that is planned and not run. Test `agg_refusals` no longer needs the test build; `extension_lifecycle` and `compat_matrix` check the function after each update path.
- Changed: pg_sift: `sift.stats()` reports the skipped units as `units_pruned_zone` (zone maps), `units_pruned_set` (small-set summaries, the `Units Pruned By Set Summary` of `EXPLAIN ANALYZE`) and `units_pruned_search` (units without a row of the search selection) in place of the one `units_pruned` (their sum is the `Units Pruned` of `EXPLAIN ANALYZE`). The units a small-set summary cleared are counted when the scan passes them, as the others are, where they were counted when it began: a scan that ended early (a top-N) over-reported them.
- Added: pg_sift: `sift.upgrade_check` warns when `key_columns` or `rollup_dims` is set on an index whose older units lack the section (column format below 10, below 9): they are read or scanned until a `REINDEX`. Test `upgrade_check`.
- Changed: pg_sift: the README PostgreSQL badge says 17, 18 and 19.
- Added: pg_sift: a distribution kit: container image recipes on the official `postgres` image (PostgreSQL 17 and 18; `packaging/docker/`, [docs/CONTAINERS.md](docs/CONTAINERS.md)) with a Docker Compose example and a CloudNativePG image-volume extension image, a five-minute [quickstart](docs/QUICKSTART.md), `bench/bench kit` (the ten store questions at 100,000 orders in docker, every answer compared with `sift.enable_agg = off`), the metadata files `packaging/pgxman.yaml` and `META.json`, and [docs/SUPPORT.md](docs/SUPPORT.md) (supported versions, upgrade paths, the downgrade rule). No image is published.
- Changed: pg_sift: an `ALTER TABLE` that adds a column to a `sift_table` past the 127 it holds, or changes or renames a column so that its declared paths count again, fails at once with the error of `CREATE TABLE` (`a sift_table holds at most 127 columns`, `54000`; an event trigger, `sift_table_ddl_check`, in `pg_sift--3.3.sql` and `pg_sift--3.2--3.3.sql`), partitions included. It used to succeed and fail the next write. `DROP COLUMN` is not checked, so that a table that 3.2 let grow too wide can be narrowed.
- Added: pg_sift: `sift.table_space(tbl)` reports `vblocks_left`, the virtual blocks of TIDs a `sift_table` has not handed out yet (up to 291 rows each, never reused; `VACUUM FULL` renumbers them). The function is replaced in `pg_sift--3.2--3.3.sql`: `ALTER EXTENSION pg_sift UPDATE`.
- Changed: pg_sift: planning and starting a scan of a partitioned table cost less for each partition: the layout, statistics and pre-aggregates of a partition's index are kept for the backend (`sift cache partition facts`) and read again only when its directory, the pages of its table or their visibility map, or a catalog entry of the index or the table change; the all-visible pages are counted from the bytes of the visibility map the table uses, not from all of every map page; a partition's statistics are kept packed (the leaves in use, not 15 kB); the partitions of a range key that its conditions leave out are found by bisecting the bounds PostgreSQL keeps, not by a proof for each; and a partition's heap is opened when the scan reaches it or its bound is first read. One aggregate over 1,000 partitions plans in about half the instructions, a generic `EXECUTE` that prunes 4,000 to one starts in less than half.
- Changed: pg_sift: a lookup join keyed by columns of an earlier lookup's table (`LEFT JOIN map ON map.kind = e.kind AND map.cat = i.cat` after `JOIN e`) makes one cache entry per distinct value of those columns, not per row of `e` that holds them: 500 entities of 4 types are 4 entries, not 500 times the combinations of the other keys, and the entry table stays in cache. On a neutral two-lookup shape (workload table) 200,000 rows: 211.9M to 184.5M instructions per execution (-13%), 105M to 78M cycles. Test `agg_lookup_chain`.
- Changed: pg_sift: the aggregate scan counts the tuples of a chunk once for the groups seen and for every aggregate that takes all of them (`count(*)`, the count of an unfiltered sum or average), in one pass over the group ids where it made one per aggregate. User instructions per execution, 100k orders: items top-N (float sum, count) 36.22M to 33.71M (-6.9%), a 100k-row entity top-N 7.09M to 6.71M (-5.5%), three keys 33.17M to 32.48M (-2.1%); shapes whose aggregates skip rows are unchanged. Results are the same (`agg_dense`, `col_agg_*`, `agg_distinct`, `agg_numeric`).
- Changed: pg_sift: the aggregate scan finds the NULL tuples of an integer, date or dictionary key in one branch-free pass and keeps them as a bitmap, where it made three passes with a branch per tuple (the id patch, the range of the non-null values, the fix after the kernels). A date-bucket aggregate with a text key, 100k rows: 15.60M to 15.03M user instructions and 15% fewer branch mispredictions per execution; three keys 25% fewer mispredictions (instructions +1.1%). Results are the same.
- Changed: pg_sift: the value of the member of an object by key (a dynamic key of a jsonb object, `metrics -> 'key'`) is read by walking the elements the key marks and finding the row of each by the offsets, where it searched every row's elements for the last marked one: 17.84M to 17.19M user instructions (-3.7%) on a one-key filtered metrics shape, 100k rows. Results are the same.
- Changed: pg_sift: the backend keeps the decoded list lengths of a unit of an immutable segment (the counts of a jsonb array leaf and their running sums, which every read of its elements starts from) with the dictionaries it already keeps (`sift.test_col_dict_cache` is the budget in test builds, 64 MB), so a scan that keeps a few rows of every unit does not decode the length chunk of the whole unit again on each query. A list scan with one key of 300 over a neutral table (2 to 3 elements per row, three runs of 15, ms): 25k rows 0.183 to 0.183, 50k 0.261 to 0.210, 100k 0.445 to 0.342, 200k 0.838 to 0.560 (normal plan 0.637); user instructions at 200k 6.98M to 4.89M. No on-disk change. Test `col_dict_cache`.
- Added: pg_sift: `CREATE INDEX`, `REINDEX` and `sift.col_refresh` of a `sift_col` index say, as a NOTICE, where a cap of its options leaves 10% of the units or more without what was asked: a `cube(..)` dim that has no histogram in a unit with more groups than the cap (4,096), and a `key_columns` leaf whose summary is a bloom filter, not its keys, past 32 keys. A query on them scans those units; `sift.col_rollup_misses` counts the scans.
- Added: pg_sift: the automatic rewrite of a `sift_col` index can be by range and bounded: `sift.col_auto_merge_partition_kb` (off by default) keeps the index as segments of about that size and rewrites one at a time, the one with most units holding a dead row (over `sift.col_auto_merge_dirty_pct` of its own units, whatever the share of the whole index), then the small segments the promotions left over the partitions, so that an index churned in one place (a retention window, a hot tail) writes that place only, holds the maintenance lock for one step and needs free space for one partition (the merge tiers leave the partitions alone; the lock is let go between steps). `sift.maintenance_wal_rate` (kB per second, 0 = no limit) makes the merges of the maintenance worker wait before the page that would pass that rate of WAL, and `sift.col_auto_merge_max_workers` (0 = no limit) caps the rewrite workers of a database that run at once (advisory-lock slots, nothing in shared memory). No format change. Tests `col_auto_merge` (regress), `col_auto_merge_partition`, `col_auto_merge_part_crash`.
- Added: pg_sift: the background pass of a `sift_table` (`sift.table_auto_defrag`, still off by default) is asked for by a delete, an update or a read (an index scan, a fetch, an aggregate) as well as by the writer that closes a write buffer, each backend at most once per `sift.maintenance_naptime`: a table that is only deleted from, or whose writes are too small to close a buffer, was never defragmented. `sift.table_auto_defrag_dead_pct` (20, the autovacuum scale factor; 100 never asks for dead rows) is the tombstone share that asks, and a pass rewrites at most `sift.table_auto_defrag_budget` MB (128) of segments, the most dead first. 200,000 rows, 12 cycles of 4% deleted and as many inserted (item/event churn, PostgreSQL 18, scratch cluster): file 19.2 MB without maintenance, 8.2 MB with the pass (4.96 MB with autovacuum on), 50 bytes a live row instead of 118, the 3-column aggregate 11.5 ms instead of 27.8; WAL 113 MB instead of 88. Delete-only churn: 6.8 MB without, 5.5 MB with. Writes too small to leave dead rows: nothing is asked for. Tests `017_tbl_autodefrag`, `019_tbl_defrag` (a heap twin, `wal_consistency_checking = generic`, a standby, a worker killed with SIGKILL in the middle of a pass), isolation `tbl_maintain`.
- Changed: pg_sift: `VACUUM` and the passes of a `sift_table` rewrite the segments that are most dead first (`tbl_defrag_order`), and a segment of an older layout no longer stops the rewrite of the ones after it; a pass bounded by the byte budget has freed the most it could.
- Fixed: pg_sift: the table `VACUUM` honours the cost delay (`vacuum_cost_delay`, `vacuum_cost_limit`; a background pass takes `sift.table_auto_defrag_delay`, now 0 to 100 ms as the cost delay allows) at every segment and page it rewrites, where it ran flat out; the worker applies it as an autovacuum worker does.
- Fixed: pg_sift: a role that is neither a superuser nor the owner of the extension could not drop anything once pg_sift was installed (**known issue of 3.2.0 and 3.2.1**): the trigger `sift.table_paths_drop` that clears the paths of a dropped `sift_table` ran as the dropping role, so every `DROP TABLE`, `DROP INDEX`, `DROP SCHEMA ... CASCADE` and `DROP OWNED` of that role (a table or index of any kind, with or without pg_sift) failed with `permission denied for schema sift`; `sift.table_set_paths` failed with `permission denied for table table_paths` for the owner of the table. Both functions are `SECURITY DEFINER` with `search_path = pg_catalog, pg_temp` now (the new script, and `ALTER EXTENSION pg_sift UPDATE` to 3.3 for an installed 3.2.x); `sift.table_set_paths` still requires the owner of the table (the session's role). Workaround on 3.2.0 and 3.2.1 until the update, as a superuser in each database with the extension: `ALTER FUNCTION sift.table_paths_drop() SECURITY DEFINER SET search_path = pg_catalog, pg_temp;` (a restore from a dump recreates the extension from the 3.2 script: run it again; `docs/UPGRADING.md`). Test `tbl_nonsuper_ddl`.
- Fixed: pg_sift: the aggregate pushdown (`Custom Scan (SiftAgg)`) no longer refuses what the normal plan answers (since 3.2.0): a role with `SELECT` on the columns a query reads but not on the whole table, or on a view and not on its table, got `permission denied for table` at run time. The scan asked for table-level `SELECT` of the invoking role; PostgreSQL's executor has checked the privileges on the columns, as the owner of the view, before it starts, and the scan adds no check. A table with row-level security read through a view of its owner (to whom no policy applies) keeps the normal plan under `table_kind`, where the scan's own row-security check refused the query.
- Changed: pg_sift: the install and the update to 3.3 grant `USAGE` on the schema `sift` to `PUBLIC`, so a DBA no longer grants it for a role to call `sift.score(ctid)` or the read-only diagnostics (the functions check the table or index as before; no table or function grant changes); `REVOKE USAGE ON SCHEMA sift FROM PUBLIC` takes it back, and an update grants it again.
- Fixed: pg_sift: `sift.upgrade_commands()` failed for every role but a superuser (`permission denied for table table_paths`) though the REINDEX hint of the update is for the table owners: it lists the indexes and tables the caller owns or may maintain (the privilege `REINDEX` and `VACUUM FULL` ask for; a superuser sees all), and reads the declared paths through the new view `sift.table_paths_maintained`, which lists the paths of those tables only (an owner could not read the paths of its own table).
- Changed: docs: `DROP EXTENSION pg_sift` leaves the schema `sift` empty with its privileges (it did in 3.2): PostgreSQL makes the schema from the control file and refuses it as a member of the extension; `CREATE EXTENSION` gives a fresh install's objects and privileges again (`extension_lifecycle` checks it), `DROP SCHEMA sift` removes it.
- Fixed: pg_sift: `CREATE PUBLICATION` and `ALTER PUBLICATION` by a role that is not a superuser failed on a server with `wal_level = logical` (`permission denied to examine "shared_preload_libraries"`, from the trigger `sift.publication_check`, 3.3 only): it runs as the extension's owner now.
- Fixed: pg_sift: `sift.workers`, `sift.col_rollup_state` and `sift.col_rollup_requests` could be read by superusers only (`permission denied for view`): every role with USAGE on the schema can, as for the functions they list (`sift.workers` shows the indexes of the tables the caller can read).
- Added: tests: `roles_ddl`, `roles_dml_query`, `roles_maintenance`, `roles_settings`, `roles_upgrade` (SQL) and `roles_cluster` (pg_dump, pg_restore, tablespace, publication) run what the features do for ordinary roles (a table owner with only CREATE on a schema, a member of the owner, roles that read, write or have no access, a role under row-level security, a role with MAINTAIN): the suites that run as a superuser could not see the DROP failure of 3.2.0 and 3.2.1.
- Added: pg_sift: `key_columns` (`sift_col` reloption, default `auto`; `off`, or a list; the one option for the keys of a column in a unit, in place of `set_columns`, `set_limit` and `key_presence`): every unit written afterwards keeps, for the columns it names, the set of its distinct keys, as a key list or a bitmap where that takes the share of the unit's bytes that the lists of its columns have left (3%), else, for a column the user named, exactly while there are at most 32 keys and as a bloom filter beyond. A key list or bitmap is the keys of a top-level `int2`, `int4` or `date` column: a flat `count(DISTINCT key)` ORs them and reads no key chunk (1,000,000 rows in 123 units of 8,130 rows, 6,313 keys, median of 9 interleaved runs: 16.1 ms scanning the units, 1.11 ms (14.5x; the normal plan takes 314 ms); 2.0x to 8.6x on the foreign keys of three 200,000-row tables), and an equality, `IN` or range test skips the units that cannot hold a match where the zone map cannot (a key spread over all units). **`auto` is the default:** `CREATE INDEX` keeps the key list or bitmap of the `int2`, `int4` and `date` columns that are a member of a foreign key or are named `*_id`, that are not unique and that the cube does not hold, where a list fits in half of the units (+1.1% to +1.4% of the index on the three copies, never a bloom filter), writes them to the option as `auto: a, b` so that a rebuild keeps them, and tells as a NOTICE which columns, how many bytes and which it left out and why; `key_columns = 'off'` turns it off, `'a, b'` names the columns (a column the list names that cannot have a list gets the filter). Column format minor 10 for exact keys and filters and 15 for lists and bitmaps (`sift.col_format_minor` 10 and 15; a default 3.3 index already carries minors 7, 8 and 12, which a 3.2 library refuses, so this adds no refusal; at a setting below 15 the default writes none). `sift.col_verify` recomputes them, `sift.col_sections` lists them as `sets`, EXPLAIN ANALYZE shows `Units Pruned By Set Summary`, and the cost of a scan counts the units they skip. `sift.upgrade_commands()` lists an index with the option and older units as a gain from REINDEX. Tests `col_set_summary`, `col_key_presence`, `col_key_columns_auto`, `col_caps`, the minor 10 and 15 steps of `extension_lifecycle`, `compat_matrix`, `compat_31_column`, `upgrade_check`, `col_crash_matrix`.
- Fixed: pg_sift: the choice between a shared common table expression and the inline plan added up the cost of the `Custom Scan (SiftAgg)` arms below a `Custom Scan (SiftAggRows)` node, which reads the table once for all of them and costs less than they do added up, so a plan that is a third of the shared one kept the shared common table expression: three `UNION ALL` arms over the elements of a jsonb array, counted `DISTINCT` above a spine, 30,000 rows: 1,084 to 1,224 ms on the normal plan, 212 to 229 ms in the scan now (PostgreSQL 18, 2026-10-22, loaded host, 3 runs). The node now counts as its own cost. Rows are the normal plan's. No format change. Test `agg_cte_inline_rows`.
- Changed: pg_sift: a common table expression that two aggregates read is planned inline, and kept when the scans are the cheaper part, where it was kept only when the whole inline plan cost half of the materialized one: the plan above the scans (a `FULL JOIN` of two series on a jsonb array and a window over it, with the 2,000 rows PostgreSQL assumes for a series in a select list) costs the same in both and hid a gain of 60 times. Two series of weeks over two jsonb documents of the same events, `FULL JOIN`ed and windowed: 591, 1,068, 2,559 and 5,395 ms (25,000, 50,000, 100,000 and 200,000 rows) and now 27, 33, 60 and 90 ms with both series in the scan. No format change. Tests `agg_cte_inline_upper`, `agg_natural_cost` (a scope of a few entities now takes the scan, 0.6 ms against 1.8 ms).
- Changed: pg_sift: the extension SQL goes from 3.2 to 3.3 (`pg_sift--3.3.sql`, `pg_sift--3.2--3.3.sql`): run `ALTER EXTENSION pg_sift UPDATE` in every database. The functions added since 3.2.0 (`sift.col_refresh`, `sift.col_clustering`, `sift.stats`, `sift.stats_reset`, `sift.col_rollup_misses`, `sift.explain_why`) are in the 3.3 scripts only; the 3.2 scripts are the released ones, checked by `scripts/ci.sh --selftest`. Test `test/t/extension_lifecycle.sh`.
- Added: pg_sift: `sift.table_bulk_groups` (on by default since 3.3, `off` keeps the write buffers): `COPY`, `CREATE TABLE AS`, `REFRESH MATERIALIZED VIEW` and the rewrite of `ALTER TABLE .. SET ACCESS METHOD` into a `sift_table` write column groups straight away (the rows of a run of four superblocks are held in the backend, written and published together, with the versions of the loading transaction, or none for `COPY FREEZE`), where they went through write buffers that a later promotion copies into groups. 200,000 rows, one client, median of 3, PostgreSQL 18 (the host was busy): 286 bytes of WAL per row for the `COPY` and 23 more for the promotion, 79 in all with the groups; 2.8 microseconds per row from `COPY` to promoted, 2.3 with the groups; the file is 3% smaller. Only a table nothing reads while the load runs qualifies, and any other keeps the write buffers: one with a trigger (foreign keys included), a unique or exclusion index, a publication, a domain column, or a user-defined function in a default, generated column, `CHECK`, index expression or index predicate (`sift._test_tbl_bulk_refusal` says why). `INSERT .. SELECT` and single-row inserts are unchanged. No format change: the groups are the unfrozen groups of the 3.1 write path, and an older library refuses them as it refuses those (`scripts/old-lib-check.sh`). Tests `tbl_bulk_eligible`, `tbl_bulk_groups` (a heap twin for every kind of load, rollback, savepoint, error, many loads in one transaction), isolation spec `tbl_bulk_groups` (what other sessions and the transaction see, the rows waiting in the backend with their index entries in the btree), `t/019_tbl_bulk_groups.pl` (primary and standby under `wal_consistency_checking = generic`, `kill -9` in the middle of a `COPY`, immediate stop). Made the default after measuring a `COPY` of 200,000 rows (13 columns) into a `sift_table`, backend CPU 0.52 -> 0.39 s, WAL 44.2 -> 1.1 MB, the relation 1.2 MB either way; 20 loads of 2,000 rows: WAL 8.8 -> 0.3 MB, relation 6.1 -> 2.5 MB.
- Added: pg_sift: `sift.col_clustering` reports `depth_max` per leaf, the most units whose zones hold one value (the deepest overlap; `point_units` is the average and hides a few deep values), so the effect of rows that arrive out of order can be watched. SQL only: no format change.
- Added: pg_sift: `sift.col_clustering(index)` reports per integer, date and timestamp leaf how many units an equality or a range touches on the zone maps and whether the rows are in order (`clustered`, `partial`, `scattered`), so that a user knows if sorting the table helps; `sift.stats()` and `sift.stats_reset()` expose the counters of the `Custom Scan (SiftAgg)` scans of the session (scans, workers, units read, pruned and answered from pre-aggregates, rows decoded and read again, lookup cache hits and misses); `sift.col_rollup_misses()` counts, per index and reason slug, the scans the pre-aggregates did not answer and whether another `rollup` or `preagg` option would. The counters are folded in once per scan from the counters `EXPLAIN ANALYZE` shows, in backend memory (no shared memory, no `shared_preload_libraries`, no change to the scan loops). No format change.
- Changed: pg_sift: PostgreSQL 19 is a supported major (17, 18 and 19), no longer experimental: the tree compiles against its release candidate without a warning (an intended case fall-through is marked with `sift_fallthrough`), and the focused planner, table access method, isolation and TAP tests give the results they give on 18, also with the new `enable_eager_aggregate` on.
- Changed: pg_sift: the ON clause of a lookup join `jsonb_build_array(a1, ..) = jsonb_build_array(b1, ..)` (how a series of buckets is joined to rows by several columns) takes each element of an integer, date or text type that equals a column of the joined table as a hash key of the lookup, a NULL equal to a NULL, and keeps the other elements as a condition of the table's rows, where it tested every row of the table for every distinct input. A series of 701 days LEFT JOINed to the unnested documents of the table, one date element: 160, 173, 181 and 186 ms (25,000, 50,000, 100,000 and 200,000 rows, a cost that does not follow the rows) and now 3.0, 6.1, 7.8 and 15.5 ms; the same join on a plain `=` takes 2.7 to 13.8 ms. No format change. Test `agg_join_array_key`.
- Changed: pg_sift: planning on a partitioned table reads each partition's column index once per plan however many arms or CTEs scan the table, and keeps the statistics of up to 8192 indexes per backend (it kept 8, so a table of more partitions read every segment again for every plan); the planner no longer looks up the float operator family for every clause of every partition.
- Changed: pg_sift: every execution of a generic plan on a partitioned table (a parameter on the partition key) translated the bound of each partition again to drop the ones the parameter refutes; the bounds are kept for the backend until a relation changes, which takes the start of a scan of 4000 partitions from 48 to 30 million instructions.
- Fixed: pg_sift: a column segment written under `sift.col_format_minor = 4` or 5 (the setting for readers of 3.1) carried the zone flag of a JSON null that only 3.2 knows, so `sift.col_verify` of pg_sift 3.1 reported such an index as corrupt (its answers were right); the flag is now written from minor 6 on. Test `compat_matrix` (the releases 3.0.2, 3.1.0 and 3.2.0 write, this tree takes over, and they read what it writes at the lowest and at their newest minor).
- Added: pg_sift: the `sift.col_*_ops` classes of a `sift_col` index cover boolean, real, double precision, numeric(p <= 18, s) and char(n) (`col_bool_ops`, `col_float4_ops`, `col_float8_ops`, `col_numeric_ops`, `col_bpchar_ops`), `col_text_ops` takes `LIKE 'abc%'` and `^@` with a literal prefix (an exact prefix test, or a superset the heap scan filters for other patterns), and the index offers `Index Scan` and `Index Only Scan` (`amgettuple`, `amcanreturn` for the integer, date, timestamp, boolean, float and numeric columns; off with the rest under `sift.enable_col_scan`): a plan that stops early or reads a few rows builds no bitmap, a covering query does not read the heap (the visibility map decides), and `pg_stat_user_indexes` and `EXPLAIN` count the scans. The cost of a scan counts the units the small-set summaries (`key_columns`) skip: 1,000,000 rows, `rnd = c` returning rows 1.9 to 2.5 ms without the option and 0.18 to 0.23 ms with it (btree 0.04), `id = c` 0.09 ms (0.06), the six columns of the table in one index 28.9 MB (30.3 MB with `key_columns = 'id, rnd, s_short'`) against 170 MB for six btrees. Text and uuid equality and `LIKE` still test every entry of the segment dictionary (30 to 150 ms at 1,000,000 rows). Test `col_scan` (142 queries over the heap, a btree, the bitmap scan and the index scan through the build, the write buffer, updates, deletes, a rollback, the promotion and a merge).
- Added: pg_sift: a `sift_col` index built with the operator classes `sift.col_int2_ops`, `col_int4_ops`, `col_int8_ops`, `col_date_ops`, `col_timestamp_ops`, `col_timestamptz_ops`, `col_text_ops` and `col_uuid_ops` is a bitmap index scan (`sift.enable_col_scan`, off by default): `WHERE id = 5`, `IN` lists, ranges and `IS [NOT] NULL` on any subset of the columns of the index return rows through `Bitmap Heap Scan`, with `BitmapAnd`/`BitmapOr`, parameterized nested loops, partial and expression indexes. Nothing stored changes and the default class is the same, so an existing index is untouched; `col_text_ops` refuses a nondeterministic collation. 100,000 / 1,000,000 rows, `WHERE id = c` returning rows 0.10 / 0.11 ms (the heap scan 10.0 / 78.8, a btree 0.04 / 0.05), a range of 20% 3.4 / 24.3 ms (btree 2.5 / 29.7), `grp = c AND ts BETWEEN` 0.35 / 1.59 (btree 0.20 / 1.71); a random int equality 0.26 / 1.99 (btree 0.03 / 0.05) and a distinct text or uuid equality 4.5 / 41.7 and 4.2 / 64.4 (btree 0.06 and 0.04) are slower than a btree. The index is 0.03 to 0.56 of the btree's size on the same column (200,000 and 1,000,000 rows). Test `col_scan` compares 53 queries with the heap and a btree over the build, the write buffer, updates, deletes, a rollback, the promotion and a merge; docs/COL-SCAN.md has the verdict for each btree workload.
- Added: pg_sift: `scripts/plangate.sh`, a plan-choice gate: 96 neutral query shapes (`test/plan/shapes.sql`: a filtered group by one to three keys, top-N, joined and lateral lookups, two-level aggregates, spine joins, `EXISTS` over elements, partitions, pre-aggregate answers, `count(DISTINCT)`, fused arms, windows over aggregates, text search) on a generated table of 20,000 to 50,000 rows, each run as the normal plan, as the planner chooses and as `Custom Scan (SiftAgg)`. It fails on a result that differs between the paths, a plan other than the one the shape expects and a cost ratio (scan over normal) outside the shape's band; the times (win, tie, lose at 1 ms and 20%) fail it only without `--warn-time`. The job `plan-gate` runs without timings in the quick tier and at 50,000 rows with them in the full tier.
- Added: pg_sift: a top-N scan (`ORDER BY key LIMIT n` over a `sift_col` or `sift_table`) can run with parallel workers: each takes windows of superblocks from a shared counter and keeps a heap of its best rows, they share the first-key rank of their n-th row so that units and rows after it are left out, and the leader merges the heaps by the keys and the TID, so the rows and their order are those of one process (`EXPLAIN` shows `Workers Planned` and `Workers Launched`). The planner prices the start of the workers, so scans of a few milliseconds stay serial: 200,000 rows, a row-wise arithmetic filter and a random key, `LIMIT 1000`, workers forced: 63 ms without workers, 32 with 2, 21 with 4. Test `col_topn_parallel` and its isolation spec.
- Added: pg_sift: `sift.explain_refusals` (off by default; no plan changes and no cost when off): a query that kept the normal plan instead of `Custom Scan (SiftAgg)` reports, as a notice, `SiftAgg not used: <slug>: <reason>` with the first reason each candidate path gave up for (30 stable slugs in one registry, `src/agg/refuse.h`, listed in docs/USAGE.md; a refusal no site names is `other`), or `cost` when the path was built and the normal plan is estimated cheaper. The planner is asked without running the query by `sift.explain_why(query)` (arm, used, slug, reason, scan and normal cost; in the 3.3 scripts). Test `agg_refusals` asserts the slug of 34 shapes in one table and that every plan is the same with the setting on.
- Added: pg_sift: `sift.col_refresh(index [, max_segments])` rewrites the segments of a `sift_col` index whose pre-aggregates are not what the options ask now, so that `ALTER INDEX .. SET (cube_minmax, preagg, rollup)` reaches the units written before it without a `REINDEX`. A header-only census of the stored cubes finds the stale segments; each is rewritten alone as a merge of it does (the same rows and answers) and replaces the old one in one WAL-logged directory change under the maintenance lock, which readers and writers do not wait for, so a cancel or a `kill -9` leaves the segments done and the one in hand as it was and the call, made again, goes on; `max_segments` bounds a call and 0 counts. A segment of a minor above `sift.col_format_minor` is left as written, so that under 3 the index stays what the 3.0 library reads (the real 3.0.1 library verifies and answers from the refreshed segments). The planner counts the units that hold the histogram of a dim, its min and max and the rollup of a list, and prices the scan answered from them by that share: a half refreshed index costs between the two (min/max `GROUP BY` plan cost after 0, 1, 2, 3 steps and done, 200,000 rows: 676, 664, 645, 613, 543). Adding the min and max to four segments of 25,000 / 50,000 / 100,000 / 200,000 rows takes 23 / 36 / 50 / 81 ms (a rebuild 24 / 33 / 44 / 76 ms) and the index grows 7.5 / 16.4 / 15.2 / 17.3 kB of pre-aggregates over 270 / 426 / 754 / 1,401 kB; the space of the old segments is reused after `VACUUM`. No format change. Tests `col_refresh`, `col_preagg_minmax` (the census of cut and damaged cubes), the isolation spec `col_refresh`, `test/t/col_refresh_crash.sh`, `test/t/col_refresh_compat.sh`.
- Changed: pg_sift: `count(DISTINCT col)` of a 32-bit or text column the cube stores a histogram of (no key, or the column as the key; tests of the column or zone-covered tests of other columns) is answered from the groups of the histograms of the units whose rows are all visible, and the other units are scanned into the same set (`sift.agg_preagg_distinct`, on; `EXPLAIN ANALYZE` counts the units in `Units From Pre-aggregates`). No stored data changes. 25,000 to 200,000 rows, 8 values: 0.25, 0.64, 0.94, 1.47 ms scanned and 0.10, 0.15, 0.33, 0.52 ms from the histograms (median of 3, load 10 to 18).
- Changed: pg_sift: the walk of a top-N scan (`sift.topn_walk`) bounds the units by the zones of the sort keys after the first too (`sift.topn_zone_keys`, on): a first key that ties between units (`ORDER BY status, id LIMIT 10` with a few statuses) is decided by the zone of the next integer key, where every unit whose zone ties with the n-th row was read. 25,000 to 200,000 rows, 1,280 rows a unit, LIMIT 10: `ORDER BY g DESC, id DESC` reads 20, 40, 79, 157 units without it and 1, 2, 2, 2 with it; `ORDER BY k_mix, id` 16, 34, 71, 143 and 5, 5, 5, 5. The rows are those of the plain plan (`col_topn_order` fuzz).
- Added: pg_sift: a top-N scan (`ORDER BY key LIMIT n` over a `sift_col` or `sift_table`) takes `bool`, `time` and `money` columns as sort keys, besides the integer, date, timestamp, `numeric(p <= 18)` and float ones: each is one integer leaf whose order is the order of the type. `text`, `interval`, `uuid` and the other wide types, and expressions, keep the normal plan.
- Added: pg_sift: the coverage of two README rows (`ORDER BY .. Every refusal slug of `sift.explain_refusals` now says whether it is by design (`src/agg/refuse.h`, `docs/USAGE.md`), and a query that calls a volatile function reports the new slug `volatile` instead of the slug of the site that met it.
- Changed: pg_sift: the parallel top-N scan (workers on `ORDER BY key LIMIT n`) bounds the superblocks of its walk by the zones of the sort keys after the first too (`sift.topn_zone_keys`), as the serial walk does: each participant tests the superblock it takes against the n-th row of its own heap on the keys in order, and the shared first-key rank as before, where a first key that ties between units kept every unit. 2 workers, 25,000 to 200,000 rows, 1,280 rows a unit, LIMIT 10: `ORDER BY k_mix, id` reads 16, 34, 71, 143 units without it and 5, 5, 5, 5 with it; `ORDER BY g DESC, id DESC` 20, 40, 79, 157 and 1, 2, 2, 2 (the 7 to 12 ms of the scan are the start of the workers at these sizes: the serial walk takes 0.1 to 1 ms). The rows are those of the plain plan with 0, 2 and 4 workers (`col_topn_parallel`).
- Added: pg_sift: a top-N over an inner join to a table that only filters (`SELECT f.. FROM fact f JOIN dim d ON d.k = f.k WHERE d.attr = x ORDER BY f.key LIMIT n`) runs in the column scan: the keys of the table's rows that pass its conditions become a set of the scan (`k IN (keys)`, the join filter the aggregate scans already use), which the top-N holds exactly. Taken only when the table has a unique index on its join column (no row matches twice), the join is inner on one integer column of the scanned table, the ON clause and WHERE read the table's columns alone, and the result has none of its columns; every other join keeps the normal plan (`sift.explain_refusals`: `topn_shape`). The set is read when the scan starts, so changes of the table between two runs count; no workers with a join. Normal plan / top-N join, 25,000 to 200,000 rows, `d.label < 'label_5'` (half the keys), LIMIT 10: 9.3, 18.7, 39.5, 88.1 ms / 0.30, 0.26, 0.39, 0.48 ms; `d.label = 'label_7'` (one key): 13.8, 16.7, 36.1, 71.4 ms / 0.28, 0.32, 0.95, 2.05 ms. Test `col_topn_join`: 17 shapes (7 taken, 10 refused) with the rows of the plain plan, a 120-query differential fuzz through changes of the table, deletes, moved rows, the write buffer and its promotion, and a generic plan.
- Changed: pg_sift: the outer aggregate over per-row aggregates (the arms of a `UNION ALL` grouped by a row id: counts, `sum` / `avg` of the cells, `count(DISTINCT id)` and `count(DISTINCT parent_id)`) runs with workers: a row is in one item of work, an item flushes the cells of its last row before its table is dumped, the leader adds the tables and gives the distinct ids of the key the arms group by its own key ids, and the float sums of the cells use the lanes (the rows are the same for 0, 2 and 4 workers and the normal plan's; the largest input of an inner `avg(float8)` travels with the tables). A lateral `LIMIT 1` child of a unique key inside the arms is shared with the workers as a lookup table. The planner prices the workers on the arms' work (about 35 to 70 ns a tuple an arm) and launches none below the break-even: 200,000 documents of 2.3 elements, two arms with a parent id and a float average: 64.6 / 47.9 / 25.0 ms with 0 / 2 / 4 workers, the same with no hidden key 32.9 / 32.0 / 11.0, three joins (a `VALUES` list of spans, a table, a unique-key lookup) 56.1 / 31.0 / 25.0; at 25,000 and 50,000 documents the natural plan stays on one process (workers cost 1.5 to 2 ms there) and from 100,000 it takes the workers PostgreSQL gives the table (1 at 100,000, 2 at 200,000; `ALTER TABLE .. SET (parallel_workers = n)` raises them). Sums over a joined column (`SAA_M_*`), a hashed `NOT IN` sub-plan and a child plan run per combination of leaves keep one process (their scans take 7, 4 and 61 ms for 200,000 documents).
- Added: pg_sift: `test/t/pg_upgrade.sh` runs `pg_upgrade --check` and the upgrade of a cluster with `sift`, `sift_col` and `sift_table` objects (a tablespace, enum, uuid, interval and timestamptz leaves, JSON paths, open write buffers and promoted segments) from the previous PostgreSQL major to this one when both are installed (PostgreSQL 17 to 18 verified; `SIFT_PG_OLD_BIN` names another), otherwise to the same major, and checks the answers, `sift.verify`, `sift.col_verify`, the preflight, and writes, `VACUUM` and merges afterwards. Nothing in the files or the library needed a change. `docs/UPGRADING.md` has the commands (preflight, `ALTER EXTENSION UPDATE`, `pg_upgrade`, going back to an older release).
- Added: pg_sift: an upgrade preflight, `scripts/upgrade-check.sh [--to RELEASE] [--strict] DATABASE...` (`--all` for every database), which runs `sift.upgrade_check()` (`scripts/upgrade_check.sql`, loaded in a transaction that is rolled back, so a database whose extension is older can be checked) and prints one row per finding with its status (`ok`, `warn`, `blocker`) and advice: the extension against the library, the metapage format and column formats of every `sift` index, `sift_col` index and `sift_table` and the first release that reads them (`--to 3.1` makes what 3.1 cannot read a blocker, with the `sift.col_format_minor` and the `REINDEX` or rewrite that write what it reads), options that wait for a `REINDEX` (`cube_minmax` and JSON `paths` on older units, the deprecated `mutable_limit`), settings in the server, `ALTER DATABASE` and `ALTER ROLE` that the library does not define, that switch a feature off or that keep an older column format, and test-only objects left behind. `sift.index_format(rel)` is the one new library function (the metapage format of a relation). No format change. Test `upgrade_check`.
- Added: pg_sift: `superunit` (`sift_col` reloption, 0 = off, the default, or 2 to 256): the merged sums and cube of every group of that many consecutive units of a segment are stored once more (0.5% to 1.5% of the index), and a scan whose units are all exact answers a group from that one summary: `GROUP BY` a 20-value key with `count` and `sum` over 400,000 rows (391 units, groups of 64): 3.09 ms from the units, 0.22 ms from the groups, and 0.18 to 0.22 ms from 25,000 to 400,000 rows where the units go from 0.34 to 3.09 ms. A group with a page written since the last VACUUM, a predicate that cuts through it or a shape its summary lacks (a float sum) is read as units: the rows are the same (`sift.agg_super = off` is the units' answer). Column format minor 12, written only for an index with the option under `sift.col_format_minor = 12` (the default; 3, 4, 6, 7, 8, 9 and 10 keep the formats their libraries read): an index without the option writes the bytes it wrote before. A 3.2 library refuses a minor 8 segment with the REINDEX hint. Units written before the option have no summary; `REINDEX` or a merge writes them.
- Fixed: pg_sift: a query that casts the member of a `jsonb_each` to two types (`sum(CAST(e.value AS integer)), avg(CAST(e.value AS float8))`) next to a value the leaf cannot store raised the message of the float8 cast where the normal plan raises the one of the cast it evaluates first (`integer`); such a query keeps the normal plan. Several dirty rows with different errors raise the one each plan reaches first. Test `agg_member_cast_dirty`.
- Changed: pg_sift: `LEFT JOIN LATERAL jsonb_each(doc) r ON r.key = 'k'` (the member of one key) runs in SiftAgg on PostgreSQL 17 too, as it does on 18, while no unit may hold a JSON null there: the cast of a JSON null raises before 18 where the scan reads NULL, so a unit with one keeps the normal plan (the same guard as the casts of a path). No format change. Tests `agg_left_each`, `agg_dirty_semantics`, `agg_correlated_fanout` (their PG17 expected file is gone).
- Changed: pg_sift: a comparison, `IN` list or `IS NULL` test of the cast of a `jsonb_each` member (`WHERE CAST(e.value AS float8) > 3`) next to an aggregate that casts it no longer keeps the normal plan when some unit holds a value the leaf cannot store: the row read again raises the cast error before the test, as the normal plan does for every member that reaches it. Another test of the members, a join or a conjunct that may raise in the same query keeps the normal plan. No format change. Test `agg_member_cast_dirty`.
- Fixed: pg_sift: the first write of a `sift_table` writes the metapage and then the directory block; a session that looked at the file in between found one block and failed reading the second (a read error, no damage), and a second first writer skipped the layout for the same reason. A file shorter than both blocks is a table not laid out yet. Test `009_tbl_concurrent` (the state is made by cutting the file after a clean stop).
- Changed: pg_sift: a group key that is an `int8` or a `numeric(p,s)` column (not an expression of it) is answered from the cube's histogram of the column, as an `int4` or text key is, where the units were scanned: 200,000 rows, `GROUP BY` an int8 or a numeric key with `count`, `sum`: 1.9 to 2.2 ms scanned, 0.19 to 0.22 ms from the histogram.
- Added: pg_sift: `sift.table_async_promote` (off by default): the backend of a `sift_table` that closes a write buffer asks a background worker (the one of `sift.table_auto_defrag`, `sift.async_maintenance` on) to promote it, freeze the settled groups and merge, where it did that itself. Single-row autocommit `INSERT`s, 100,000 rows, one client, 4 MB buffers: the two statements that closed a buffer took 25-31 ms (about 55 ms in all) and now at most 2.8 ms; the median, p99 and p99.9 are unchanged. From half of the directory on the writer still settles itself. No format change. Test `018_tbl_async_promote` (a heap twin, a crash while the writers and the worker run, `sift.async_maintenance` off).
- Added: pg_sift: `sift.table_backpressure` (0 = off, the default, or a percent above 50): a writer of a `sift_table` that needs a new write buffer while the directory is past that percent waits for the maintenance lock and merges (up to 16 steps instead of 4) before it goes on, where it waited only from the last two entries. No format change. Isolation spec `tbl_backpressure` (a session holding the maintenance lock), `tbl_cap`.
- Changed: pg_sift: `sift_table` keeps up to four fetch scans, one per superblock an index scan or join fetches from, where it kept one that decoded a whole unit again each time the fetches moved to another superblock (an index scan over updated rows, whose versions sit in two): a self-insert scan over 16,000 rows of updated 1 KB rows takes 0.2 s where it took 9.9 s (PG18).
- Added: pg_sift: a `sift_table` stores, with the cube of every unit, the calendar cells of each `date`, `timestamp` and `timestamptz` column (column format minor 8, written under `sift.col_format_minor` 8 or more; a 3.2 library refuses such a segment, 7 or less writes none): the groups of its days, else of its months, else of its years, the finest that fits 1,024 cells of 8 rows or more and 5% of the unit's bytes, the grain recorded in each unit (a `timestamptz` is cut in UTC days). `date_trunc` (year to week and day), `extract` and `date_part` (year, quarter, month, week, isoyear, dow, isodow, doy, day), `::date` and their combinations as the `GROUP BY` key, a range on the column, and `extract(...) = k` or `IN` on a date are answered from the cells, without a stored generated column, where every row was read; a key that changes within a day (`hour`) or reads the session time zone of a `timestamptz` (write `AT TIME ZONE 'UTC'`) scans, and says why in `sift.col_rollup_misses`. Opt out: `SET sift.col_format_minor = 7` before the writes. A `sift_col` index asks for the same cells with the reloption `calendar_columns = 'ts, d'` or `'all'` (default off, `ALTER INDEX ... SET`; only a schema without JSON paths). Details, grains and sizes: docs/COL-PREAGG.md section 2.
- Added: pg_sift: `rollup_dims` (`sift_col` reloption, default off, `rollup_dims = 'kind, sub'` or `'kind, sub: amount'`): every unit written afterwards also holds the rollup of its rows by two to six scalar columns, a cell for each distinct combination with its rows and, per integer or numeric measure, the count, the exact sum, the minimum and the maximum. `GROUP BY` of two or more of them, a test of one with a group by another, and a time bucket with a column are answered from the cells of the exact units (`Pre-aggregate Rollup: rows`, `Units From Pre-aggregates` in `EXPLAIN ANALYZE`), where the one-key cube scanned them: 200,000 rows, `GROUP BY kind, sub` 2.1 ms scanned, 0.63 ms from the cells; `GROUP BY month, kind` 3.2 ms, 0.41 ms; the index grows 0.9% to 4.6% for two low-cardinality columns and 19% for three. A unit keeps at most a tenth of its rows in cells (64 to 4,096): a column of 300 or more values stores nothing and the units scan, with the same answer. Column format minor 9, written only for an index with the option under `sift.col_format_minor = 9` (the default; 8 and lower ignore it and write what a 3.2, 3.1 and 3.0 library read): an index without the option writes the bytes it wrote before. A 3.2 library refuses a minor 9 segment with the REINDEX hint. Units written before the option scan; `REINDEX`, a merge or `sift.col_refresh` writes the cells.
- Added: pg_sift: `cube_minmax` (`sift_col` reloption, default off): the cube of every unit written afterwards also holds the minimum and maximum of each integer measure per group, and `min()`/`max()` of a measure grouped by a dim of the cube are answered from the groups of the exact units (200,000 rows: 2.6 ms scanned, 0.23 ms from the groups; the index grows 1.0% to 3.0%). Column format minor 7, written only for an index with the option under `sift.col_format_minor = 7` (the default; 3, 4 and 6 keep the formats a 3.2, 3.1 and 3.0 library reads): an index without the option writes the bytes it wrote before. A 3.2 library refuses a minor 7 segment with the REINDEX hint. Units written before the option scan; `REINDEX` or a merge writes them.
- Added: pg_sift: the arms of a two-level aggregate (`UNION ALL` of per-row aggregates under an outer aggregate) can be joined to a spine of buckets, a `VALUES` list, a series, a small table or a sub-select, with `spine JOIN arms ON m.key = s.key` or `spine LEFT JOIN arms ON ..` (also `IS NOT DISTINCT FROM`, or `jsonb_build_array(..) = jsonb_build_array(..)` on several columns): `Custom Scan (SiftAgg)` reads the spine's rows once, adds each cell of the scan to the group of every spine row its keys equal, and the spine's columns are group keys; with `LEFT JOIN` a spine row no cell matched is a group of NULL arm columns with `count(*)` 1, so empty buckets stay. The normal plan joined every per-row group with the spine before grouping: 200,000 documents with a 3-key lookup in the arms and a two-row spine, 23,004 ms in the normal plan and 23,391 ms with the arms scanned alone, 205 ms as one scan (25,000: 1,114 / 1,121 / 40 ms; 50,000: 1,757 / 1,736 / 68; 100,000: 3,172 / 3,150 / 105; median of 3, one backend, busy host, scan chosen with `seq_page_cost = random_page_cost = 1000`). Any other join, a condition that is no equality, a WHERE that reads the arms and a spine of more than 10,000 rows keep the normal plan. Test `agg_two_level_spine`.
- Changed: pg_sift: documented and tested: a `LIMIT 1` child over a key the table repeats, without an `ORDER BY` that makes the order total, keeps the child, since the row it returns is the first its plan meets and the index and the heap order differ for a row updated in place (`agg_first_row_lookup`).
- Changed: pg_sift: the planner counts the values a column of a correlated child can take as the child's rows for each distinct input of its own lookup, not as one: a second `LEFT JOIN LATERAL (.. LIMIT 1)` keyed by a column of the first's row was priced for a single input and kept its child, which then ran for every distinct value. Both laterals are now lookup tables when their tables are small against their inputs (two keys then a text key, 60,000 facts with 60,000 distinct inputs, median of 3, warm, 2026-10-22, PostgreSQL 18.6: tables of 25k / 50k / 100k / 200k rows 19.7 / 34.9 / 44.6 / 33.6 ms, were 28.0 / 60.7 / 79.8 / 72.6 ms with the second child run per input; 72 / 108 / 202 / 162 ms in the normal plan, host busy).
- Changed: pg_sift: a `LIMIT 1` child with an `ORDER BY` (`.. WHERE t.a = x AND t.b = y ORDER BY t.ts DESC LIMIT 1`, the latest row of a key) is read as a first-row lookup when a unique index holds the key and the order columns, so that no two rows tie: up to three integer, date or timestamp columns, either direction, `NULLS FIRST` or `LAST`; an order column the index holds must be `NOT NULL` (a primary key in the order does it). The lookup table keeps the first row of each key by the order (among the rows that pass a condition on the row, as in the previous entry). A repeated version, a text or expression order, a nullable order column under a unique index and more than three columns keep the child. Two keys over three versions of each, 60,000 facts with 60,000 distinct inputs, median of 3, warm, 2026-10-22, PostgreSQL 18.6: tables of 25k / 50k / 100k / 200k rows 6.4 / 11.4 / 18.8 / 13.2 ms against 36.8 / 65.6 / 79.9 / 84.0 ms in the normal plan.
- Changed: pg_sift: a `LIMIT 1` child that tests its row besides the unique key (`.. WHERE t.a = x AND t.b = y AND t.score > 100 LIMIT 1`: comparisons of columns with constants, null tests, `AND`/`OR`/`NOT`) is read as a first-row lookup too: the lookup table is loaded with the rows that pass, and a key whose row fails has none, as the child returns none. A condition that can raise an error, a list, a function or the outer row's value keeps the child. Three keys and a condition, 60,000 facts with 60,000 distinct inputs (median of 3, warm, 2026-10-22, PostgreSQL 18.6): table of 25k / 50k / 100k / 200k rows 12.6 / 20.5 / 27.4 / 52.2 ms against 52 / 111 / 145 / 191 ms in the normal plan.
- Changed: pg_sift: the planner prices a row of a table read as the first-row lookup of a `LIMIT 1` child at what it costs (0.2 microseconds to hash, against 1.6 for a run of the child), not as a row of any joined table: the lookup is taken when the table has fewer than about 8 rows for each distinct input, where it was 2. Three keys, 60,000 facts with 60,000 distinct inputs (median of 3, warm, 2026-10-22, PostgreSQL 18.6): a 200,000-row table 32.6 ms (143 ms in the normal plan, about 100 ms when the child ran in the scan), 100,000 rows 25 ms (105 ms), 50,000 rows 15 ms (75 ms), 25,000 rows 9 ms (45 ms). Few inputs keep the child.
- Changed: pg_sift: a `SiftAgg` path reports to the plan above the groups of every row the lookup lists of the scan can pass, where PostgreSQL's count of a list is a default guess (the rows of a sub-select or a CTE, a filter on a jsonb path): a CTE of one row no longer makes a join of the aggregate to a grouped copy of itself on a null-safe key run that copy's aggregate again for each row. The price of the path is not changed; the plan above it is now priced for those groups, so the scan is taken where that plan was a nested loop priced for one row (the neutral shape at 50k / 100k / 200k rows: groups estimated 50,000 / 100,000 / 200,000 against 16,667 / 33,333 / 66,667 made, where it was 2 / 5 / 10).
- Changed: pg_sift: the planner prices the opening of the partitions of a partitioned table, which grows faster than their number (the scan takes 0.75 ms more than the normal plan's `Append` at 200 partitions, 9 ms at 800, 27 ms at 1,600 and 111 ms at 3,200, 11 ns for each pair of partitions), where it priced the scan as if it opened one source. 200,000 rows in 100 / 400 / 1,200 / 3,200 partitions, `GROUP BY` of one column, planning plus execution (median of 5, warm, 2026-10-22, PostgreSQL 18.6, a busy host): 100 and 400 keep the scan (3.3 ms against 34.5 ms, 22.8 against 39.5), 1,200 now takes the normal plan (46 to 52 ms, where the scan took 53 to 57), 3,200 takes it too (106 to 148 ms, where the scan took 366 to 450). A plan that prunes to a few partitions at plan time pays for those only.
- Fixed: pg_sift: a filter on a joined table that sits in the `WHERE` of a sub-select on the nullable side of a `LEFT JOIN` from a spine (the periods of a report) filters the rows of the scan again before the match, where PostgreSQL's move of that `WHERE` into the join's `ON` clause made it a test of the spine's lookup: the joined table was read whole, and a filter that matches nothing no longer ended the scan, which read every unit (an events table joined to a dimension table whose filter matches no row, two periods: 25k / 100k / 200k rows 2.7 / 4.2 / 7.8 ms and 2 / 8 / 16 units, now 0.05 / 0.06 / 0.04 ms and none, as the normal plan's 0.04 ms; median of 3, warm, 2026-10-22, PostgreSQL 18.6). A condition that reads the spine stays in the match.
- Changed: pg_sift: SiftAgg reads a partitioned table also when a parameter of a generic plan or a stable function (`d >= $1`, `k = ANY ($1)`, `ts >= now() - interval '30 days'`) prunes its partitions when the plan starts: the scan evaluates the operand when it starts and does not read the partitions whose bounds it refutes (`Partitions Removed` in `EXPLAIN ANALYZE`; parallel workers take the leader's superblocks). Before, a parameter on the partition key kept the normal plan, and a stable bound read every partition. A table split by hash on a column compared with a parameter keeps the normal plan.
- Added: pg_sift: SiftAgg joins a lateral sub-select, a `UNION` / `UNION ALL` of arms, `generate_series`, `unnest` of an array column or a set-returning function that reads the joined row. PostgreSQL's parameterized plan of it is the child, run once per distinct input, with duplicates, NULL-extended rows and errors as in the normal plan. A call of a document column that cannot raise (`jsonb_exists`, `->>`, `jsonb_typeof`) takes the index expression column of that call as the parameter. A `UNION ALL` / `UNION` of sub-selects, a sub-select of a statement-level CTE and a `WITH` inside the sub-select are lookup plans too.
- Added: pg_sift: the rows of a joined table can be kept in shared memory for the other backends (`sift.agg_lookup_shared_kb`, 0 = off by default, superuser; dynamic shared memory taken as entries are kept, no restart or preload): a backend that loads a table writes its rows once in the flat form of a parallel scan, and a backend that finds the table unchanged (the check of `sift.agg_lookup_cache_kb`: all pages visible, same LSNs) reads them in place, so a fresh connection pays the attach, not the load. First query of a fresh backend, 60,000-row fact table, one integer key, load / shared read: 25,000 mapping rows 11.1 / 9.8 ms, 50,000 13.1 / 10.4, 100,000 21.5 / 12.1, 200,000 35.6 / 17.0; three text keys 18.7 / 14.2, 24.0 / 16.4, 45.6 / 18.1, 52.0 / 25.9 (the attach itself is 0.6 ms at 200,000 rows); eight fresh connections at once, 200,000 rows: slowest 59.5 ms, 13.5 ms with the entry there. The backend that writes the entry pays for it once (200,000 rows: +16 ms). Tests `agg_lookup_shared`, isolation spec `agg_lookup_shared`, TAP test `018_lookup_shared` (a backend killed while it reads an entry).
- Fixed: `make check` on macOS. The stock bash 3.2 failed on the syntax of bash 4 and `make check` exited 0 with no test run; it now stops with `needs bash 4+ (brew install bash)`, a run that selects no job fails, and `scripts/cpu_time.py` and the job claim no longer need Linux (`prctl`, `flock`). `scripts/ci.sh --selftest` checks that a script with bash 4 constructs carries that first-line check. CI builds the Homebrew formula and runs `brew test` and a few regression tests on macOS arm64.
- Changed: with the directory of a `sift` or `sift_col` index full and the merges failing, a forced seal (`sift.promote()`, `sift.merge(index, true)`) leaves the open write buffer open with a NOTICE instead of failing with "has 128 segments", and the error of the refresh that finds no write buffer says what the last failed step was; the limit is `max_segments` of `sift.test_limits` in a test build. Test `dir_full` (isolation).
- Added: `sift.workers`, one row per `sift` and `sift_col` index with the segments, the sealed write buffers waiting, whether maintenance is running, and, when the last background step failed, the failures in a row, when, the earliest retry, the SQLSTATE and the message (the log was the only place; the metapage keeps the last one in space the page had: no format minor, builds without it ignore it and read pages without it as no failure).
- Changed: `sift.standby_conflicts` is on by default: the primary logs a recovery conflict when it reuses blocks a merge freed, so a hot standby without `hot_standby_feedback` waits for its long query up to `max_standby_streaming_delay` and then cancels it, as for a heap query, where it failed at once with "segment recycled during scan". `off` restores that. Test `maint_standby`.
- Changed: `sift.col_auto_merge` promotes the write buffer of a `sift_col` index and leaves its segments when the buffer is the only reason (an insert-only table no longer has its whole index rewritten at every 1% of new rows: 12 cycles of 2% inserts on 200,000 rows wrote 4.6 MB of segments, from 33.4 MB, with the same query times); new settings `sift.col_auto_merge_wal_pct` (the rewrite of all segments waits until the server has written 100 / that percent times the bytes of the index since the index was last written) and `sift.col_auto_merge_max_active` (it waits, with a backoff, while that many client backends run a statement); the churn simulator `bench/churn/churn.sh` measures them.
- Changed: when the filters of the source keep under a 32nd of a unit's rows (a scope of one to five keys that are spread over every unit), `SiftAgg` goes on with the unit as those rows alone, renumbered as in the unit of a match set, where the leaves, the offsets and owners of the elements of a list and the aggregates still worked over the whole unit. A one-key scope over 25k-200k rows took 0.17-0.40 ms (scalar columns) and 0.33-2.4 ms (the elements of a list) and now takes 0.11-0.33 and 0.24-1.3 ms, against 0.08-0.24 and 0.12-0.55 ms for the index probes of the normal plan; five keys at 200k rows 0.90 to 0.75 ms and 2.5 to 1.8 ms. Ten keys and more, and the units with rows read again from the heap, are unchanged. No on-disk change.
- Fixed: pg_sift: the first statement of a session that has not preloaded the library ran the ordinary plan when it had bound parameters (the extended protocol of a driver, `pgbench -M extended` or `-M prepared`, `EXECUTE` of a prepared statement with arguments): 220 to 260 ms against 3 to 5 ms for the next one, with no `Custom Scan (SiftAgg)` in the plan. The executor start plans it again through the hook with the parameters' types and values, as it did for a statement without parameters. A query inside a function keeps its plan. `agg_first_statement` covers both.
- Fixed: `sift_table`: an `UPDATE` or `DELETE` that scans a table while a `VACUUM` (or the background defrag) replaces its tombstone log failed with `failed to fetch tuple being updated`; the statement now keeps the tombstones it already holds (the bug is also in 3.2.3)
- Fixed: `sift_table`: a bitmap-scan `UPDATE` or `DELETE` that a `VACUUM` interrupted by promoting the write buffer silently skipped the rows of the promoted blocks (1530 of 3000 rows updated, no error); the scan now reads the write buffers it began with (the bug is also in 3.2.3)
- Fixed: pg_sift: a `GROUP BY` of an expression of a `date` column (`extract(dow FROM d)`, `d + 1`, `date_trunc('month', d)`) failed with `invalid memory alloc request size 17179869248` when the column held both `infinity` and `-infinity`, which are the two ends of the 32-bit range: the scan asked for a lookup table over all of it. Such a key now keeps its ids in a hash table, as a key of an integer column does, and the rows are those of the normal plan; the bug is also in 3.2.3, where the error reads `a column range of 4294967296 values is too wide for the aggregate scan`.
- Fixed: pg_sift: a `GROUP BY` on a `date` column whose selected rows were all `infinity` (or all `-infinity`) stopped the backend: the range table of the group keys stepped over the end of the 32-bit range and wrote far outside the table (also in 3.2.3 and earlier; found while fixing the line above).
- Changed: pg_sift: the first row of each group asked with a window function (`WHERE rn = 1`, `rn <= 1` or `rn < 2` over a sub-select or a CTE read once, with the one window function `row_number() OVER (PARTITION BY .. ORDER BY ..)`) is also planned as `SELECT DISTINCT ON (..) .. ORDER BY ..`, and runs in the scan, where the order is a total one, instead of sorting every row through the window. A window it leaves as written (`rank()`, another window function, no `PARTITION BY` or `ORDER BY`, a sub-select that groups or limits) is named `window_first` by `sift.explain_why`.
- Changed: pg_sift: an aggregate of one table that a `LATERAL` sub-select or a scalar sub-select ties to the outer query by equalities (`(SELECT sum(f.a) FROM f WHERE f.k = o.k)`) is also planned as the table grouped by those columns and joined to the outer query, and that plan is kept when it runs in the scan and costs less than a lookup of the group for each outer row: a few outer rows keep the lookups. An outer row whose key has no row, or is NULL, gets 0 for a `count` and NULL for every other aggregate, as before; a sub-select that groups, limits, joins, reads the outer query outside the equalities or has an aggregate with a value over no rows stays as written.

## 3.2.3 - 2026-10-23

- Fixed: pg_sift: three reads and writes of memory the relation cache can free, found with `debug_discard_caches` and AddressSanitizer. A catalog read can reload the cached state of an index (`rd_amcache`) while an index function still holds it: the insert of a row that flags a leaf (a value the index cannot store, or a JSON null) wrote its write-buffer hint there after `sift_dir_update` (a segmentation fault in `col_index_crash`, or `could not open file ... target block`); the heap check of `sift.col_verify(index, true)` and the building of an index row of a table with a toasted value read the state after the detoast. Each now works on a copy or reads the fields it needs first. The effect is a backend crash (the server restarts every session) or corrupted heap memory in a narrow window, which a concurrent `VACUUM`, `ANALYZE` or `ALTER` of the table can open; no privilege is needed to be exposed to it. New tests: `col_insert_discard`, `col_row_relcache`.
- Fixed: pg_sift: the first statement of every new backend ran the ordinary plan: the library loads while the planner opens the first column index, after the planner hook had its chance, so the CTE-inline and lateral-flatten decisions were skipped (a statement of two arms over one CTE: 233 ms on the first statement of a backend, 3.7 ms on the second, rows equal). The executor start now plans a plain `SELECT` again through the hook, once per backend, when the hook never ran. Pools that recycle connections meet the case on every new backend; `session_preload_libraries = 'pg_sift'` avoided it. Not fixed in 3.2.3: a first statement with bound parameters (the extended protocol of a driver, `EXECUTE` with arguments, `pgbench -M extended` and `-M prepared`) still plans without the hook, once per backend (220 to 260 ms against 3 to 5 ms for the next statement); `session_preload_libraries = 'pg_sift'` or `shared_preload_libraries = 'pg_sift'` avoids it. New test: `agg_first_statement`.
- Fixed: macOS: `scripts/tap.sh`, and so every Perl TAP test, stopped on the BSD `sha256sum` (`-c --quiet` on standard input); `scripts/sha256.sh` runs GNU `sha256sum` or `shasum -a 256`, and `scripts/ci.sh --selftest` fails on a script or test that calls the system checksum tool directly. The SQLite build's portable-list filter reads `#` in a way GNU make 3.81, the stock macOS make, accepts (it stopped `make check` with "unterminated call to function filter-out").
- Fixed: `make check` on macOS. The stock bash 3.2 failed on the syntax of bash 4 and `make check` exited 0 with no test run; it now stops with `needs bash 4+ (brew install bash)`, a run that selects no job fails, and `scripts/cpu_time.py` and the job claim no longer need Linux (`prctl`, `flock`). `scripts/ci.sh --selftest` checks that a script with bash 4 constructs carries that first-line check.

## 3.2.2 - 2026-10-22

- The store size rows show the index or columnar data with the heap beside it; README.md drops the places table, the Vulkan table and the rows that are sums of others.
- Fixed: pg_sift: security: with row-level security on a table, the planner could estimate the rows of a partition or an inheritance child of it from the statistics of the `sift` index, which count the rows the policy hides, so the row counts of `EXPLAIN` (and the plan chosen) for a role under the policy could reveal information about the hidden rows (how many of them hold a term). The estimate is the planner's default now, as it already was for a table without partitions and as the documentation says ("no index estimate" under row-level security); the refusal of `sift.score()` on such a partition or child names the table under the policy (it said "needs a sift index on every partition"). No data was modified and no query result was affected: the rows a policy hides were never returned. Affected: 3.0.0 to 3.2.1 (3.2.0 and 3.2.1 tested, 3.0 and 3.1 by reading the code), a server where a role without `BYPASSRLS` that is not the owner reads, through its parent, a partitioned or inheritance table with a policy whose partitions or children carry a `sift` index. Nothing is needed before upgrading beyond installing the new library (no setting, no `REINDEX`, no change of the extension SQL). Test `rls_stats`.
- Fixed: pg_sift: the aggregate pushdown (`Custom Scan (SiftAgg)`) no longer refuses what the normal plan answers (3.2.0 and 3.2.1): a role with `SELECT` on the columns a query reads but not on the whole table, or on a view and not on its table, got `permission denied for table` at run time. The scan asked for table-level `SELECT` of the invoking role, where PostgreSQL has checked the columns, and the view owner's rights, before the scan starts; the scan's own check is gone (a partitioned table too), its row-level security check stays. A view over a table with row-level security keeps the normal plan.
- Known issue: pg_sift: in 3.2.0, 3.2.1 and 3.2.2 a role that is neither a superuser nor the owner of the extension cannot drop anything in a database where pg_sift is installed: the event trigger function `sift.table_paths_drop()` is not `SECURITY DEFINER`, so every `DROP` of that role (a table, index or schema, with or without pg_sift) fails with `permission denied for schema sift`. The extension SQL cannot change in a 3.2 patch release; `ALTER EXTENSION pg_sift UPDATE` to 3.3 fixes it. Until then, run once in each database with the extension, as a superuser (and again after restoring a dump, which creates the extension from the 3.2 script): `ALTER FUNCTION sift.table_paths_drop() SECURITY DEFINER SET search_path = pg_catalog, pg_temp;` (`docs/UPGRADING.md`).

## 3.2.1 - 2026-10-22

- Added: bench: `bench/bench duckdb` measures pg_duckdb (the official release image, DuckDB reading the heap and a Parquet file it writes) against pg_sift and PostgreSQL on the store benchmark's orders and ten questions, in two containers with equal limits, warm and cold; the questions pg_duckdb cannot run are reported as not supported.
- Fixed: pg_sift: a set filter (`col = ANY($1)`, `col IN (..)`) of thousands of ids was slower in 3.2.0 than in 3.1.0 (3M facts, 100,000 ids: generic plan 10 to 125 ms, literal list 121 to 238 ms; 10,000 ids 5 to 15 ms). 3.2.0 sent the sorted members of every `IN` list with gaps along with its zone test, and the column scan rebuilt that test (copy, sort) at each range of units it began; a list of more than 256 members meets the zone of nearly every unit and is tested by its range again, as in 3.1 (a list of widely spread keys in the hash table keeps its members). Same data, host and run (7 runs, median): 100,000 ids generic plan / custom plan / literal list 9.8 / 110 / 123 ms, 10,000 ids 4.5 / 14 / 16 ms; `IN (SELECT ..)` of 100,000 members 141 to 27 ms. Test `agg_zone_or`.
- Fixed: pg_sift: a column segment written under `sift.col_format_minor = 4` or 5 (the setting for readers of 3.1) carried the zone flag of a JSON null that only 3.2 knows, so `sift.col_verify` of pg_sift 3.1 reported such an index as corrupt (its answers were right); the flag is now written from minor 6 on. Test `col_zone_jnull`.
- Fixed: pg_sift: a `LEFT JOIN LATERAL (.. LIMIT 1)` child read as a first-row lookup table (3.2.0) lost every group when the query summed or averaged a `float8` column and grouped by a column of the lookup row: the scan decided its lane order for float sums before the child became a table, so the groups were merged into a result that was never read, and the query returned no rows where the normal plan returned them. `count`, `min`, `max`, integer sums and a group key from the facts were not affected. Test `agg_first_row_lookup` (every aggregate kind grouped by the lookup row).
- Fixed: pg_sift: a session that read a new `sift_table` while its first writer was between writing the metapage and the tombstone segment cached the table as laid out; its own first write then added a write buffer to a directory that was never flagged as a `sift_table`, and every later read of the table failed with "The metapage does not carry the sift_table flag". The first write now checks that the segment exists, not only that the table was read.
- Fixed: pg_sift: an `ORDER BY .. LIMIT` read from the column units no longer fails with `sift: top-N row (p,o) is not in the heap` after an `UPDATE` that leaves every indexed value as it was (a HOT update): the index entry names the root of the HOT chain, and the scan now reads the row as an index scan does, through the chain. Test `col_topn_hot`.
- Fixed: pg_sift: on PostgreSQL 18, two `EXISTS` (or `IN`) over the same table by its unique key with different conditions (`EXISTS (.. i.id = e.parent_id AND i.kind = 'k1') AND EXISTS (.. i.id = e.parent_id AND i.kind = 'k4')`) are merged by the planner into one scan of the table (self-join elimination) and `SiftAgg` applied the conditions of one of them: a count of 1,200 where the rows are 0. A semi, anti or preserved join whose table the planner removed this way keeps the normal plan. Test `agg_semi_same_table`.
- Fixed: pg_sift: a row inserted into a `sift_table` could be lost with its insert reported done. The writer that closes a write buffer swaps it in the directory and flags its pages sealed afterwards; a session that held a page of the closed buffer kept appending to it until then, and the promotion (VACUUM, `sift.table_maintain`, a worker) that ran in between released the page with the row on it. The promotion now flags every page sealed before it reads the buffer, as the promotion of a BM25 or column write buffer does. Test `009_tbl_concurrent` (`sift.table_seal_gate` holds the closing writer before it flags). No format change; present in 3.2.0.
- Fixed: pg_sift: a parallel worker of a `SiftAgg` scan over a partitioned table crashed when an item of its work lay in a partition other than the first: the leader hands the workers the superblock offsets of every partition, but a worker opened the first partition only, so the item ran on a source that was not there. A worker now opens the partition it moves to. Test `agg_par_partitions`.
- Fixed: pg_sift: every query on a table with a `sift` index scanned `pg_opclass` several times while it was planned (3.2.0): the check of a `sift.score(ctid) > c` cutoff read the operator class of `float8` before it looked at the operands of each clause. A short ranked query cost 6% to 23% more user instructions than in 3.1.0 (20,000 rows in one segment: `and` 1.23x, `phrase` 1.19x, `count` 1.14x, `rank` 1.11x, `or` 1.07x; 61 segments 1.01x to 1.03x), the loss of queries per second the 3.2.0 benchmark saw on `phrase` and `and`; now 1.00x to 1.02x (`bench/steps/cpu_gate.sh`). Test `rank_score_filter`.

## 3.2.0 - 2026-10-22

- Changed: pg_sift: a `LEFT JOIN LATERAL (SELECT .. FROM t WHERE t.a = x AND t.b = y LIMIT 1)` over a table with a unique index on the columns it compares is read by SiftAgg as a lookup table (its rows hashed by key, kept for the next query like any other lookup table) instead of running the child for every distinct input: the key has one row, so the row is the one any plan returns. The planner prices both and takes the cheaper; a key the table repeats, a condition, an order, an offset or an expression over the parameter keep the child (`EXPLAIN ANALYZE` shows `First Row Lookup Tables`). Three keys over a 100,000-row table and 60,000 facts: 47 ms (269 ms in the normal plan, 167 ms in a scan that ran the child).
- Changed: pg_sift: the arms of a `UNION ALL` that join the same lookup table and read the same rows run in one pass over the index also when they ask for other aggregates: the pass computes the union of them and each arm's set updates its own, the lookup being resolved once (three arms over a lookup join, 100,000 documents: 18.7 ms scanned arm by arm, 8.6 ms in one pass; `sift.agg_fuse = off` runs the arms alone).
- Changed: pg_sift: `sift_table` reads the directory again only when its version changed, where every row operation re-read it (and asked the file system for its length): `UPDATE` of 10% of 200,000 rows takes 171 ms instead of 134, `DELETE` 93 ms instead of 85 (median of 5, one client, 2026-10-21, PostgreSQL 18.6; the host was busy). No format change.
- Changed: pg_sift: appending a record to a write buffer page (`sift_col`, `sift` and `sift_table`) and a tombstone (`DELETE`, `UPDATE` of a `sift_table`) log the changed bytes of the page directly instead of copying the page, comparing it and copying it back for the generic WAL record, which is the same record and is replayed as before: `INSERT .. SELECT` of 200,000 rows into a `sift_table` takes 320 ms instead of 366, a loop of 100,000 single-row `INSERT`s 312 ms instead of 354, `DELETE` of 10% of the rows 14% faster, and 1 to 4% less into `sift_col` and `sift` indexes (median of 5, one client, 2026-10-21, PostgreSQL 18.6; the host was busy). No format change.
- Changed: pg_sift: the arms of a `UNION ALL` that read the same rows for the same aggregates and join the same lookup table run in one pass over the index also when their `WHERE` differs: the conjuncts every arm has are the pass's, what an arm has beyond them is tested on the rows of its own set, and the units and rows no arm needs are not read (a pass whose arms the pre-aggregates answer one by one is not made). `sift.agg_fuse = off` runs each arm alone.
- Changed: pg_sift: a row of a unit that holds a value a path cannot store is read again from the heap only if the conditions the scan can test on the stored unit (a key set, a comparison or NOT NULL on a stored column) let it through, and not at all when they reject it: a query over 200,000 rows with 4.3% such rows, a scope of 51 keys and a rare NOT NULL column reads 7 rows again, not 8,696 (8.5 to 2.2 ms, one run each). The rows and errors are those of the normal plan; no format change.
- Changed: pg_sift: the normal plan of a lookup join PostgreSQL can count (`IN (SELECT ..)`, a join on a list table) is priced for the rows the column's most common values own when the column is more than twice as uneven as a flat one, not for the mean rows of a key, so `Custom Scan (SiftAgg)` is chosen for a short list of heavy parents (5 parents holding a third of 200k rows: 34 ms against 8.6 ms); a rare list takes the same plan, 5 to 8 ms slower than the normal one at 200k rows (`sift.agg_key_skew = off` gives the mean back).
- Changed: pg_sift: the scan of `SiftAgg` is priced at the speed it runs: its own operators (reads, filters, tuples, keys, aggregate updates, the merge of workers) at 0.2 of the count the cost model gave them (`AGG_SCAN_SPEED`; the shapes it runs alone take 0.4 to 0.9 ns for an operator counted at 3 ns), the arms of a `UNION ALL` under `SiftAggRows` that run as one pass pay the first arm's price and a tenth of the others, the start-up is 0.6 ms and the worker start 22 times `parallel_setup_cost` (was 110, scaled with the scan). Shapes that were 8 to 35 times faster as a scan but lost the cost comparison (a free-key member by month, periods from a `VALUES` list, a count of an unnest) now take it; a few rows of one parent keep the btree plan.
- Changed: pg_sift: the normal plan of a query whose scope is a lookup PostgreSQL cannot count (`IN (SELECT ..)` or a join on a table filtered by a jsonb path or an expression, a sub-select, a CTE) is priced for the keys the scanned column can hold instead of its one-row guess, so SiftAgg is chosen when the list is long; a lookup whose filters have statistics, a VALUES list and a list of constants keep their estimate (`sift.agg_param_set_size = 0` turns it off).
- Added: pg_sift: a `sift_table` is defragmented without `VACUUM` (`sift.table_auto_defrag`, off by default in 3.2: an opt-in until it has run on real workloads, `sift.table_auto_defrag_free_pct`, `sift.table_auto_defrag_delay`, `sift.table_auto_defrag_indexes`): the backend that closes a write buffer asks a background worker for a pass when the tombstones reach `sift.table_dead_rewrite_pct` of the rows (a whole table VACUUM, index cleanup included, throttled by the cost delay) or the free blocks of the file reach a quarter of it (`sift.table_maintain`); the worker stays a few seconds and passes again while the writer's rows settle, and returns the free end of the file with the conditional locks of `VACUUM`. Test `017_tbl_autodefrag`.
- Added: pg_sift: `sift.table_maintain(tbl)` does the work of `VACUUM` on a `sift_table`'s column segments and file without its index cleanup: it promotes the write buffers, clears the dead rows older than every snapshot, rewrites the mostly dead segments (`sift.table_dead_rewrite_pct`), gathers the free space and returns the free end of the file, taking the table and the end-of-file lock only when they are free. The tombstone log and the index entries stay for the next `VACUUM`, which still removes them (test `tbl_maintain`, isolation `tbl_maintain`).
- Added: pg_sift: a `sift_col` index that churn scattered is rewritten by a background worker after the `VACUUM` that finds it so (`sift.col_auto_merge`, off by default): the worker seals the write buffer and merges every segment into one, as `sift.col_promote` plus `sift.col_merge(index, true)` do, under the maintenance lock only (readers and writers are not blocked; skipped when the lock is held, never on a standby), sleeping as `sift.maintenance_cost_delay` says and rewriting an index over `sift.vacuum_rewrite_limit` MB by partitions of that size. The rule: more than `sift.col_auto_merge_dirty_pct` (30) percent of the units hold a dead row, or the units listed are `sift.col_auto_merge_fragmentation` (2.0) times the superblocks they cover, or the write buffer holds more than `sift.col_auto_merge_buffer_pct` (1.0) percent of the rows. A 1% DELETE/UPDATE spread over the table used to leave a dead row in every unit and no unit answering from its pre-aggregates, with `dead_rewrite_pct` never reached. `sift.col_maintain(index)` applies the rule in a session (a `pg_cron` job) and `sift.col_maintenance_due(index)` returns the reason. The merges of the maintenance worker of a `sift_col` index now honour `sift.maintenance_cost_delay`, which only the `sift` index's did. No format change. Tests `col_auto_merge` (regress, isolation, TAP).
- Changed: pg_sift: `sift.col_merge(index, true)` seals the open write buffer before it merges, so one call is `col_promote` plus the merge: the rows written since the last promotion go into the one segment it makes. Before, they stayed in the buffer as one in-memory unit per superblock they touched, next to the merged segment (no pre-aggregates, one extra unit each). Without `force` nothing changes. `sift.merge(index, true)` of a `sift` index keeps leaving the buffer open. No format change. Test `col_merge_force`.
- Changed: pg_sift: the planner prices a top-N scan that walks its sort column by the zones (`sift.topn_walk`) with the units the zones leave, from the mean zone width the index stores plus `n` over the expected matches, instead of all of them. 396,000 rows, `ORDER BY key, id LIMIT 10` with the scan forced: the estimated cost of a key in blocks of 12,000, 3,000 and 300 rows falls from 5,143 to 757, 655 and 1,446; a random key stays at 5,142.
- Changed: pg_sift: `sift.table_dead_rewrite_pct` defaults to 30 (was 50): `VACUUM` rewrites a `sift_table` column segment once a third of its rows is dead instead of half, so a table with scattered deletes and updates keeps a file near 1.4 times its live rows instead of up to 2 times (the rewrite costs `VACUUM` about 15-20% more time in a steady delete/update/insert workload). No format change; `SET sift.table_dead_rewrite_pct = 50` restores the old behaviour.
- Fixed: pg_sift: a conjunct on the member of one key of a map leaf (`CAST(jsonb_extract_path(doc, 'k') AS integer) = 8`) next to a `jsonb_each` cast of the members keeps the normal plan when some unit holds a value the leaf cannot store: the scan dropped the row of a dirty member (the filter saw NULL) where the normal plan raises the cast error, with the message of the conjunct's own type. A query without such a conjunct is unchanged.
- Fixed: pg_sift: SiftAgg no longer raises "holds values its index cannot store" where the normal plan answers: a filter, key or aggregate on the member of one key of a map leaf next to a `jsonb_each` cast of the members (a dirty value of another key, in a row the filter drops), the string `-0` in a float column of `jsonb_to_recordset`, and a list of more than 2^20 elements are read again from the heap like the other dirty rows (the string `-0` counts as 0). Index builds over such documents already flag the value and never fail.
- Changed: pg_sift: the rows of a joined table are kept whatever the size of its heap: the memory of the rows (half of `sift.agg_lookup_cache_kb` at most) decides, not a limit of 32,768 pages. Reading a table's pages to check it has not changed costs about 0.4 microseconds a page, a twentieth of loading its rows (25,000 rows: 3.3 ms to load, 0.09 ms to check; 200,000 rows: 27 ms and 0.47 ms). A table whose rows did not fit is remembered, and not read page by page on every query.
- Changed: pg_sift: the arms of a statement that read different columns of the same joined table share one kept entry of its rows (the union of the columns), instead of one entry per arm: two arms over a 500,000-row mapping table keep 13.6 MB, not 23.3 MB.
- Changed: pg_sift: a lookup join keyed by a text column translates each value of the leaf to the table's key value once per scan, not once per combination of values it is in.
- Changed: pg_sift: the rows a backend keeps of a joined table (`sift.agg_lookup_cache_kb`) take a sixth of the memory: a key or a column value that repeats is stored once and a row holds its id (a 500,000-row mapping table keyed by three text columns: 76 MB, now 12 MB per kept table), and a probe finds the ids of a key's values before it walks the rows. Tests `agg_lookup_rows` and the text key permutation of the isolation spec `agg_lookup_cache`.
- Changed: pg_sift: a lookup join probe allocates the rows its key matches instead of one slot per row of the table (a 500,000-row mapping table: 2 MB for every combination of values): 200,000 combinations of three keys over 260,000 tuples, one backend, 571 to 427 ms with the table cached, 757 to 534 ms loaded for the query.
- Changed: pg_sift: SiftAgg counts `count(DISTINCT x)` of a column the per-row aggregates group by (a parent id under an outer aggregate over `GROUP BY id, parent_id, ..` arms) in the scan, as the distinct ids of that key per outer group (a bitmap, then a hash set, then a sort, within `hash_mem`), instead of leaving the plan to PostgreSQL's sort of every per-row group. Test `agg_arm_distinct`.
- Changed: pg_sift: the cells of per-row aggregates that group by more than the outer keys (a parent id next to the id, an element's tag) are numbered inside their row and no longer in the group table, which then holds the outer groups only: the table that grew with the data and the per-cell arrays that went with it are gone, and the results, their order and the stored data are unchanged. Two per-row arms over 200,000 rows with 442,754 elements and a `count(DISTINCT parent_id)`: 549 ms to 156 ms (median of 3, interleaved, busy host); 25,000 rows: 67 to 20 ms. Test `agg_arm_cells`.
- Changed: pg_sift: the per-row aggregates of an outer aggregate (UNION ALL arms grouped by a row id) may read a joined VALUES list, sub-select or `LIMIT 1` lookup: the outer aggregate runs in the one scan with the joins as lookups. The plans of the joined rows are sub-plans of the arms' query level (planned against its own `PlannerInfo`, not the statement's); `sift.agg_two_level = off` keeps the arms' own scans.
- Changed: pg_sift: a document that a `sift_col` list leaf cannot store (a document that is an array, a list that is a string) no longer keeps the normal plan for an `EXISTS` over the list with a lookup per element (the joined mapping of a report by label) or for an integer cast of a member next to an unnest of the list: the rows are read again from the heap and the error of a list that SQL cannot read is raised for the rows that reach it, after the join of the test or the cast, as the normal plan does.
- Changed: pg_sift: a dirty value no longer keeps the normal plan for the two reads that are exact per member: the member of one key (`LEFT JOIN LATERAL jsonb_each(doc) r ON r.key = 'k'` with `CAST(r.value AS float8 | integer | bigint)`) is read again from the heap only for the rows whose member of that key holds a value that is no number, and raises the error of the cast for them (message and SQLSTATE) while a dirty member of another key raises nothing; and an `EXISTS` over the elements of a `jsonb_to_recordset` list passes a row whose list raises, which then raises where the rows reach the aggregates, as the normal plan does when it evaluates the test. A query that mixes two casts of the member, filters on it, casts it in a branch, or tests the list under `OR` keeps the normal plan. No format change.
- Changed: pg_sift: an outer group key that is NULL in every arm of a two level aggregate (a column of a generated series, a bucket the arms do not break down) no longer keeps the normal plan: the key is NULL in every group of the scan.
- Fixed: pg_sift: a `SiftAgg` combo (a lookup join keyed by dictionary values, an expression over them) finds the entry of a combination of values by a hash that spreads small ids: with hundreds of thousands of distinct combinations the table of entries used to pile them in a few slots and walk thousands of entries per tuple (200,000 combinations of three keys over 260,000 tuples, a 500,000-row mapping table, one backend: 37.3 s, now 0.57 s). Test `agg_combo_hash`.
- Changed: pg_sift: a top-N scan (`ORDER BY key LIMIT n`) without a search over a `sift_col` key that does not follow the table (values in blocks in no order, random) reads the zones of all the superblocks first, visits the superblocks by the best row their zone can hold and ends at the first that cannot beat the n-th row (`sift.topn_walk`, default on; `off` visits them in table order). The rows are the same.
- Added: pg_sift: SiftAgg reads again from the heap the rows whose document a text path of a `sift_col` index cannot store (`doc ->> 'k'`, an `astext` leaf: a container where the path is a scalar, a document that is an array) as `->>` reads them, instead of keeping the normal plan for every query that reads the path.
- Fixed: pg_sift: a `jsonb_typeof` test of a path that the query also reads (`doc ->> 'k'`) no longer makes the aggregate scan raise `0A000` where the normal plan runs: a test that keeps a type the path cannot store (an object, an array, no value: the paths of a document that is an array) keeps the normal plan once an index unit holds such a value, one that keeps only the types the path stores (`string`, `number`, `boolean` of a text path) is still answered by the scan.
- Changed: pg_sift: SiftAgg reads a partitioned table also when a parameter of a generic plan or a stable function (`d >= $1`, `k = ANY ($1)`, `ts >= now() - interval '30 days'`) prunes its partitions when the plan starts: the scan evaluates the operand when it starts and does not read the partitions whose bounds it refutes (`Partitions Removed` in `EXPLAIN ANALYZE`; parallel workers take the leader's superblocks). Before, a parameter on the partition key kept the normal plan, and a stable bound read every partition. A table split by hash prunes the same way: the scan computes the remainder of each value an equality, `= ANY ($1)` or a stable function leaves the key (the key's hash function, the modulus and remainder of each partition, every column of a multi-column key) and does not read the partitions that cannot hold it; another test of the hash column (`k > $1`), another type of the operand and an expression key keep the normal plan.
- Changed: pg_sift: the keys of an inner or semi join (the join key filter) drop the partitions of a partitioned table that none of them can be in when the join column is a column the table is split by: a range or list bound is refuted by the range of the keys and, with few keys, by each of them, a hash level by the remainders of the keys, and no key reads no partition (`Partitions Removed`). Before, the scan read every partition and the zones of the units pruned the units of range and list partitions only.
- Changed: pg_sift: SiftAgg reads a partitioned table of up to 4,096 partitions (was 1,024), so a stable bound or a parameter that prunes a table of thousands of partitions to a few keeps the scan: the scan opens only the partitions pruning leaves (it opened all of them: 3,000 partitions pruned to one, 92 ms to 9.5 ms of execution) and releases the buffers a partition keeps pinned when it moves to the next (a scan of 7,000 partitions failed with `no unpinned buffers available` at `shared_buffers = 64MB`; it now keeps those of one partition pinned). A parallel worker opens only the partitions the leader reads. Test `agg_partitions_many`.
- Fixed: pg_sift: a `sift_col` build (`CREATE INDEX`, `REINDEX`, in a parallel worker too) and the aggregate scan's read of rows again from the heap no longer crash the backend when the index's relcache entry is reloaded meanwhile: they kept the index's cached state across the tuples, which that reload frees. The build now keeps a copy of its own. Test `col_index_relcache` (`sift.test_col_rel_inval`, testing builds).
- Added: pg_sift: a backend keeps the rows of the tables a `SiftAgg` scan joined (the lookup tables of `JOIN`, `IN (SELECT ..)`) for its next query while the table is unchanged and vacuumed (`sift.agg_lookup_cache_kb`, 256 MB per backend, 0 disables; `Cached Lookup Tables` in `EXPLAIN ANALYZE`): a mapping table of 500,000 rows is no longer loaded again for every query. A write, a rollback, a vacuum, a rewrite or a table with pages not yet visible to all transactions loads the rows again.
- Added: pg_sift: `Custom Scan (SiftAggRows)` groups the rows of a plan that reads SiftAgg nodes when an aggregate counts distinct integers (the outer `count(DISTINCT id)` of a report over per-row aggregates, joined to a spine of periods): the counts, `sum` and `avg` are kept per group and each `count(DISTINCT x)` is a set of (group, value) pairs, which moves to a sort of narrow tuples when it outgrows `hash_mem`, instead of PostgreSQL's sort of every wide row and a sorted aggregate per `DISTINCT`. `sift.agg_rows` turns it off. `avg` and `sum` of floats add in the order the rows arrive, as the scan's own aggregates do.
- Changed: pg_sift: SiftAgg reads a partial `sift_col` index (`CREATE INDEX .. WHERE pred`) when the conditions of the query imply its predicate, instead of ignoring it; the index is smaller and a query that does not imply the predicate keeps the normal plan.
- Changed: pg_sift: SiftAgg prices the rows it reads again from the heap (values a `sift_col` leaf cannot store, such as a numeric string in a number path) by the share of units that hold one instead of a fixed share of the table, and `EXPLAIN ANALYZE` shows `Units Read Again From The Heap` next to `Rows Read Again From The Heap`.
- Fixed: pg_sift: a table with several `sift_col` indexes is scanned through the first of them that gives a SiftAgg plan. SiftAgg took only the index PostgreSQL lists first (the newest), so a second index with other columns or paths left the queries only the older one can answer on the normal plan.
- Changed: pg_sift: a backend keeps the parsed dictionaries of the segments of a `sift_col` index (up to 64 MB, least recently used first) for its next query: a scan that reads a string leaf again no longer inflates and indexes the leaf's dictionary (a free text path of 60,000 entries took about 1.3 ms per segment on every query; the second query of a session takes none, with identical answers). A rewritten segment has its own. No format change.
- Fixed: pg_sift: `sift.col_verify(index, true)` no longer crashes the backend when another session's `VACUUM` or `ANALYZE` updates the index's statistics while it checks the heap: the check kept the index's cached state, which that update frees, across the tuples. Test `verify_relcache` (`sift.test_col_verify_inval`, testing builds).
- Changed: pg_sift: SiftAgg prices a scan filtered by a lookup join by the units the zones of its keys leave (the mean zone width of the leaf, a key a point of the range), and a key filter of widely spread keys skips the units whose zone holds none of them
- Changed: a scan that the cost of the rollups chose for three units in a row (their tuples cost at least a quarter more than their elements) skips the exactness check and the rollup read of the next sixteen units, so a query on an index whose rollups do not pay is not slower than with `sift.agg_preagg = off`.
- Added: the rollups of the elements of a list answer `GROUPING SETS`, `ROLLUP` and `CUBE` over the dims (each set folds the tuples again under its own columns, the keys outside it NULL), also with `FILTER`, tests on dims and derived keys; `GROUPING()` and the sets over the rows keep the scan.
- Added: pg_sift: SiftAgg runs an `EXISTS` over the elements of an array whose test reads a lookup per element (`LEFT JOIN LATERAL (SELECT .. FROM mapping WHERE mapping.k = elem.k AND mapping.kind = dim.kind LIMIT 1)`) or a column of a joined table (the label of an event by the kind of its parent): the lookup runs once for each distinct value of the columns it reads, a row passes when some element does, and a row whose elements and joined value repeat is not tested again. `NOT EXISTS`, `EXISTS` in an aggregate `FILTER` and expressions that can raise keep the normal plan.
- Changed: pg_sift: SiftAgg tests the keys of its inner and semi lookup joins before tuples are made also in a plan with a preserved side (a date spine `LEFT JOIN` rows that join tables of their own), keeps only the keys that the joins keyed by a column of the table can match (`JOIN ids ON ids.id = d.id` after `JOIN d ON d.id = f.d_id`), and reads no unit when a join has no key at all; a dictionary entry gets its id in a join when a tuple first reads it, not all of them when the scan meets the dictionary (a column of millions of free texts).
- Changed: pg_sift: the arms of a `UNION ALL` that join the same common table expression (a `VALUES` list of periods, a list of ids read by every arm) run in one pass over the index, whatever columns of it each arm reads: the list is loaded once, the scan, the lookups and the rows read again from the heap are shared. Another list, or another condition on it in one arm, keeps its own pass. A sub-select or function joined as a plan still keeps one pass per arm.
- Changed: pg_sift: the arms of a `UNION ALL` that look a name up with the same correlated `LIMIT 1` sub-select (a mapping table joined on values of the unnested list) run in one pass over the index although they read other columns of it or ask for other expressions of it (one the mapped name, one also its sub-name): the scan, the lookups and the rows read again from the heap are shared. An arm with an inner join (the `IS NOT NULL` of a mapped name) and an arm with a left join of the same sub-select (all the lines) run in the same pass too: the pass joins left and the arm with the inner join drops the lines without a row; an arm with another condition keeps its own pass.
- Changed: pg_sift: the arms of a `UNION ALL` that read the same rows for the same aggregates run in one pass over the index also when they join a table (a lookup join or a combo the arms share), read leaves of their own, or have more than four keys together (a key two arms share counts once; up to 31 distinct keys): the rows read again from the heap for values the index cannot store are read once for all the arms, and `EXPLAIN ANALYZE` shows their count on every arm.
- Added: pg_sift: SiftAgg evaluates `jsonb_exists(doc, 'k')` of the document whose list (`jsonb_to_recordset(doc->'items')`, `jsonb_array_elements(doc->'items')`) it unnests, in keys, `CASE` branches and filters, from the JSON type of the member `k` the index holds (a JSON `null` member exists); it stays with PostgreSQL elsewhere, where an array or a string document can hold the key.
- Fixed: pg_sift: SiftAgg reads again from the heap the rows of `jsonb_to_recordset` whose document holds a value the index cannot store (a number written as a string) also when the query reads the column inside an expression (a `CASE` key, a bucket of the value): the query kept PostgreSQL's plan before.
- Fixed: pg_sift: SiftAgg joins a sub-select whose plan PostgreSQL gave a condition pushed down into it (the run condition of a ranking window: `IN (SELECT .. FROM (SELECT .., row_number() OVER (..) rn ..) r WHERE rn <= 3)`) instead of keeping the normal plan: the plan of the sub-select enforces the condition and does not output its column, and the scan no longer tests it again.
- Added: pg_sift: SiftAgg runs aggregates whose derived table has a `WITH` of its own (a ranked top-N over an aggregate, `UNION ALL` arms over one CTE): the CTEs of a derived table that reads nothing around it are lookup plans of the scan, and a CTE of it that several aggregates read is planned inline like the statement's own.
- Added: pg_sift: SiftAgg runs an `EXISTS` over the elements of an array that is written with a sub-select PostgreSQL pulls up, and the conditions of the nullable input of an outer join (a date spine `LEFT JOIN` a sub-select or CTE) that test its arrays (an `EXISTS`, `jsonb_path_exists`) and the aggregate filters that read a test computed there: the `ON` conjuncts that read only the scanned table filter its rows before matching, and a test guarded by the absent input is read through the guard when it is not true for an absent input.
- Changed: pg_sift: SiftAgg prices the filters and expressions on the values of an object member or array element once per distinct value (PostgreSQL's default number of distinct values, not once per row), so a per-key metric aggregate with `FILTER (WHERE value >= 8)` is chosen when the scan is cheaper.
- Fixed: pg_sift: SiftAgg no longer fills a table of the whole range of an integer group key (a site id of a table ordered by another column, whose units hold few values far apart) before it uses the hash table: such a key now costs the groups it has, not its range, which made a per-key aggregate several times slower than PostgreSQL.
- Fixed: pg_sift: SiftAgg aggregates `GROUPING SETS` / `ROLLUP` over group keys computed inside the nullable input of an outer join (a date spine `LEFT JOIN` a CTE or sub-select that derives the keys) on PostgreSQL 18: the keys are the expressions of the input, NULL for the rows no input row matched, instead of the query being left to PostgreSQL.
- Fixed: pg_sift: SiftAgg counted the preserved row of an outer join that no row matched as one value of `count(DISTINCT key)` when the key is the unique column of the scanned table (a date spine `LEFT JOIN` an empty selection answered 1 where PostgreSQL answers 0).
- Changed: pg_sift: SiftAgg prices a lookup join without keys (a date spine joined on an expression) by the distinct values the scan's own conditions leave and by the rows a set-returning spine makes, not by every distinct value of the table and PostgreSQL's 1000 rows a call. Aggregates over a date spine and correlated arrays of a joined row pick the scan on their own costs where it is faster.
- Fixed: pg_sift: SiftAgg no longer reads a value the index cannot store (a string, an object or an array in a `float8` path, a scalar where an array is declared) as NULL when the query takes a key of an object leaf with free keys (`(metrics ->> 'k')::float8`, `(metrics -> 'k')::float8`) or `jsonb_array_length(doc -> 'l')`: such a query keeps the normal plan, so a numeric string converts and anything else raises PostgreSQL's own error, and a plan made before the value was written raises the scan's error. In the columns of `jsonb_to_recordset` a string `NaN` or `Infinity` converts instead of raising the scan's overflow error (the string `-0` still raises it).
- Added: pg_sift: an expression key or predicate that reads one column twice under binary-compatible types (the column, and the column relabeled for a comparison with a `char(n)` column of an `IN (SELECT ..)`) runs in SiftAgg.
- Added: pg_sift: a correlated child of SiftAgg (a `UNION` arm, a lateral sub-select) can read the value of a `jsonb_each` member and cast it: the child is given the jsonb the stored number stands for, runs only for the members a row reaches, and raises PostgreSQL's own cast error. `x IS TRUE` over an existence test (`jsonb_path_exists`, `@?`) runs in the scan.
- Added: pg_sift: a lateral sub-select that PostgreSQL leaves in place because its query is read on the nullable side of an outer join (`dates LEFT JOIN cte`) is pulled up inside its own query level when that gives a SiftAgg plan that costs less, so `jsonb_each(doc)` joined to a mapping table runs in the scan instead of as a correlated child of the whole document.
- Added: pg_sift: `LEFT JOIN LATERAL jsonb_each(doc) r ON r.key = 'k'` (a constant key) runs in SiftAgg on PostgreSQL 18 as the member `k` of each row, with its NULL columns for the rows without it and the casts of its value (also inside a correlated child) raising PostgreSQL's own errors.
- Fixed: pg_sift: SiftAgg no longer treats `LEFT JOIN LATERAL jsonb_each(..)` / `jsonb_array_elements(..)` as an inner join: the rows without a member (NULL columns) were dropped and the ON clause was ignored. Such queries keep the normal plan.
- Added: pg_sift: SiftAgg joins a lateral sub-select, a `UNION` / `UNION ALL` of arms, `generate_series`, `unnest` of an array column or a set-returning function that reads the joined row. PostgreSQL's parameterized plan of it is the child, run once per distinct input, with duplicates and NULL-extended rows as in the normal plan. A call of a document column that cannot raise (`jsonb_exists` or `?`, `->>`, `jsonb_typeof`) takes the index expression column of that call as the parameter. A child that can raise (a division, a cast of text, a sub-select) is joined only when the query joins nothing else, and one that reads a column of a query above the aggregate's own never is, so no statement fails only under the scan. A `UNION ALL` / `UNION` of sub-selects, a sub-select of a statement-level CTE and a `WITH` inside the sub-select are lookup plans too.
- Added: pg_sift: SiftAgg reads the columns of `jsonb_to_recordset` from an index that flags values its typed leaves cannot store (a numeric string in a number path, a document that is an array or a scalar, a number in a text path). The flagged rows are read again from the heap and read as the function reads them, so the answer, errors included, is the normal plan's; `EXPLAIN ANALYZE` shows `Rows Read Again From The Heap`. No format change.
- Added: pg_sift: an inner or semi lookup join on an integer or date leaf tests the leaf against the keys of the loaded table before tuples are made (`sift.agg_join_filter`, `Join Key Filters` in `EXPLAIN ANALYZE`): units the keys' range misses are not read and the other rows are tested as a set instead of through the combo cache.
- Changed: pg_sift: a combo of one small key finds its cached entry by array index instead of a hash probe and `memcmp`.
- Added: pg_sift: `UNION ALL` arms of one statement that join the same table with the same columns and conditions load it once (`sift.agg_share_lookups`, `Shared Lookup Tables`); nothing is kept between statements.
- Added: pg_sift: hashed `IN (SELECT ..)` / `NOT IN (SELECT ..)` tests inside group keys, predicates and aggregates of an outer join's nullable input, row-valued tests and tests of an `int8` column run in SiftAgg (the scan asks the sub-plan for each tuple that reaches it).
- Fixed: pg_sift: grouping by more than eight keys with the two-level aggregate no longer overflows its key mask.
- Fixed: aggregate null-input markers cannot alias PostgreSQL executor parameters; preserved-input expressions can be explained on PostgreSQL 17. Predicate-only lookups account for testing every candidate row.
- Added: grouping expressions and aggregate FILTER predicates can read up to eight scalar inputs. Expression caches store and compare every input, with storage sized to the actual tuple width.
- Added: aggregate FILTER predicates over lookup columns preserve duplicate matches and NULL extensions, and evaluate expressions only after all joins select their input.
- Fixed: aggregate planning ignores unique LEFT joins PostgreSQL has removed, avoiding a planner crash. Preserved buckets can contain several dependent lookups; empty buckets retain NULL values for inner expressions and constants.
- Fixed: VACUUM relocation invalidates cached write targets when each replacement is published, including when maintenance later errors before truncation; failed unpublished copies return their allocations and follow maintenance backoff.
- Added: Homebrew formula in the source repository for PostgreSQL 17 and 18, tested on macOS arm64 with BM25, extension upgrade and column aggregate smoke tests; bitmap dispatch uses the existing scalar kernels on non-x86 CPUs.
- Added: pg_sift: `col @@> ANY (array)` is scanned as the OR of its elements, scored and ranked like `col @@> 'a OR b'` (`sift.score()`, `ORDER BY sift.score(...)`, `LIMIT`, `count(*)` and the filters work with it); a NULL element is skipped and a NULL or empty array matches no rows. Measured against the bitmap scan it replaces: `count(*)` 1.6 to 54 times faster, `LIMIT 10` and `LIMIT 1000` 1.5 to 130 times faster (0.1 ms slower on queries that take under 0.2 ms), the full match list equal or up to 2 times faster, planning under 0.6 ms at 100 elements. A list with no score or order that is estimated above 100,000 rows (after a `LIMIT`) keeps the bitmap scan, which was as fast or faster there.
- Changed: column format minor 6 guarantees complete JSON kinds for declared paths, including element leaves. This fixes wrong answers after merging a minor-3 segment into minor 4: JSON null could become a missing key. Old minors 3 through 5 and legacy write buffers remain readable; queries needing kinds those inputs cannot prove keep the normal plan, including partition parents. REINDEX and a sift_table VACUUM FULL reconstruct the kinds from the documents. No extension SQL changes are needed.
- Compatibility: a 3.1 library refuses a minor-6 column segment with a format error and REINDEX hint. Install 3.2 on every reader before writing minor 6. `sift.col_format_minor = 4` keeps new narrow segments readable by 3.1; 3 keeps the 3.0 layout, while wide leaves still require at least minor 5. Schemas without JSON paths keep their previous minor.
- Fixed: VACUUM returns internal free space of `sift_col`, `sift_table` and `sift` to the operating system by moving tail segments into existing free runs before truncating. Relocation preserves the stored format, reserves every destination below the planned end, and shares one write allowance across all moves. `vacuum_truncate = false` disables it; a conflicting reader skips it without waiting.
- Changed: `sift_col` VACUUM promotes sufficiently filled open write buffers and continues merging within `sift.vacuum_rewrite_limit` (default 128 MB). The same setting separately caps page relocation; zero retains the first column merge and ordinary free-tail truncation while disabling relocation. No on-disk format change.
- Added: pg_sift: SiftAgg answers DISTINCT ON over indexed columns. The first row of each group by the ORDER BY is kept while the rows pass, with no sort, in the leader and the workers, when the ORDER BY holds the columns of a unique `NOT NULL` index; a tie-break that is not unique, a volatile or array value and groups that do not fit the hash memory keep the normal plan (`docs/AGG.md`). Test `agg_distinct_on`; fixtures `do_*`.
- Added: pg_sift: SiftAgg aggregates over the preserved side of outer joins. `RIGHT JOIN`, `FULL JOIN` and a `LEFT JOIN` whose right side is the indexed table keep the rows of the other table that nothing matched, as one row of NULLs in the groups; a scan with such a join runs without workers.
- Added: pg_sift: SiftAgg pushes down ctid range predicates. `ctid` `=`, `<`, `<=`, `>`, `>=` and `BETWEEN` against a constant TID, at the top level of the `WHERE`, read only the superblocks of the range and test the rows of its first and last block (HOT chains included); `<>`, expressions, `OR`, lists, partitioned tables and top-N keep the normal plan (`docs/AGG.md`).
- Added: pg_sift: SiftAgg on partitioned tables whose partitions carry the same sift_col index. The partitions left after pruning are read as one source (each through its own index, whatever its attribute numbers, with parallel workers over all of them) and `EXPLAIN` lists them; a partition without the index or with other options, an inheritance parent, `ONLY`, and a parameter that prunes at run time keep the normal plan. Test `agg_partitions`; fixtures `pa_*`.
- Added: pg_sift: SiftAgg pushes down `NOT IN (SELECT ...)` and uncorrelated `IN` sub-selects under `OR`. PostgreSQL's hashed `SubPlan` runs once as the lookup table of the column, so the NULL rule is the normal plan's; a correlated, volatile or raising sub-select keeps the normal plan, and the scan runs without workers. Test `agg_subplan_lut` and the `t_not_in_subquery*` coldiff fixtures.
- Added: pg_sift: sift_col indexes inet, cidr, macaddr, macaddr8, timetz, bit and varbit columns. A value is stored as the image of the datum and `Custom Scan (SiftAgg)` reads it through the dictionary of datums, so keys, `count(DISTINCT)`, `min`/`max` (`inet`, `cidr`, `timetz`), conjuncts, `OR`/`CASE`/`FILTER` and expressions such as `host(ip)` are PostgreSQL's own functions on the value (`10.0.0.1/24` and `10.0.0.1/32`, or `12:00+01` and `11:00+00`, stay different values). `GROUP BY` a `bit` or `varbit` column keeps the normal plan (PostgreSQL has no hash operator class for them). `point`, `tsvector`, `xml` and `json` stay refused at `CREATE INDEX`. The column format minor is still 5; the new type tags 20 to 22 are in the same unreleased schema. Tests `col_agg_types33` and the `n_net_*` coldiff fixtures.
- Added: pg_sift: `sift.score()` compared with a constant in `WHERE` (`>`, `>=`, `<`, `<=`, either side; a constant, a parameter or a stable expression of them) filters the ranked scan: the scan tests the cutoff on the score it computed, so the rows equal those of a subquery that filters on the score, with `LIMIT`, `OFFSET`, `ORDER BY` and prepared statements. Any other use of the score in `WHERE` (`OR`, `=`, `<>`, an expression of the score) is still an error. Test `rank_score_filter`.
- Added: a `sift_col` index accepts `numeric(p, s)` of any precision (and scale up to 18 or beyond, or negative), unconstrained `numeric`, `money`, `interval`, `time`, enum, `uuid`, `bytea`, array, range and multirange columns. A value is stored as ordinary leaves of the index: a `numeric(p, s)` of 19 to 38 digits as two 128-bit lanes, an unconstrained or larger `numeric` as a canonical image whose byte order is the numeric order (`1.0` and `1.00` are one value, NaN and the infinities sort as PostgreSQL has them), `money` as an `int64`, `interval` as its total (months, days and microseconds in PostgreSQL's comparison order), `time` as microseconds, an enum as its OID (its order is read from the catalog when a query runs, so `ALTER TYPE ... ADD VALUE` after the index was built is safe), and `uuid`, `bytea` and the other varlenas as the image of the value. `Custom Scan (SiftAgg)` pushes down, for these columns, a `WHERE` conjunct (`= <> < <= > >=`, `BETWEEN`, `IN`, `NOT IN`, `IS [NOT] NULL`, decided on the stored lanes for the numeric, money and interval ones; `Wide Filters In Source` in `EXPLAIN ANALYZE`), `OR`, `NOT`, `CASE` and `FILTER (WHERE ...)` trees, `GROUP BY` and `count`, `count(DISTINCT)`, `min`, `max`, `sum`, `avg` and expressions (casts, `n + 1`, `IS DISTINCT FROM`, `= ANY`), by calling PostgreSQL's own functions on the value, so the rows, their text and the errors are the normal plan's (every shape is a coldiff fixture and a regress check). A group of equal values spelled differently (`1.0`/`1.00`, `1 mon`/`30 days`) shows the first spelling seen, as a serial hash aggregate; a key or an aggregate over such a column runs without parallel workers, and the planner keeps the normal plan for a column with a distinct value per row. `point`, `tsvector`, `xml`, `json` and the other types without a leaf form are still refused at `CREATE INDEX`. Before, `CREATE INDEX` refused all of these types. Column format minor 5 (`SIFT_COL_FORMAT_WIDE`, numbered in `docs/FORMATS.md`): a segment is written as minor 5 only when its schema has such a leaf, indexes without JSON paths keep writing minor 3 or 4 and stay readable by 3.1, and a 3.1 library cannot open an index that has one of these columns (drop it first). `sift.col_verify` checks the lanes, the image and the interval total against the heap; `sift_table` stores these types as before (and now also `numeric(p, s)` with a scale over 18). Tests `col_index_wide`, `col_format5`, `col_agg_wide`, `col_agg_wide_keys`, `col_agg_types32`, the `w_*` and `m_*` coldiff fixtures and the `colwide` fuzz target.
- Added: top-N scans accept `LIMIT`/`OFFSET` parameters of generic plans and an OR of searches on one index. `ORDER BY .. LIMIT $1` keeps the top-N plan (the scan reads the value when it starts: up to 100000 rows in a heap, `NULL` or more rows in a sort of the matches that still reads the heap only for the rows the `Limit` takes), and `WHERE col @@> a OR col @@> b ORDER BY .. LIMIT n` takes the union of the matches of the arms.
- Added: pg_sift: sift_table returns the row's xmin, xmax, cmin and cmax. They read zero before; now `xmin` and `cmin` are the stored versions of the row (write buffer record or the version section of its column group; `FrozenTransactionId` once frozen) and `xmax` and `cmax` those of its last delete, update or lock tombstone (0 if none). A row locked by several transactions shows the last locker, not a MultiXactId. Tests `tbl_sysattr`, the isolation specs `tbl_sysattr` and `tbl_sysattr_groups` and the `t_sysattr_xmin` coldiff fixture.
- Added: a set filter sent as one array parameter (`dim_a_id = ANY($1::int4[])` of a prepared statement on a generic plan) takes `Custom Scan (SiftAgg)` by default. The planner cannot see how long a parameter array is and assumed 10 elements, which kept the bitmap scan at any size unless the caller switched index and bitmap scans off; the paths that fetch the rows of such a set one at a time are now costed for `sift.agg_param_set_size` elements (default 1000, 0 gives the old guess back), only for a query that can run as `SiftAgg`. 3M events, a 12-month window, `count` and `avg` per channel, generic plan: 4.8 / 5.6 / 10.4 ms at 1,000 / 10,000 / 100,000 members against 10.3 / 32.4 / 100.5 ms before and 6.0 / 17.8 / 129.8 ms for the same set as a literal `IN` list; a set of about 100 members is 2.3 ms slower (use a custom plan or `SET sift.agg_param_set_size = 10`). `docs/USAGE.md` shows how a client binds the `int4[]` (libpq, Sequel). Test `agg_set_array`.
- Fixed: the overflow bitmap buffers of `SiftAgg` and `sift_table` reads were sized in 64-row words, not in the 512-row units the readers fill: a numeric leaf with an overflow bit or NaN in a unit that did not end on 512 rows wrote up to 56 bytes past the buffer (found by the memory-context checks of a PostgreSQL built with assertions).
- Fixed: a `LIKE 'prefix%'` on a text column of a `sift_col` index, and a comparison on an unconstrained `numeric` one (stored as images), was turned into a lookup table over the whole dictionary of its segment again for every unit and every range of the scan, so `Custom Scan (SiftAgg)` took time in proportion to units times dictionary entries. On 2 000 000 rows with 200 000 distinct keys `LIKE 'key-12%'` took 171 ms (3.1.0; 73 ms for the normal plan); on a million distinct `numeric` values `n > c` took 0.5 s and `n BETWEEN a AND b` 2.3 s against 0.06 s for the normal plan. The test is now made once per segment and predicate: 10 ms, 17 ms and 33 ms (PostgreSQL 18, 2026-10-20). The planner also prices the dictionary of such a numeric conjunct. Test `col_index_dict_tests` (`tests_made` of `sift._test_col_counters`).
- Fixed: `col NOT IN ()` / `col <> ALL('{}')` with an empty array parameter or constant did not keep the rows whose `col` is NULL in `Custom Scan (SiftAgg)` (the condition is true for every row); found by `agg_set_array`.
- Fixed: `col = ANY(array)` / `col IN (..)` on an integer or date leaf returned every row of a unit instead of the matching ones when the list had gaps (or an empty, `NULL`-only or very wide list) and the unit's values lay inside its range: the unit was answered from its pre-aggregates as if the range were the set. The pre-aggregates now answer such a unit only when the list holds every value of its range (`PredState.in_dense`); a `NULL` element no longer drops the range test that prunes units. Test `agg_set_array` (it recorded the wrong rows for an empty array and one out-of-range query before).
- Changed: `col IN (..)` / `col = ANY($1)` / `NOT IN` on an `int8`, `timestamp` or `timestamptz` leaf is tested natively like an integer leaf: a byte table over the range of the list (up to 4 MB), else a hash table of the values, with the same zone pruning, pre-aggregate and in-source filter rules. It was a PostgreSQL expression evaluated once per distinct value of the column against the array searched linearly, which made an array parameter of 1,000 to 5,000 `int8` keys on a 200,000-key column 20 to 450 times slower than the normal plan once the generic plan took `SiftAgg`. A `text` array parameter (and any other expression on a text leaf) now hashes the array once instead of searching it per distinct value (3,100 ms to about 25 ms for 1,000 keys). `docs/USAGE.md` lists the key types the array form pushes down (`uuid` and `bytea` cannot be indexed).
- Added: `Custom Scan (SiftAgg)` answers a semi or anti join (`IN (SELECT ..)`, `EXISTS`, `NOT EXISTS`) on an integer or date leaf from a byte table over the keys of the joined table, built when the scan starts (`sift.agg_semi_lut`, default on; `sift.agg_semi_lut_size`, default 4 MB of table per scan and per worker, wider keys keep the generic lookup join; `Semi Lookup Tables` and `Semi Lookup Table Size` in `EXPLAIN ANALYZE`). The table's own conditions are tested once per row of the table, and a key outside the range of the leaf is ignored. Synthetic 3M-event dataset, 12 months, `count` and `avg` per channel: membership `IN` of 1,000 to 5,000 members 6.0 to 7.0 ms (`EXISTS` 5.6 ms and `NOT EXISTS` 9.7 ms at 1,000) and a dimension attribute of 3,300 or 17,000 entities 10.0 to 13.3 ms (with the lookup load below), against 63 to 91 ms on the normal plan and 25 to 36 ms with the generic lookup join. `NOT IN (SELECT ..)` stays on the hashed subplan. Tests `agg_semi_lut` and the `j_semi_lut_*` coldiff fixtures.
- Changed: the planner prices the semi and anti joins of a `SiftAgg` path by the rows PostgreSQL expects the join to keep, so a set filter is chosen over the hash semi join; and a parallel `SiftAgg` is planned for a query whose semi joins PostgreSQL pulled up from a sub-select (the query-wide parallel hazard counted them as restricted; the relations' own flags, as PostgreSQL's grouping uses, decide now).
- Added: the lookup table of a `SiftAgg` join (the table of `IN (SELECT ..)`, `EXISTS`, or an inner join to an attribute table) is loaded with only the rows its own conditions can match. The planner collects the `column = value` conditions of the join (`SiftAggJoin.idxatt` and the rest); the executor evaluates the values once and reads the longest matching prefix of a btree index of the table (an index scan, or heap scan keys on a table without one), and the conditions it cannot push are applied to each row as it is read, so the table keeps only the rows that can match. A group of 1,000 members in a membership table that has a btree on `(group_id, ..)`: the generic lookup join 70.8 to 22.9 ms and the byte table 24.8 to 5.5 ms. The planner prices the load by the rows it expects the condition to keep (`AGG_IDX_ROW`) instead of the rows of the table. `EXPLAIN ANALYZE` shows `Lookup Rows Read`, `Lookup Rows Loaded`, `Lookup Index Scans` and (with `TIMING`) `Lookup Load Time`; `sift.agg_lookup_pushdown = off` loads the whole table as before. Test `agg_lookup_load`.
- Added: pg_sift: `Custom Scan (SiftAgg)` joins to VALUES lists, sub-selects and functions, and aggregates over a ranked top-k. The right side of a lookup join (`JOIN (VALUES ..)`, `JOIN (SELECT .. GROUP BY / LIMIT / DISTINCT / UNION ..)`, `JOIN generate_series(..)`, `LEFT JOIN`, `IN (SELECT .. ORDER BY sift.score(ctid) LIMIT n)`) is run by PostgreSQL's own plan once when the scan starts, shown under the node in `EXPLAIN`, and its rows become the lookup table: duplicate keys multiply rows, NULL keys match nothing, the result is the normal plan's. It stays on the normal plan when the right side is volatile, `ON true` and not lateral, has no equality, is estimated above `sift.agg_lookup_max_rows` (new, default 1000000), is read only by a condition PostgreSQL pushed into it, or is the nullable side of an outer join to the table; no parallel workers. Tests `agg_plan_tables` and the `pt_*` fixtures.
- Added: pg_sift: SiftAgg tests an array element against a column of the same row (`EXISTS (SELECT 1 FROM jsonb_to_recordset(o.items) i(qty int) WHERE i.qty = o.tenant_id)`, also `<> < <= > >=` and `NOT EXISTS`): the element vector is compared with the value of the column of each element's row. The column must be a stored column of the same type class as the member; an expression of it, `OFFSET` and a second array keep the normal plan.
- Added: pg_sift: SiftAgg pushes down `EXISTS` over `jsonb_to_recordset` of an array that an index expression returns, `jsonb_path_exists` with nested paths (`$.a[*] ? (@.b == "x")`, `$.a[*].b`) and text-valued (`astext`) members of a list of objects in them, with plain-text equality only (a number or boolean literal, a strict path and a NOT of a test that may be NULL keep the normal plan); about 500 times faster than the normal plan at 300,000 items.
- Fixed: a correlated `EXISTS` over the recordset of an index expression failed at plan time with `invalid attnum N for relation "o"` while `sift.enable_agg` was on: the planner deparsed the expression with the range table of the sub-select. Found by an event analytics workload (`test/col/workload.sql`).
- Added: pg_sift: SiftAgg pushes down `x = ANY (ARRAY (SELECT ...))` of an uncorrelated, stable sub-select that cannot raise: the InitPlan runs once and its rows are the lookup table of the leaf (no parallel workers, as for a hashed sub-plan).
- Added: pg_sift: per-item arms under an outer aggregate fuse into one scan when the inner `sum` or `avg` is a float8 expression PostgreSQL evaluates, such as the cast of a text-valued leaf (`avg(n::float8)`); they ran one by one (33 times slower at 300,000 items). The arms must share one WHERE and have no inner `FILTER`.
- Fixed: pg_sift: SiftAgg no longer runs `LEFT JOIN LATERAL jsonb_each(doc) r ON r.key = 'k' COLLATE <non-deterministic>` as the member `k` of each row: such a collation equates `k` and `K`, so the join matches both members and the counts and sums differed from PostgreSQL's.
- Fixed: pg_sift: SiftAgg bounds the rows it keeps of a correlated child (`sift.agg_lookup_max_rows`) by the inputs that can reach the child, also when it reads a column of another joined table with several rows for each scanned row; it ran the child for tens of thousands of inputs although the setting was a thousand.
- Fixed: pg_sift: a query that joins `LEFT JOIN LATERAL jsonb_each(doc) r ON r.key = 'k'` and holds an `EXISTS` over `jsonb_each` of another document crashed the backend while `sift.enable_agg` was on: the scan read the elements of the `EXISTS` as that member.
- Fixed: pg_sift: SiftAgg no longer runs a `FULL JOIN` of the scanned table and a sub-select whose column is a non-strict expression (`coalesce(name, 'x')`) as the value of the expression over NULL columns: the rows the join finds no match for have NULL there, and the groups of the two plans differed.
- Fixed: pg_sift: SiftAgg answered a unit of `jsonb_to_recordset` from its rollup although an element held, in a column the function declares and the query does not read, a value the column's type refuses (a string where it is a float8); PostgreSQL raises for it, and now the unit is scanned and raises too.
- Fixed: pg_sift: SiftAgg no longer fails with `a column of the aggregate scan holds values its index cannot store` for a `jsonb_to_recordset` query that has a condition on the same list, when a document that the condition drops holds a value one of the function's declared columns refuses; the query keeps PostgreSQL's plan, which never gives that document to the function.
- Changed: pg_sift: the arms of one statement that join the same table with the same columns and no condition of its own share the rows they load also when their join keys read a different number of columns (`Shared Lookup Tables`).
- Fixed: pg_sift: SiftAgg gave a correlated child of `jsonb_each` the number as the float the index stores, so a cast to `numeric` or the jsonb itself lost the scale the document wrote (`12.50` came back as `12.5`); a child now reads the member only through integer and float casts, and the others keep PostgreSQL's plan. No query of the 301-query analytics corpus read a member that way.
- Changed: the default `preagg` cube keeps up to eight measures of each dim (decimals and floats first, then integers), where it kept three, so `sum`, `avg` and `count` of more integer columns are answered from the histogram of the group key. The dims a unit keeps are still chosen by their size with the first three measures under the 2% budget, and a dim that is kept carries all its measures. No format change: units written before hold three measures and are scanned for the others until a merge rewrites them. Index size on a ten-column table: +0.0% to +1.6%.
- Changed: when a unit keeps a few of its rows after the filters of the source (under an eighth), `SiftAgg` makes the dictionary of a wide leaf (a `numeric` of more than 18 digits or of any other shape, an `interval`, `money`) for those rows only, where it converted every distinct value of the unit; a one-key scope that read a `numeric` measure took 1.0-3.6 ms for 25k-200k rows and now takes 0.26-1.3 ms. No on-disk change.
- Changed: the arms of a `UNION ALL` that read the same rows for the same aggregates share one pass also when one of them has no `GROUP BY` (a total beside its breakdowns): the total is a set of the pass without keys, and has its row when no row reached it. The pass launches the workers once for all of them.
- Changed: `SiftAgg` evaluates `sum` and `avg` of an `int8` column plus, minus or times a constant (`sum(val + 1)`, `avg(5 - val)`, `sum(3 * val)`) as an exact 128-bit program with PostgreSQL's own `bigint out of range` error for a result outside `int8`, where it ran the generic aggregate, and a unit whose stored sums and zone prove that no value raises is answered from them as `mul * sum + add * count`. A unit that may hold a value that raises is scanned. No format change.
- Changed: `SiftAgg` answers an aggregate with a `FILTER (WHERE ...)` on the dimension of the cube that the query groups or filters by (`count(*) FILTER (WHERE kind = 3)`, `sum(val) FILTER (WHERE kind IN (1, 4))`) from that dimension's histogram: the groups that pass the `FILTER` add to the aggregate and the others add nothing to it. FILTERs on another leaf, or on two dimensions, scan as before. No format change.
- Changed: the workers of a `SiftAgg` scan read the lookup tables from the shared memory, where each loaded its own copy: the leader reads a joined table (or runs the sub-select, `VALUES` list or function) once and writes the rows for them (`Lookup Rows Shared` in `EXPLAIN ANALYZE`). A join of a sub-select, `VALUES` list or function, which kept the scan in one process, now runs with workers; the planner asks for none when the tables have more than a quarter of the rows of the scan (writing them costs more than the workers save).
- Changed: `SiftAgg` prunes units by an `OR` of zone tests (two periods of a date column, `BETWEEN`s, equalities and `IN` lists of one or several leaves, an AND of ORs): each disjunct is tested on its own and a unit is read when any may match, where an `OR` pruned nothing (`Units Pruned`); an `IN` list with gaps is tested against its members, not only its range. The cost of the scan counts the units an `OR` leaves. No format change. 3.4M-row fact, a count over two 30-day windows a year apart: rows in date order 1,446 to 44 units and 16.9 to 3.5 ms, rows in arrival order 1,442 to 1,319 units (the earlier window spans most of the heap); a grouped aggregate with a 51-key filter on the same windows, rows in date order 137 to 16 ms.
- Changed: the planner counts the units a query's comparisons and `IN` lists leave on a sample of the real zones of the `sift_col` index (up to 2048 units, the same in every integer leaf), so windows on leaves that go together, and short lists of keys, are priced by the units they meet; a unit that few rows pass reads its payload leaves at those rows only, and is priced so (`SiftAgg` is chosen on its own for more selective scopes of a large table)
- Changed: the join key filter of `SiftAgg` covers `int8` fact keys and integer keys that span more than 4 million values (a hash table of the keys, replacing the sorted array of a wide `IN` list); a join that returns no column of the dimension passes the filtered tuples without a lookup (`Join Key Passes`). 3M-row fact, 200k-row dimension: `int8` key at 50% of the dimension 227 to 31 ms, sparse `int4` keys 211 to 53 ms.
- Changed: `SiftAgg` answers an inner lookup join that returns no column of the joined table (a dimension that only filters, `JOIN dim d ON d.id = f.dim_id WHERE d.country < 30`) from the byte table of the semi join when no key of the kept rows repeats; 100,000 of 200,000 dimension keys over 3,000,000 rows: 76 to 27 ms.
- Changed: a unit with a rollup is answered from it only when its tuples cost less than the scan of its elements (`sift.agg_rollup_cost` scales the tuple, 0 always answers from the rollup; `EXPLAIN ANALYZE` shows `Units Scanned By Cost`). Rollups whose tuples are a tenth of the elements were slower than the scan they replaced (14.6 against 7.0 million instructions for `GROUP BY kind`); the planner prices the exact units with the same costs.
- Changed: a unit answered from its rollup decodes only the dims and measures the query reads (and of a measure only the counts, the sums or the minima and maxima it adds up); the other columns are passed over without being stored. 100,000 documents, 29 units of 1,440 tuples: 17.3 against 24.6 million user instructions for `GROUP BY kind` with `count`, `avg`, and 22.6 against 28.7 for three keys.
- Added: the rollups of the elements of a list answer aggregates with `FILTER (WHERE ...)` on dims (`count(*) FILTER (WHERE band >= 8)`, the shares of a metric band): each filter runs on the tuples. A filter on a measure still scans.
- Added: `SiftAgg` answers the integer cast of a jsonb member the index holds as float8 (`(metrics -> 'k')::int`, `CAST(jsonb_extract_path(metrics, 'k') AS bigint)`) in a filter, a group key or an aggregate: the cast runs over the jsonb the stored float stands for, so it rounds as numeric does and raises `integer out of range` for the rows that reach it (PostgreSQL 18). Test `agg_jsonb_int_cast`, fixture `j_jsonb_int_cast`.
- Added: `SiftAgg` reads a `jsonb_each` member through `CAST(value AS float8)` over a `col.* float8` path although some unit holds a value that is no number: the flagged rows are read again from the heap and a row that reaches the cast with a string, boolean, object or array raises `cannot cast jsonb <type> to type double precision`, as the normal plan does; the other rows no longer lose the pushdown. Fixed: a row read again whose HOT chain moved to a later TID than another one made the scan fail with `rows must be added in ascending TID order`.
- Added: the rollups of the elements of a list answer a `GROUP BY` of a function of one dim (`left(day, 7)`, `upper(kind)`, a `CASE` band of an integer dim, `coalesce`), evaluated once per distinct value of the unit's tuples. Before, any expression key scanned every unit.
- Fixed: before PostgreSQL 18 the cast of a JSON null to a number or boolean raises (`(doc -> 'k')::float8`, `CAST(value AS float)` of a `jsonb_each` member); the aggregate scan read it as NULL. A cast of a leaf that may hold a JSON null now keeps the normal plan on PostgreSQL 17 (a leaf without one, and `->>` with its cast, keep the scan). Format: the zone flag `SIFT_COLZ_JNULL` (column format 6, informational) tells which leaves hold one; a write buffer keeps the leaves that hold one in its directory entry (`sum_dl`, as the overflow leaves are kept in `reserved`), and a segment that cannot tell a JSON null from a missing key counts as holding one.
- Added: `EXPLAIN (ANALYZE, VERBOSE)` of a `SiftAgg` node that does not answer units from the pre-aggregates prints `Pre-aggregates: <reason>` (the aggregate is `count(DISTINCT)`, has `FILTER`, a group key is an expression, a conjunct tests a measure, a joined table, grouping sets, and so on). Test `agg_rollup_shapes` lists the shapes of an element-level analytics query with their reason and the units answered.
- Changed: `SiftAgg` loads the joined table of a lookup join through its btree for a `column IN (list)` condition (`dim.country IN (3, 17, 41)`) as for `column = value`, when the statistics expect the list to keep a small share of the table; with a 200,000-row dimension a filter on 3 countries loads 10,000 rows instead of 200,000.
- Changed: `SiftAgg`: a joined table that is joined to another joined table by their key (`fact.k = a.id AND a.id = b.id`, inner joins) is probed with the column of the scanned table, so the keys of `b` (a materialized set of ids) prune units and filter rows like a table joined to the fact directly. A selective set reached through a dimension no longer reads every unit.
- Added: SiftAgg lookup keys and expressions may read earlier lookup outputs, retaining duplicate matches and NULL extensions. Materialized CTEs can supply lookup rows through their PostgreSQL plans.
- Added: SiftAgg predicate-only joins to child plans, including overlapping date ranges, and preserved date buckets over a nullable JSON expansion. Duplicate matches and empty-bucket NULL inputs keep PostgreSQL semantics.
- Added: pg_sift: SiftAgg pushes down jsonpath `!=` and constant variables where the result equals the heap's. The same forms push down with `!()`, a `null` literal, `@>` with a JSON null member and a strict path without `!()`; a JSON null member is told from a missing key by the complete kinds of column format minor 6; older units keep the normal plan when their kinds are uncertain. What stays on the normal plan: a literal of another JSON type than the member's, a number against a `float4` / `float8` member, strict `!()`, `jsonb_path_match` / `@@`, `@>` with a float or `{}`.
- Added: `SiftAgg` group by up to 8 columns (7 with grouping sets), 8 joined tables and 16 combos; `bool_and`, `bool_or` and `every` run in parallel workers.
- Added: `SiftAgg` reads the `numeric(p > 18)`, unconstrained `numeric`, `interval`, `uuid`, `bytea`, `inet`, array and range columns of a `sift_table` through a dictionary of the values, as a `sift_col` index does: keys, `count(DISTINCT)`, `min`, `max`, `sum` and WHERE conjuncts (also under `OR` and `FILTER`) of those columns and of `money`, `time` and enums run in the source. A `money`, `time` or enum conjunct or key of a `sift_table` raised `XX000` `unexpected operand type` before.
- Added: the jsonpath `!=` and `!()` shapes that tell a JSON null from a missing key run on the declared paths of a `sift_table` too.
- Added: a query that writes the expression of a stored generated column of a `sift_table` (`GENERATED ALWAYS AS (date_trunc('month', ts)::date) STORED`) reads the column as a key, a filter or an aggregate argument, with its pre-aggregates, like an index expression of `sift_col`: a monthly sum over 300,000 rows from 1.4 ms to 0.2 ms.
- Added: `SELECT sift.table_set_paths('t', 'doc', 'doc->ch text, doc->w int4, doc->items[]{k int4, p float8}')` declares typed jsonb paths of a `sift_table` (the `paths` grammar of a `sift_col` index): promotion, merge and `VACUUM FULL` write a leaf per path next to the exact bytes, and `->>`, `->`, casts, `jsonb_to_recordset`, `jsonb_array_elements` and `jsonb_typeof` of the path run in `Custom Scan (SiftAgg)` once every segment has the leaves (`VACUUM FULL` after declaring); a row of another JSON type under a path keeps the normal plan for the queries that read it. The declaration is the extension table `sift.table_paths` (dumped with the database, cleared when the table is dropped). No format change; the extension SQL goes from 3.1 to 3.2 (`pg_sift--3.2.sql`, `pg_sift--3.1--3.2.sql`): run `ALTER EXTENSION pg_sift UPDATE TO '3.2'` in every database. Tests `tbl_paths`, `test/t/extension_lifecycle.sh`, `scripts/coldiff.sh --table` runs the fixture corpus on a `sift_table`.

## 3.1.0 - 2026-10-19

Feature release. The extension SQL goes from 3.0 to 3.1: install the new files and run `ALTER EXTENSION pg_sift UPDATE TO '3.1'` in every database. BM25 indexes, `sift_col` indexes and `sift_table` data of 3.0.x keep working across it, with no `REINDEX`; what 3.1 writes afterwards is in newer formats (`docs/FORMATS.md`) that a 3.0.x library refuses. Highlights: smaller BM25 indexes (segment format 3), rollups of the elements of a list that answer any `GROUP BY` over their dimensions and refresh on demand, a `sift_table` write path with per-row versions, field weights and facet counts, the JSON type tag of `jsonb` leaves and calendar group keys, aggregates from the bitmap of a chunk, and an opt-in Vulkan backend (off by default, not part of the packages).

Changes:

- Changed: the opt-in valgrind lane (`SIFT_CI_JOBS=valgrind scripts/ci.sh quick`) runs in a pinned Debian bookworm container (`SIFT_VALGRIND_CONTAINER`) on a host whose glibc has no debug symbols, and `scripts/pg-assert.sh` works without `pg_config`. Tooling only, not part of any gate.
- Added: a group key built of `date_trunc` (second to millennium), `::date` or `extract` (minute to millennium, `dow`, `doy`, `week`) of a `timestamp` column is evaluated by PostgreSQL once per second, minute, hour or day step of the data instead of once per row, and the planner estimates its groups from the column statistics: month-over-month questions on a `sift_table`, which stores no month leaf, now plan as `SiftAgg` by default (`docs/COL-PLAN.md`, calendar keys; `timestamptz` keys stay combos). 1M orders on a `sift_table`, one backend, same rows: revenue per month 244 to 9.1 ms, distinct customers per month 945 ms (a `GroupAggregate`) to 12.7 ms, revenue per month and category 193 to 11.5 ms; with 1 or 3 workers the answer is as fast and no worker is started. Test `agg_bucket`.
- Changed: `UPDATE`, `SELECT ... FOR UPDATE` and a point read of a `sift_table` after a transaction that wrote cost about 1.2M instructions instead of 9M (100,000 rows, a random row, one statement per transaction). The scan that fetches rows by TID was dropped whenever the backend's xmin moved, which every write does; it is kept while the directory has not changed (`test/t/013_tbl_fetch_xmin.pl`).
- Added: three jsonb shapes run in `SiftAgg` instead of the normal plan, without a format change: `jsonb_array_elements_text` over an array of strings (`tags[] text`), the tests `?`, `?|`, `?&` and `@>` on that array, and `jsonb_array_length` of an indexed array as a key, a filter or an argument (100,000 documents: 85 to 2.6, 33 to 1.8 and 42 to 3.0 ms). A number or a boolean among the strings raises the pushdown's usual error (its hint turns the pushdown off); a JSON null where `jsonb_array_length` looks counts as absent, where the normal plan raises an error. Test `agg_jsonb_array`.
- Added: the path type `astext` (`scores.* astext`, `items[]{score astext}`): a leaf that holds what `->>` gives for a string, a number (as `numeric_out` prints it) or a boolean, so a regular expression or any text function on the text of a number runs on the column. Index definition only, no segment change (a 3.0 library reads it as a text leaf). `jsonb_each_text` with a regular expression on the value and `e->>'score' ~ ...` over an array of objects (100,000 documents): 417 to 7.0 and 408 to 6.6 ms. Never used for `@>`, `?`, jsonpath or `jsonb_each`. Test `agg_jsonb_astext`.
- Added: the extension SQL goes from 3.0 to 3.1 (`pg_sift--3.1.sql`, `pg_sift--3.0--3.1.sql`, `default_version = '3.1'`): `sift.col_segments` lists the column format minor of each segment (`format`) and the bytes of its pre-aggregates (`preagg_bytes`), `sift.table_segments` the format. Run `ALTER EXTENSION pg_sift UPDATE TO '3.1'` in every database after installing the files; `sift_col` indexes, `sift_table` data and BM25 indexes keep working across it. Test `col_format4`, `test/t/extension_lifecycle.sh`.
- Added: column format minor 4 keeps the JSON type of every value a `sift_col` leaf could not hold (half a byte per such value, nothing for a chunk without one; `sift.col_verify` checks it against the heap), and `jsonb_typeof(path) = 'number'` (also `string`, `boolean`, `object`, `array`, and `IN` over them) on a scalar leaf of the index runs in `SiftAgg`: the rows of the other types are dropped without the overflow error (100,000 documents, a number among strings: 100 to 2.1 ms; 1,000,000: 396 to 11.9 ms). A kept row whose value the leaf cannot hold, or whose type an index written before 3.1 did not keep (the hint is `REINDEX`), still raises the error of the pushdown. Every segment 3.1 writes is minor 4, which a 3.0.x library refuses (`sift.col_format_minor = 3` keeps writing minor 3). Tests `col_tags`, `agg_jsonb_typeof`, `agg_jsonb_typeof_parallel`.
- Added: the other `jsonb_typeof` shapes run in `SiftAgg` too, on a scalar or `astext` leaf of a `sift_col` index: `<>` and `NOT IN` (a path with no value is NULL and is dropped, as PostgreSQL does), `= 'null'` (a JSON null, which minor 4 now tells from a missing key by a kind bitmap per chunk, only where the chunk has one), `IS [NOT] NULL`, the type as a `GROUP BY` key (the five types, `'null'` and a NULL group), and the type in an `OR`, `NOT`, `FILTER`, `CASE` or function. A query that reads no value of the leaf no longer raises for a kept row the leaf cannot hold. An index built by a prerelease 3.1 has no kinds and answers `= 'null'` and `astext` types with the hint `REINDEX`. Not taken: `sift_table` (one opaque jsonb leaf, needs declared paths) and a guard of the same path's values in an expression. Tests `agg_jsonb_typeof`, `agg_jsonb_typeof_parallel`, `col_tags`, the `k_typeof_*` fixtures.
- Added: column format minor 4 stores a **rollup of the elements of a list** in the pre-aggregates of a unit: the distinct tuples of up to 6 low-cardinality leaves of an array of objects, an array of scalars or a map (the dims) with the element count and, per numeric leaf, the count, sum, min and max. `GROUP BY` any subset of the dims (`GROUP BY e.kind, e.flag` of `jsonb_to_recordset`, `jsonb_array_elements`, `jsonb_each` keys), tests on the dims (`=`, `<>`, `IN`, `IS NULL`, `LIKE`, `OR`) and `count`, `sum`, `avg`, `min`, `max` are answered from the tuples of the units that are all visible (`Units From Pre-aggregates`, `Pre-aggregate Dims`; the others are scanned in the same plan, also by parallel workers). 100,000 documents, 28 units: `GROUP BY kind` 3.1 to 0.12 ms and the two-key `GROUP BY kind, flag HAVING` 3.9 to 0.13 ms; 1,000,000 documents, 276 units: 28.9 to 2.0 ms and 33 to 1.3 ms; the sidecar grows 1.7 kB a unit (+2.0% of the index) and a serial `CREATE INDEX` about 4% (instructions). Count(DISTINCT), percentiles, a test on a leaf that is not a dim and keys of more than 4096 tuples a unit scan as before. A minor 3 segment has no rollups and is read as before; a 3.0.x library refuses a minor 4 segment. Tests `col_preagg_elem`, `agg_preagg_elem`, `agg_preagg_rollup`, `agg_preagg_elem_parallel`, the `pa_elem_*` and `pa_roll_*` fixtures and the crash and replica TAPs.
- Added: the `sift_col` option `rollup` names the dims of the rollups (`WITH (paths = 'items[]{kind text, sub text, flag text, score float8}', rollup = 'items[].kind, items[].sub')`; `rollup = 'off'` disables them) and is checked at `CREATE INDEX` and `ALTER INDEX ... SET`: an error names a leaf that is not declared in `paths`, is not a leaf of an array element or a map value, or is not a boolean, integer, date, numeric or text leaf. Without it the unit takes the leaves of at most a 32nd of its elements (the dim of most distinct values is dropped first) within 2% of its bytes.
- Added: a `SELECT` that scans exact units only because their rollup is missing, has other dims or lacks a measure queues the segment (`Rollup Refresh Queued` in `EXPLAIN ANALYZE`) and a background worker rewrites it with the options the index has now; the query writes nothing, its result is the same, a hot standby never queues, VACUUM, merges and writers are not blocked and a crash leaves the old segment. 1,000,000 documents: every unit answers from the rollups 1.9 to 3.1 s after `ALTER INDEX ... SET (rollup = ...)`. `sift.col_rollup_on_demand` (on), the index option `rollup_on_demand` (true) and `sift.col_rollup_worker` (on) turn it off; `sift.col_rollup_state`, `sift.col_rollup_requests` and `sift.col_rollup_refresh(index)` are in the extension SQL of 3.1 (fresh install and `ALTER EXTENSION pg_sift UPDATE` from 3.0). Tests `col_rollup_refresh`, the isolation spec `col_rollup_refresh`, `test/t/col_rollup_ondemand.sh`, `test/t/extension_lifecycle.sh`.
- Changed: `count`, `sum` and `avg` (with or without `FILTER`, ungrouped or grouped by text columns) of an integer or float column or of the elements of an array of objects are aggregated from the bitmap of the rows of a chunk instead of a list of tuples (`sift.agg_dense`, `Dense Chunks` in EXPLAIN ANALYZE; `docs/COL-PLAN.md`): `count(*) FILTER (WHERE ..)` per flag over 387,593 elements costs 10.9M instructions instead of 34.0M, `GROUP BY` kind with `count` and `avg` 26.9M instead of 41.1M. The sums add in the same order, so the results do not change.
- Docs: `docs/COL-SHRED.md` measures jsonb arrays and objects on `sift_col` against plain PostgreSQL, GIN and ClickHouse arrays (bench files in `bench/sql/jsonb`) and proposes paths for `sift_table` and element rollups for the pre-aggregates (column format minor 4); `docs/COL-TAM.md` documents why the file of a `sift_table` keeps the write pages of a load after a first VACUUM on an idle database.
- Changed: the benchmark harness (`bench/bench`) runs by size (`--size smoke|quick|standard|full`, `bench/bench --list`), caches the cells of the other engines by key and host, resumes an interrupted run (`bench/bench resume`, exit status 75), measures the single-client place cells in one persistent libpq process and compares two result files with a bootstrap interval (`bench/bench compare`); `bench/bench final` is the staged release pass and replaces `bench/analytics`. Tooling only, no change to the extension.
- Added (3.1, experimental, opt-in): Vulkan compute for aggregates (`docs/GPU.md`). `make VULKAN=1` builds a backend that runs the grouped and filtered integer counts and sums of a `sift_col` index on a Vulkan device: `count(*)`, `count(col)`, `sum` and `avg` of integer and `numeric` columns, at most four comparisons, one key of an integer or dictionary-coded column with at most 64 values per unit, exact (16-bit halves added in 128 bits, so the result does not depend on the order). Settings `sift.gpu` (`off` by default; `auto` takes grouped aggregates only, `on` also the ungrouped), `sift.gpu_min_rows` (5000000, for `auto`), `sift.gpu_device` and `sift.gpu_cache_mb` (256); they exist and are forced off in a build without Vulkan (one NOTICE, the CPU answers). Clean, all-visible stored units run on the device and the others on the CPU in the same query; the compressed chunks stay on the device between queries (per backend). A device that fails runs the query on the CPU and is not used again in the session (one NOTICE). `EXPLAIN ANALYZE` shows `GPU: units n / total`, the device and the upload, kernel and readback times. Float aggregates, `min`/`max`, text filters and `sift_table` stay on the CPU. The default build, the packages and CI have no Vulkan dependency: `libvulkan` is loaded with `dlopen`, the Vulkan declarations and the SPIR-V are checked in.
- Fixed: a wide wildcard, regex or fuzzy leaf scored 0 next to an optional phrase. In a query such as `/lev.*/ OR "two words"` whose regex (wildcard or fuzzy) leaf has more terms than `sift.max_expansions`, the documents that match the leaf lost its contribution to `sift.score()` (a document matching only the leaf scored 0, one matching both scored the phrase alone), so the ranking differed from the documented score of a wide leaf (one term, tf the number of words it matches). Counts and the set of matches were right. Wide prefixes were not affected. No `REINDEX` is needed.
- Fixed: a comparison of a `numeric(p, s)` leaf with a constant that no stored value can equal or that lies beyond every value (`n = 5.555` in a column of two decimals, `n < 1e20`, `n <> 'NaN'`) returned wrong rows when the units answered from their pre-aggregates (an ungrouped aggregate, or grouped by one column): the test no longer prunes the zones by a range it does not have. Test `agg_numeric_const_cmp` runs every operator against such constants with and without the pushdown, the pre-aggregates, the write buffer and a promotion.
- Changed: a transaction that writes through many commands (a loop of single-row `INSERT`s, `INSERT ... ON CONFLICT`, savepoints and subtransactions, several concurrent writers filling the same write buffers) no longer stops at the 128 entries of the directory. Once the directory is half full, every sealed write buffer becomes a column group whatever wrote its rows: the rows that are final are frozen, the others keep their transaction and command in the group (version vectors, column format 1.4; a run of single-row inserts takes no bits per row). Scans, index scans, bitmap scans, parallel scans, fetches, `SiftAgg`, `ANALYZE` and the index builds read the visibility of each row from them, at every isolation level, so the rows of a running or aborted transaction stay invisible to everybody else, and its own rows follow `cmin` as in a heap. `VACUUM` clears the rows of an aborted writer after the index cleanup, freezes a group once its live rows are committed and older than every snapshot, and the merges keep the versions. A loop of 10,000,000 single-row `INSERT`s into a table with a primary key, in one transaction, takes 58 s with a directory of at most 65 entries and a backend resident size of 189 MB at the default `shared_buffers` (flat from 4,000,000 to 8,000,000 rows; the resident size counts the shared buffers the backend touched); the same load rolled back is gone after `VACUUM` (the relation goes from 134,432 blocks to 7). A write that finds the directory full waits for the promotion in progress before it fails; the error is now for a write buffer that holds a deleted row `VACUUM` has not seen or a speculative insertion in progress. The load speed of a 1,000,000-row `COPY` and the cost of a single-row `INSERT` are unchanged. Tests `tbl_writepath`, `tbl_writepath` (isolation) and `014_tbl_writepath`. See `docs/COL-TAM.md`, sections 3 and 7.
- Changed: the on-disk format. A `sift_table` writes its column segments in the format `sift.col_format_minor` says (4, or 3 to keep them readable by 3.0), and a segment with rows that are not final is column format 4 whatever it says, with a version section after the superblock directory; the metapage format of a table goes to 1.3 when the first such segment is published, and is never lowered. A 3.0.x library refuses such a table with a format error (it would read the rows of a running or aborted transaction as final); `VACUUM FULL` under `sift.col_format_minor = 3` rewrites the table in files a 3.0.x library reads again (`test/t/old_lib_refusal.sh`). Groups that 3.0.1 tagged with their writer are read as constant versions, frozen by `VACUUM` when committed and dropped when aborted.
- Added: `sift.table_segments` reports `format`, `unfrozen`, `min_xid` and `max_xid` of a group (`pg_sift--3.1.sql` and the update step `pg_sift--3.0--3.1.sql` declare the 12 columns).
- Fixed: an `INSERT ... ON CONFLICT` row kept its speculative flag after the insertion was confirmed, which kept its write buffer from being promoted.
- Changed: the live-row counts are exact for a frozen group and, for a group that is not frozen, count the rows that are committed or the transaction's own (`ANALYZE` and the planner's estimate read them).
- Not done in 3.1: a write-buffer backpressure that merges first at 75% of the directory; a bulk write that goes straight to groups; promotion concurrent with the writers.
- Changed: smaller BM25 indexes (segment format 3). On 1M short titles the index goes from 11.27 to 7.93 MiB (7.88 to 4.54 MiB with `positions = off`); on 50,000 articles from 31.13 to 30.21 MiB (12.95 to 12.02 MiB). Three parts shrink: the per-page document length rows take 4 or 8 bits where the superblock's longest document allows, the posting containers of mid-frequency terms are packed (about 9 bits per posting instead of 16), and dictionary entries lose a byte of lengths and one of document frequency per term and two bytes per inline key. Queries cost the same or less (count queries 17% fewer instructions on the titles, ranked top-10 within 1%).
- Upgrade note, BM25 segment format 3: no `REINDEX`. The new library reads format 2 segments (3.0) next to format 3 ones and writes format 3 for new segments, merges and VACUUM rewrites; the metapage becomes format 1.3 with the first format 3 segment, **and a 3.0 library refuses such an index** (`sift index "NAME" has on-disk format 1.3, this build reads formats 1.0 to 1.2`, hint `REINDEX the index.`), so going back to 3.0 needs a `REINDEX` with the 3.0 library. `sift.merge(index, true)` rewrites every segment of an index in format 3 at once.
- Added: `sift.facet_counts(table, key, query, facets [, filter [, hits]])` returns the counts of several facets of one search in one call: terms facets with a top N, an exact count of the other values and a NULL bucket, range, histogram and date-histogram buckets (with gap filling, capped at 10000 buckets), arrays counted once per row, columns of any type, and the facets of a multi-select (a facet ignores its own `selected` values). Every count is exact. Each facet is its own `GROUP BY` scan (the ranges of a list share one scan of `FILTER` aggregates), a `Custom Scan (SiftAgg)` with the search as the row selection when the columns are in a `sift_col` index and the ordinary plan otherwise, so a table with no column index, a `sift_table` and a role under row-level security work. `sift.facet_sql(...)` returns the statement for `EXPLAIN`. `sift.facets` is unchanged. Tests `facet_counts`, the isolation spec `facet_counts_writes` and the column fixtures `h_search_facet_scans` and `h_search_facet_counts`.
- Added: `sift.sections(a, b, ...)` builds the key of an index with `fields` without writing `chr(29)`, and `sift.field_query(query, 'title^3, body' [, index])` weighs the fields per query: each word, prefix, wildcard and fuzzy term is searched in every listed field with the weight times its boost, a row scores the sum of its fields' BM25, and the match set is that of the unweighted query. Phrases, regular expressions and `NOT` terms are not scoped to a field. Test `field_weights`: the scores equal a per-field SQL reference and the top-k equals the full ranking for 60 weight combinations.
- Docs: [docs/FIELDS-FACETS.md](docs/FIELDS-FACETS.md) has the design of both and what is left out (BM25F, a multicolumn index, an analyzer, phrases and highlights per field). The comparison row S7 (Facets) is now `Yes`; S6 (Field weights) stays `Partly`.

Upgrade notes:

- **Extension SQL:** 3.0 to 3.1 (`pg_sift--3.0--3.1.sql`). Install the files, then run `ALTER EXTENSION pg_sift UPDATE TO '3.1'` once per database. No index or table is rebuilt.
- **Formats** (`docs/FORMATS.md`): a BM25 index becomes metapage format 1.3 with its first segment of format 3; a `sift_col` index writes column format minor 4 (type tags, kinds, element rollups); a `sift_table` becomes metapage 1.3 with its first group that carries a version section. Segments of 3.0 are read next to the new ones, and a merge or `VACUUM` rewrite writes the new format.
- **A 3.0.x library refuses the new structures** with a format error and a `REINDEX` hint (the version section would otherwise show rows of running or aborted writers as final), so install 3.1 on every server that opens the data (standbys included) before the first write.
- **Going back to 3.0.x:** first rewrite every `sift_table` with the 3.1 library under `sift.col_format_minor = 3` (`VACUUM (FULL)`), which a 3.0 library reads again; then install the 3.0 files and `REINDEX` every BM25 and `sift_col` index with the 3.0 library (a release 3.1 build always writes BM25 segment format 3). There is no 3.1 to 3.0 extension script. `test/t/old_lib_refusal.sh` covers the refusal and the `VACUUM (FULL)` path.
- **Behaviour changed:** `GROUP BY` of calendar keys on a `sift_table`, the new `jsonb` shapes and the element rollups plan as `SiftAgg` by default (`EXPLAIN` shows it; `sift.enable_agg = off` restores the normal plan). `sift.score()` of a wide wildcard, regex or fuzzy leaf next to an optional phrase now includes the leaf (3.0.x scored a document that matched only that leaf 0).
- **Vulkan:** `make VULKAN=1` only; `sift.gpu` is `off` by default and forced off in the packages, which have no Vulkan dependency.

## 3.0.2 - 2026-10-19

Bug fixes for 3.0.1: the aggregate pushdown for the values a `sift_col` index cannot store (the write buffer, prepared statements, a `VACUUM` that skips the indexes, a scan that is running when another session writes, numbers out of the range of their path, a NaN in a `numeric(p, s)` column), a text test on a member of a free-key object, and a much cheaper `ORDER BY <column>, sift.score(ctid) DESC LIMIT n`. Install the new files; the extension SQL does not change (it stays 3.0), so there is no `ALTER EXTENSION pg_sift UPDATE` step, and no index or table is rebuilt.

Changes:

- Fixed: a value a `sift_col` leaf cannot store that was still in the write buffer, or written after a prepared statement's generic plan was made, still raised "a column of the aggregate scan holds values its index cannot store". The directory entry of a write buffer now records the leaves it holds such a value of: the appender sets them before its row goes in and invalidates the table's plans in every session at once (committed or not), and the planner reads the mask with the segment zones, so the query keeps the normal plan and the rows are the normal plan's (a prepared statement plans again on its next use). Queries that read other leaves stay on the scan. A scan that is already running when another session writes such a value returns the rows of its snapshot (next entry). Test `col_agg_overflow_buffer`.
- Fixed: after the rows that held a value a `sift_col` leaf cannot store were repaired, the queries that read the leaf kept the normal plan until `REINDEX` when the `VACUUM` skipped the vacuuming of the indexes (dead rows on under 2% of the heap pages: `ambulkdelete` is never called). The `VACUUM` of the index now looks up the heap row of each row that holds such a value, clears the liveness of those that are dead to every snapshot (HOT chains followed, live rows kept) and the merge rewrites the segment. Test `col_agg_overflow_vacuum`.
- Test `col_agg_overflow_range`: a JSON number out of the range of its `int2`, `int4`, `int8`, `float4` or `float8` path keeps the normal plan (its rows, or its error) in every aggregate shape.
- Fixed: a `Custom Scan (SiftAgg)` that was running, or planned, when another session wrote a value a `sift_col` leaf cannot store raised "a column of the aggregate scan holds values its index cannot store" for the unit that held it, whether the writer was in flight, rolled back or committed after the query's snapshot. Only a row the snapshot sees is an error now: before a unit is filtered, the scan checks the rows that hold such a value against the snapshot (the live ones, and the heap for their pages), so the rows are the normal plan's. Isolation spec `agg_overflow`.
- Fixed: a `numeric(p, s)` column that holds a NaN raised the same error for the kernel shapes (`min`, `max`, `sum`, `avg`, `count`, `count(DISTINCT)`, a group key, `ORDER BY ... LIMIT`, a comparison, `n + 1`) where the normal plan returns the NaN. The scan now gives PostgreSQL's numeric order and arithmetic, in segments and in the write buffer: NaN is greater than every number and equal to itself, `sum`, `avg` and the arithmetic give NaN for a NaN operand, a comparison of two expressions orders them as `numeric_cmp` does, and the cast of a NaN to an integer raises `cannot convert NaN to integer` for the rows the query takes. Test `agg_numeric_nan`; fixtures `n_numeric_nan_*`. A `sift_table` table still raises for it.
- Fixed: `WHERE col @@> q ORDER BY name, sift.score(ctid) DESC LIMIT n` (the score not the first sort key, also with a leading timestamp or through a join) scored, verified and fetched every match, and ran its searches for 1024 hits, 16 times as many, then 4 times as many, each from scratch, because a Sort between the scan and the `LIMIT` gave the scan no limit hint. A scan that nothing limits (a Sort above it, or no `LIMIT`) now ranks every match in one search (a drained `ORDER BY score DESC` on 40,122 matches: 108,994 heap checks to 80,244, two per match, the minimum of a verification and a fetch). A `LIMIT` with no `ORDER BY` is the scan's limit hint (1,874 heap checks to 16).
- Added: a keyed scan for `ORDER BY <columns of the table>, sift.score(ctid) DESC [, more] LIMIT n [OFFSET m]`. A pass over the matches in heap order keeps the rows of the best `n` leading keys and those tied with the last, and the ranker scores only them; the answer is the one of ranking every match and sorting afterwards (ties, NULLS FIRST/LAST, DESC, OFFSET, joins and prepared statements, checked against that on 4,700 generated queries). The planner picks it by cost; `EXPLAIN` shows `Order: name, score DESC` and `Key Passes`. User-space instructions of the transaction, before to after, at 100K rows (40,122 matches): `ORDER BY name, score DESC LIMIT 5` 170.5M to 53.9M, `ORDER BY created_at DESC, score DESC` 184.3M to 51.0M, through a join 160.6M to 55.2M (an owner with half of the rows: 231.3M to 54.4M), `ORDER BY id DESC LIMIT 5` 169.4M to 63.3M; at 1M rows (399,781 matches) 2124M to 543M, 2261M to 504M, 2023M to 544M and 2112M to 629M. Heap checks per match 2.7 (3.3 at 1M) to 1.07, and the heap pages read are the same cold. `ORDER BY score DESC LIMIT n` and the unordered scans are unchanged (1.00). A tie larger than the limit under an Incremental Sort takes a second pass (`ORDER BY name, score DESC, id LIMIT 5` with a large tie group: 0.85 of the instructions at 100K rows, 1.03 at 30K, 0.73 at 1M). Not covered: a partition, a leading key of another table of a join, the filter pushdown and a generic plan whose query term is unknown keep the plain plan. Regressions `rank_sort_key` and `rank_unbounded`.
- Fixed: a text test (`=`, `<>`, `<`, `IN`, `LIKE 'x%'`) on a member of a free-key object (`m.* text`, `m->>'k1' = 'v4'`) raised `sift_col: predicate on leaf N of N` instead of running in the aggregate pushdown; the planner offered the encoded-text pruning to a virtual leaf. Test `agg_freekey_text`.

Upgrade notes:

- **Extension SQL:** unchanged (3.0). Install the files; nothing else is required. The index and table formats are those of 3.0.0 and 3.0.1, so their indexes and `sift_table` tables open as they are, and going back to 3.0.1 needs nothing.
- **Behaviour changed, `numeric(p, s)` with a NaN:** the aggregate pushdown (`Custom Scan (SiftAgg)`) now follows PostgreSQL for a NaN instead of raising "a column of the aggregate scan holds values its index cannot store": NaN is greater than every number and equal to itself, `sum`, `avg` and arithmetic give NaN, and the cast of a NaN to an integer raises `cannot convert NaN to integer`. A `sift_table` table still raises for it.
- **Behaviour changed, `ORDER BY <column>, sift.score(ctid) DESC LIMIT n`:** the planner can pick the new keyed scan (`Order: name, score DESC` in `EXPLAIN`), which scores far fewer rows; the rows returned are the same as ranking every match and sorting afterwards. A scan with no limit (a Sort above it, or no `LIMIT`) ranks every match in one search. Plans of these queries change; their results do not.
- **Values the index cannot store:** queries that read a leaf holding such a value keep the normal plan (its rows, or its error) also while the value is in the write buffer, in a prepared statement planned before it was written, after a `VACUUM` that skipped the indexes, and for a scan that was running when another session wrote it. A prepared statement plans again on its next use.
- **Free-key objects:** a text test on a member of a free-key object runs in the aggregate pushdown instead of raising `sift_col: predicate on leaf N of N`.

## 3.0.1 - 2026-10-19

Bug fixes for 3.0.0: a `COPY` or `INSERT ... SELECT` of any size into a `sift_table` in one transaction, four aggregate pushdown fixes (a crash, a wrong refusal, a stale overflow flag and a missed row) and the CI test harness. Install the new files; the extension SQL does not change (it stays 3.0), so there is no `ALTER EXTENSION pg_sift UPDATE` step, and no index or table is rebuilt.

Fixes:

- Fixed: a single `COPY` or `INSERT ... SELECT` into a `sift_table` is no longer limited to about 500 MB (about 4 million rows of the orders layout): it failed with `Too many write buffers waiting for promotion` because the rows of a running transaction cannot be promoted into columns. Once the directory is half full, the write buffers that one command of one transaction wrote become column groups that carry the writer's transaction and command, are invisible to everybody else until it commits (at every isolation level, through scans, indexes, bitmap scans, parallel plans and `SiftAgg`), and are dropped by `VACUUM` if it aborts (a rollback, a savepoint rolled back, a crash) or frozen once it is committed. A 5,000,000-row `COPY` into a table with a primary key, in one transaction, loads in 18 s with a backend peak of 182 MB; 10,000,000 rows take 32 s and 180 MB; the load speed of a 1,000,000-row `COPY` is unchanged. A transaction that writes through many commands (a loop of single-row `INSERT`s) or concurrent writers into the same write buffers are still limited by the directory, and the error now says so. Upgrade note: a library of 3.0.0 reads a group that still carries a writer (not yet frozen by `VACUUM`) as frozen, so the rows of a rolled-back load would show up: run `VACUUM` on the tables loaded by this version before going back to 3.0.0. See `docs/COL-TAM.md`, sections 3 and 7.
- Fixed: a backend crash (SIGSEGV) in `Custom Scan (SiftAgg)` for per-item aggregates under an outer aggregate (the arms of a `UNION ALL` over a shared CTE) when an inner aggregate has a `FILTER` on a text leaf (`count(*) FILTER (WHERE p = 'x')`). The scan did not register the predicates of the inner aggregates' `FILTER`. Test `col_agg_arms`.
- Fixed: a query that reads a `sift_col` leaf a unit flags with an overflow value (a JSON string in a `float8` path, a text over its declared length) raised "a column of the aggregate scan holds values its index cannot store" where the normal plan reads the value from the heap. The planner now keeps the normal plan for such a query, so the rows are the normal plan's; the queries that read other leaves are still pushed down. The write buffer (until `sift.col_promote`) and a cached generic plan made before the value was written still raise the error. Test `col_agg_overflow`.
- Fixed: the overflow flag of a unit stayed after the rows that held the value were repaired and vacuumed, until `REINDEX`. `VACUUM` now rewrites a lone segment that flags the overflow of a leaf only in dead rows, whatever its share of dead rows.
- Fixed: `ORDER BY <column> LIMIT n` through the top-N scan of a `sift_col` index returned other rows than PostgreSQL when an `UPDATE` had moved a row: the new version of an updated row is stored wherever the heap has room, with any value, and the scan stopped at the first window whose zone had nothing better, so a row moved to a middle superblock was missed. A window is now read when the zones of its units, taken together, can beat the n-th row (the probes only choose the visit order); the in-order cases cost the same instructions as before (1.00x at 80 superblocks). Test `col_topn_moved`.
- Test infrastructure, no product impact: the CI harness flakes of the 3.0.0 release commit are fixed with a test that fails first. The jobs that install a library flavor run after the lanes before PostgreSQL 18; the regression clusters run without autovacuum, so `VACUUM` removes what a test deleted; the `VACUUM`s of `col_preagg_maint` wait for the cleanup lock; the standby of the WAL consistency test reserves WAL from the creation of its slot; `query_dos` no longer leaves a 1 ms timeout armed for its own `RESET`. `scripts/release.flakes` is empty.

Upgrade notes:

- **Extension SQL:** unchanged (3.0). Install the files; nothing else is required. The index and table formats are those of 3.0.0, so 3.0.0 indexes and `sift_table` tables open as they are.
- **Going back to 3.0.0:** run `VACUUM` on every `sift_table` loaded by 3.0.1 first. A `COPY` or `INSERT ... SELECT` too big for the write buffers is stored as column groups that carry the writer's transaction until `VACUUM` freezes them; a 3.0.0 library reads a group that is not yet frozen as frozen, so the rows of a rolled-back load would show up.
- **`sift_col` overflow:** a query that reads a leaf which a unit flags with an overflow value (a JSON string in a `float8` path, a text over its declared length) now runs the normal plan and returns its rows; queries that read other leaves are still pushed down. The write buffer (until `sift.col_promote`) and a cached generic plan made before the value was written still raise "a column of the aggregate scan holds values its index cannot store". `VACUUM` clears the flag when only dead rows held the value.
- **Limit of 3.0.0 removed:** the "Known issue" of 3.0.0 below (a `COPY` of more than about 500 MB failing with `Too many write buffers waiting for promotion`) is fixed. A transaction that writes through many commands (a loop of single-row `INSERT`s) or concurrent writers into the same write buffers are still limited by the directory, and the error now says so.

## 3.0.0 - 2026-10-19

The column track joins the BM25 index in one extension. Install the new files, then run `ALTER EXTENSION pg_sift UPDATE TO '3.0'` in every database (the extension SQL goes from 1.2 to 3.0; `pg_sift--1.2--3.0.sql`). Existing BM25 indexes keep working with no `REINDEX`.

Highlights:

- **Column engine**: `sift_col`, an index access method that stores the columns of a heap table once more, in compressed columns with zone maps, and `sift_table` (`CREATE TABLE ... USING sift_table`), a table access method with native MVCC that also takes BM25 indexes. `GROUP BY` queries with `count`, `count(DISTINCT)`, `sum`, `avg`, `min` and `max`, filters, `GROUPING SETS`/`ROLLUP`/`CUBE` and text search as the row selection run as `Custom Scan (SiftAgg)` on the column data when the planner can prove the result is the plain plan's; anything else keeps its plan (`SET sift.enable_agg = off` turns the pushdown off). Every unit is written with its pre-aggregates (exact sums and histograms, `docs/COL-PREAGG.md`).
- **Smaller and faster segments:** the chunk codec picks the smallest cascade of encodings per chunk, filters run on the encoded form with scalar, AVX2 and AVX-512 kernels that give identical results (`sift.simd`), and the dictionaries of a segment use it too (column format 1, minor 3). `CREATE INDEX` and `REINDEX` of a `sift_col` index are parallel.
- **BM25:** a segment keeps its live documents, so a vacuumed-but-not-yet-merged index counts and scores as its live documents (`count(*)` costs 1.16 to 1.28 times the clean index's instructions at 10 to 45% deleted, was 6.6 to 6.7 times), and `sift.score()` before and after a merge are equal. `WHERE col @@> q ORDER BY <columns> LIMIT n` is answered from the column units.
- **One term, the write buffer:** "mutable segment", "mutable lane" and "delta segment" are now the write buffer. The reloption `mutable_limit` is `write_buffer_limit`; `mutable_limit` stays accepted as a deprecated alias.
- **Fixes carried from 2.0.3:** `count(*)` and `sift.facets` no longer count rows deleted before the query's snapshot when a `VACUUM` runs at the same time (the same race is fixed in the column scans), and two more corrupt-index hardenings (a damaged index raises a corruption error instead of reading past the data or crashing a backend).

Upgrade notes:

- **Extension SQL:** `ALTER EXTENSION pg_sift UPDATE TO '3.0'` in every database after installing the files.
- **BM25 indexes:** no `REINDEX`. New segments are written in segment format 2 and the metapage becomes format 1.2 with the first one; **a 2.x library refuses such an index**, so going back to 2.x needs a `REINDEX` there. Older segments are read as they are and rewritten as format 2 when merged.
- **Column indexes:** `sift_col` and `sift_table` are new in 3.0, so no 2.x index needs a rebuild. A `sift_col` index built by a pre-release build of 3.0 (column format minor below 3) is refused with the hint `REINDEX`.
- **Renames:** `mutable_limit` is `write_buffer_limit`, with the old name accepted as an alias.
- **Known issue (fixed in 3.0.1):** a single transaction can write about 500 MB (about 4 million rows of the orders layout) into a `sift_table`; a larger `COPY` fails with `Too many write buffers waiting for promotion`. Load larger data in several transactions, for example 1,000,000 rows each (`docs/USAGE.md`, Limits).
- **Test infrastructure, no product impact:** GitHub CI on the 3.0.0 release commit showed three harness flakes, all fixed in 3.0.1: PG17 `difftest` (the `hardened` job reinstalls `pg_sift.so` under the running lanes: "Permission denied"), PG18 `plan_estimate` (a snapshot older than the `DELETE` keeps the rows a `VACUUM` would remove) and PG18 `tap-004_wal_consistency` (the standby of one phase has no replication slot). The release gate (`scripts/ci.sh release`) passed 17 of 17 jobs on this tree.

- Changed (3.0): the "mutable segment", "mutable lane", "write lane" and "delta segment" are one term, the **write buffer**. The reloption `mutable_limit` is now `write_buffer_limit` (kB, 64-1048576, default 4096); `mutable_limit` stays accepted as a deprecated alias with the same bounds, `ALTER INDEX .. SET / RESET` and a quiet restore of dumps of 1.x and 2.0 (`pg_class.reloptions` keeps what was typed; the new name wins when both are set). The `kind` of `sift.segments` is `buffer` and `sealed buffer` instead of `mutable` and `sealed` (`immutable` is unchanged), so a script that matches the old values has to change; the texts of corruption and verify messages say "Write buffer" where they said "Mutable". On-disk formats are unchanged. Only new in 3.0, so nothing released changes there: `sift_col`, the settings `sift.table_write_buffer_limit` and `sift.table_inline_promote`, the kinds of `sift.table_segments` and `sift.col_segments`, `sift.col_index_info.buffers`.
- Security: a damaged index whose dictionary entry stores 4294967295 as the count of a term's tf nibbles no longer makes a query read about 4 KB
  past the term's blob. The byte counts of the term's L1 bits and nibbles were summed in 32 bits, so that count wrapped to zero and passed the
  check that the stream is inside the blob; they are 64-bit now and the term raises the corruption error. Bytes that are not what the writer
  wrote (a bad disk, a torn page, a file edited with the server stopped) were the only way in.
- Security: a damaged index whose term blob ends between its codec and the next 4-byte boundary, with a large group count, could make a query
  read far past the blob (a crash). The length left after the codec wrapped around when the codec length was not a multiple of 4; the term now
  raises the corruption error.
- Added: `WHERE col @@> q [AND filters] ORDER BY <columns> LIMIT n` is answered from the `sift_col` units (`docs/COL-SEARCH.md`, "Top-N"): a bounded heap over the match set, zone maps to skip units and stop early on columns that follow the table, and the heap read for the `n` winners only. Q10 shape: 7.1 to 2.4 ms at 1M, 191 to 19.5 ms at 10M. Setting `sift.enable_topn`; `EXPLAIN` shows `Top-N:`. The planner compares it with an ordered btree on the sort column by cost: the top-N is charged the opening of the match set whole, and a correlated sort column is read best superblock first.
- Added: text search as the row selection of the aggregate pushdown. `SELECT ... FROM t WHERE <filters> AND col @@> q GROUP BY ...` on a table with a `sift` and a `sift_col` index (or `sift_table`) takes the matches from the BM25 index without reading the heap, keeps only those rows in the column units and aggregates from columns (`docs/COL-SEARCH.md`); facet-style counts per category are the same query. The plan shows `Search:` in `EXPLAIN`.
- Added (3.0, `ALTER EXTENSION pg_sift UPDATE TO '3.0'`): the column track, one extension and one set of kernels.
  `sift_col` is an index access method that stores the columns of a heap table once more, column by column with zone maps
; jsonb attributes become typed columns for the paths of its `paths` option (shredder, `docs/COL-SHRED.md`);
  `sift_table` (`CREATE TABLE ... USING sift_table`, `docs/COL-TAM.md`) is a table access method with native MVCC that also takes BM25
  indexes, with `SiftAgg` aggregate pushdown over its column segments; its limits and the one case that needs `shared_preload_libraries` (publishing it) are in `docs/USAGE.md`. A plain scan decodes only the columns a query reads, scans go both ways (`SCROLL` cursors), `VACUUM FULL` and `CLUSTER` write column
  segments directly, and `VACUUM` counts dead rows as a heap does, rewrites mostly dead segments (`sift.table_dead_rewrite_pct`) and gives the free end of the file back. The scripts are
  `pg_sift--3.0.sql` and `pg_sift--1.2--3.0.sql` (no second extension). Settings: `sift.enable_col`, `sift.enable_agg`, `sift.simd`
  (all kernels). The library reports 3.0.
  `CREATE INDEX` and `REINDEX` of a `sift_col` index are parallel on large tables (`max_parallel_maintenance_workers`): each process writes the segments of the superblocks it claims.

- Changed (3.0): segment format 2 for vacuumed documents. A segment now keeps its live documents by offset in each superblock and, for each
  term, the number of its postings that VACUUM dropped. A dirty segment (vacuumed documents, not yet merged) therefore counts and scores as
  its live documents: `count(*)` of a term costs 1.16 to 1.28 times the clean index's in user instructions at 10 to 45% deleted (was 6.6 to 6.7
  times with 2.0.1), and N, df and avgdl exclude vacuumed documents, so `sift.score()` before and after the merge are equal (`live_stats` test).
  Indexes of 2.x keep working with no `REINDEX`: their segments are read as they are (the old statistics, until they are merged) and rewritten
  as format 2 when merged; the metapage becomes format 1.2 with the first format 2 segment, and a 2.x library refuses such an index. The
  SQLite and MariaDB hosts keep writing format 1. The index grows by 4 bytes per term of each segment (+1.4 to 2.4% in `scripts/merge-bytes.sh`).
  Queries that read postings (OR, rare terms, scans) still pay for the dead ones until the merge (up to 2 times at 45% dead).
  Stored 2.0.0 files are the fixture of the `compat_2x_fixture` test.

For each chunk it picks, from exact sizes, the
  smallest cascade of constant, frame of reference (vertical bit-packing or plain bytes), delta, run-length, sorted dictionary, decimal-float
  (ALP-style) and common-divisor (timestamps of whole seconds, amounts in cents) encodings, and layers LZ4 or zstd on a stream only where the cost rule says the saved bytes are worth the decode time. Filters run on the
  encoded form with scalar, AVX2 and AVX-512 kernels that are bit-identical and selected by `sift.simd`; what is written never depends on the CPU.
  A damaged chunk is a clean `ERRCODE_DATA_CORRUPTED` error (corruption fuzz under ASan/UBSan). The per-column `compression` policy (`auto`, `none`,
  `lz4[:N]`, `zstd[:N]`, explicit encodings) is parsed here and applied by the column engine (reloption `compression`, per column `'auto; col=zstd:9'`). Build: LZ4 and zstd are linked when their development packages are
  installed (`liblz4-dev`, `libzstd-dev`; the packages depend on them); `make SIFT_NO_LZ4=1` / `SIFT_NO_ZSTD=1` build without one. Tests: `colcodec_basic`,
  `colcodec_simd` and `make -C tools/colcodec test`.

- Added (column format 1.2, clean break from the pre-release 1.1): pre-aggregates of every unit, written with it by every path that writes a unit: exact 128-bit sums and sums of squares of the integer columns and, per low-cardinality column (dictionary text, integers, dates, `numeric(p, s)`), a histogram with the row count and the non-NULL count and sum of up to three measures per group, so counts and sums per country, category or month need no scan of the units that are clean and all-visible. Reloption `preagg` (`default`, `off`, `cube(dim: measure, ...; ...)`); about 1% of the index on the store benchmark. `sift.col_unit_agg`, `sift.col_unit_hist` and the columns `preagg_bytes` and `preagg_units` of `sift.col_index_info` show them (`docs/COL-PREAGG.md`). Segments written by a build before this one are refused with a REINDEX hint.

- Changed (column format 1.1, clean break from the pre-release 1.0): the leaf type in the stored schema is a host-independent tag
  (`SiftColType`: bool, int2/4/8, float4/8, numeric, text, bytea, date, timestamp(tz), uuid, jsonb, other) instead of a PostgreSQL type OID, so
  the format and its schema hash port to other hosts; the OIDs are mapped at the PostgreSQL boundary only (`sift_col_type_of_oid`,
  `sift_col_def_typid`). Segments written by a build before this one are refused with a REINDEX hint.

- Added (column format 1.1): text and bytea of any length in the column engine. A string longer than 255 bytes (or a dictionary past 64 MiB) is
  stored inline in its chunk with a sortable 8-byte prefix, so there is no length limit below 1 GB per unit and the `0A000` that `SiftAgg` raised
  for long strings is gone. Equality, `IN`, comparisons, prefix and length tests run on the encoded form (prefix zone maps prune units; the
  prefix filters rows before any string is compared); sources can offer the same to the planner through the optional `prune_str` / `filter_str`
  callbacks of `src/agg/sift_colsrc.h`. `sift.col_leaves` has a new column `inline_chunks`. Mutable-lane strings have no length limit either.

- Added: `CREATE INDEX ... `char(n)` and
  `name` are pushed to `SiftAgg` like text; a `bpchar` without a length is stored but never filtered or grouped by the column plans, because
  equal values can differ in trailing blanks. Test: `col_index_chars`.

- Fixed: a `sift_col` scan (and `SiftAgg`) counted rows that its snapshot does not see when a `VACUUM` ran after the scan began (found by the
  differential suite under the sanitizer: 280 against 218 orders, and by the `soak-col` job). The scan took "live in the index" on an
  "all-visible" heap page for "visible", and the two facts were read in the wrong order or from the wrong generation: the liveness of a unit
  came from the directory the scan copied at its start (or a cached flag) while the VM was read later, after a `VACUUM` had removed the dead
  tuples and made their pages all-visible; a merge that replaced the scan's segments meant that no later `VACUUM` pass reached them at all;
  and the mutable lane is a copy made when the scan begins. The all-visible mask is now read first and the unit's dirty flag from its LIVE
  page after it, the shortcut is dropped once a segment of the scan's copy has left the directory, and rows of the mutable lane are always
  checked against the heap. Tests: `col_vacuum_scan` (a VACUUM, a merge, DML and promotion between the
  open and the reads of a scan) and the `soak-col` job (`scripts/colsoak.sh`). Present in `v3` as well. The BM25 count path
  (`sift_eval.c`: the dirty flags of a segment copied at its open, the all-visible mask read later) had a similar read order; it is fixed
  by the `count(*)` entry below.

- Fixed: `-0.0` in a float column keeps its sign (the column engine stored it as `+0.0`, and a `sift_table` float column holding it raised
  `0A000` under `SiftAgg`); grouping and `DISTINCT` treat `-0` and `+0` as one value, as PostgreSQL does.

- Added: aggregate pushdown on column indexes (`sift.enable_agg`, default on). A `GROUP BY` query over a table with a `sift_col` index
  (or over a lateral unnest of an indexed jsonb array or object) with `count`, `count(DISTINCT)`, `sum`, `avg`, `min` and `max`, filters,
  `IN` lists and group keys that are expressions of one column runs as `Custom Scan (SiftAgg)` on the column data when the planner can
  prove the result is the plain plan's; everything else keeps its plan (`docs/COL-PLAN.md`). `scripts/coldiff.sh` runs a corpus both
  ways and compares the results, including float tolerances, errors and the DML states of the table.

- Added: the aggregate pushdown now matches `GROUPING SETS`, `ROLLUP`, `CUBE` and `GROUPING()` (all sets in one scan, with the NULLs and
  bit masks PostgreSQL gives), float4/float8 group keys (`-0` and `0` are one group, every NaN is one group), `count`, `count(DISTINCT)`,
  `min`, `max`, `sum` and `avg` of the columns of a lookup join, and a `WHERE` on the outer query of per-order aggregates (quals on the
  keys filter the rows, quals on the inner aggregates are a per-order HAVING). Anything else keeps its plan.

- Added: numeric(p <= 18, s) predicates, group keys, expressions and distinct counts in the aggregate pushdown. A `numeric` column is an int64
  leaf with a scale: `= <> < <= > >=`, `BETWEEN`, `IN` and `IS NULL` against constants and parameters (scaled with PostgreSQL's `numeric_mul`,
  `numeric_floor` and `numeric_ceil`; NaN, infinite and out of range constants are decided without a scan), `GROUP BY` the column or an
  expression, `count(DISTINCT)`, `sum`/`avg(DISTINCT)`, and `+ - *`, `abs`, `round`, `trunc`, `floor`, `ceil` and integer casts in aggregate
  arguments, keys and filters, with the scales PostgreSQL gives and exact int128 arithmetic; division, other
  functions, casts and wider numbers are evaluated by PostgreSQL over the decoded values, once per distinct tuple of leaf values
  (`docs/COL-PLAN.md`, Expressions PostgreSQL evaluates). `GROUP BY` an `int8` column is pushed down too (`docs/COL-PLAN.md`, Numeric).

- Added: the vectorized operators of the aggregate pushdown: comparisons and range tests on
  int32, int64 and double vectors, dictionary and integer lookup tables, bitmap algebra, selection vectors, gathers, group ids, sums,
  min/max, per-group accumulation and adjacent-distinct counting, as one scalar, AVX2 and AVX-512 table chosen at run time by `sift.simd`.
  Every level returns bit-identical results (doubles follow PostgreSQL's float8 order, the float sum has one fixed accumulation order).
  The generic operations (integer comparisons, null flags, dictionary lookups, bitmap algebra, selection vectors, gathers, parent
  ordinals) are slots of the column kernels, one implementation each.
  The `aggkern_simd` regression test and `tools/aggkern` (`make test`, `make bench`) check every operator at every level against scalar.

- Tooling: `bench/bench analytics` measures aggregate queries over jsonb (objects with dynamic keys, arrays of objects) on a deterministic
  orders corpus: the queries S1-S7 on store sets of 10, 400 and 4,000 stores and the whole-table scans G1-G8 as the CPU time and instructions
  of one backend, the baseline plan against the sift_col plan, optionally against flat tables and a throwaway ClickHouse; `analytics
  clickbench` runs the 43 ClickBench queries.

- Fixed: `count(*)` through the index (and `sift.facets`, an unordered scan that returns no column, and the aggregate pushdown with a text search or a `sift_col` index) could count rows its snapshot does
  not see while a `VACUUM` ran. A `VACUUM` clears the index entries of the tuples it removes and only then marks their pages
  all-visible; a scan reads the liveness of a segment (the directory's dirty flag, the dirty-SB bitmap, the live bits, the mutable
  segments' records) before it reads the visibility-map bit of the pages, so when the `VACUUM` finished in between it took the page as
  all-visible, took the liveness as read before the removals, and counted the dead documents (rows deleted before the query began and not
  yet vacuumed: the index count was higher than the sequential count of the same snapshot, by up to the rows deleted). A merge or a promotion that
  replaced the segments under the scan did the same, because the `VACUUM` after it clears the bits of the new segments and the scan
  still read the old ones. The heap decides every row this can touch: a `VACUUM` that removed rows now ends with a directory update, and a scan
  that finds the directory changed since it read it (after the map, per superblock) does not take any page as all-visible for the rest of the scan.
  Rows returned by a search (`SELECT ... WHERE body @@> ...`) were never affected: the heap checks them. No change to the on-disk format or the SQL.
  Found while fixing the same race in the column engine; seen on 2.0.2 and every earlier release as an index count above the sequential
  count (96 differing counts in a 90 s run with 6 readers, deleters, updaters, inserters and a `VACUUM`/`sift.merge` loop; none without the `VACUUM`).
  Tests: `vacuum_agg_snapshot`, `vacuum_agg_stress` and `vacuum_count_snapshot` (the scan waits before the map read while the `VACUUM`, merge or promotion runs: clean segment, dirty segment
  whose second superblock turns dirty, merge, mutable segment, page-by-page scan) and `vacuum_count_stress` (the run above, 40 s).

## 2.0.2

No change to the extension SQL (it stays 1.2: no `ALTER EXTENSION pg_sift UPDATE` step), to the on-disk format or to the results: install the new files, no `REINDEX`. Two fixes (the first is a security hardening, the second a correctness fix) and no new behavior.

- Security: a query that read a damaged index could read outside the term it was decoding, write to the stack, or never end. The query path
  follows the lengths, counts and offsets that a term blob stores (the sizes of its blocks and groups, its Rice parameter, the extents of
  its bitmap and array containers, the nibbles and escape bytes of its tf stream) without the exhaustive checks of `sift.verify`, and used them
  unchecked. Bytes of the index that are not what the writer wrote (a bad disk or controller, a torn page, a file edited with the server
  stopped, a physical copy from a damaged or hostile source) could therefore make a backend read past the blob (an out-of-bounds read, or a
  `memcpy` with a negative length), write past a stack array, or return one group without end. Fuzzing the
  decoder (libFuzzer with ASan and UBSan) found 35 distinct crash inputs on 2.0.1 in five minutes; the decoder is now clean on all of them
  and on about 4 million further executions, and on the six reproductions of the audit. **There is no remote vector**: no SQL role can write
  index pages, so this needs damage on disk or write access to the data directory. What to do: install 2.0.2. A damaged index now fails the
  query with `ERRCODE_DATA_CORRUPTED` (`XX001`, `sift: corrupt posting data: ...`, naming what is wrong) instead; it still has to be
  rebuilt (`REINDEX`), which is needed only for an index that is actually damaged, and `sift.verify` reports it. Checks are made once per blob
  header, per group and per block, not per posting. Also fixed: `sift.verify` read one nibble past the tf stream of a term whose L1 bits
  counted more nibbles than the stream holds, and a varint of any length was accepted by the decoder (it ended on the zero bytes after the
  blob). Test: `read_corrupt` (every byte of ten term blobs, LIST and GROUPED, with and without tf, set to five values, and every length of the
  dictionary entry changed: the read finds the blob intact or raises `XX001`/`XX002`, and one case per kind of damage with its message), run
  under `SANITIZE=1` in CI. Cost, user instructions per operation against 2.0.1 (`bench/steps/cpu_gate.sh`, interleaved, 7 rounds): the
  queries 0.995x to 1.006x (the cells with a tight interval 1.000x to 1.003x), `VACUUM` after deleting 80% of the rows 1.007x (it opens
  every term blob of the segment it rewrites), `CREATE INDEX` 1.004x, `COPY`/inserts/`sift.merge`/`sift.promote` 1.000x to 1.003x.
- Fixed: a lazy `VACUUM` running next to queries could reuse the index blocks of a segment while a query was still reading them. A
  `VACUUM` (manual or autovacuum) is left out of snapshots and of the xid horizon, its own xid included, so a query that began after the
  `VACUUM` had taken its xid saw that xid as passed, and the blocks the `VACUUM` released with it (the segments it merged or rewrote) became
  reusable while the query still held the directory that listed them. The query then failed with `sift index "t" is corrupted`
  (`Block 23: expected page type 4 generation 107, found type 8 generation 0`): the page checks every read makes turn the reuse into an error;
  no wrong result was seen. Reproduced on 2.0.1 within seconds by 24 pgbench clients (inserts, deletes, and searches of 3,000 stable
  rows with a content check) with `VACUUM` in a loop, and not without the `VACUUM` loop; with the fix the same run passes (three runs). **Who is
  affected:** indexes that are searched while `VACUUM` or autovacuum runs on their table and the segments get rewritten (dead documents,
  merges by the `VACUUM`); only the reuse of released blocks changes, never the contents of a segment. What to do: install 2.0.2; the
  error is transient (retry), nothing is stored wrong, no `REINDEX`. A `VACUUM` now releases with the next xid, so the blocks it frees are
  reusable once a later transaction has started (its own truncation of the free end of the file, which holds an exclusive lock that keeps
  every query out, counts them as reusable at once, so the file still shrinks in the `VACUUM` that freed the space). Test:
  `test/t/maint_readers.sh` (inserters, deleters, searches of fixed rows with a content check, and `VACUUM` in a loop); `vacuum_truncate` takes
  a transaction between the `VACUUM` and the merge that reuses its space.
- Docs: sample queries and documents in the documentation and tests use neutral wording; no change for users.

## 2.0.1

No change to the extension SQL (it stays 1.2: no `ALTER EXTENSION pg_sift UPDATE` step), to the on-disk format or to the results: install the new files, no `REINDEX`. One behavior changes for roles under row-level security (first entry).

- Security: a role under row-level security or reading a security barrier view could learn about the rows the policy hides. The ranked scan and
  the match estimate ran before the security quals of the relation, so the BM25 statistics of the whole index (document count, document
  frequencies, average length) entered the scores of the visible rows, and the planner's costs and row estimates followed the postings of the
  hidden rows. With a policy hiding 3,000 of 3,020 rows, a visible row of a term that 20 visible documents have scored 0.000184, a value
  computed from the 3,020 documents. What a role could infer is aggregate (how many documents the hidden rows have and how common a term is
  in them); the hidden rows themselves, and the rows and counts the queries return, were never exposed. Affected: any table with a sift index
  that is read under a policy (`ENABLE ROW LEVEL SECURITY`, for the roles the policy applies to) or through a `security_barrier` view, by a
  role that can run ranked queries or `EXPLAIN`; not affected: tables without row-level security, and the table owner and the roles that
  bypass it. The match must now be securely promotable, as an index qual must be, and the estimate uses the default selectivity when the
  relation has security quals: for those roles the rows come from a sequential scan, there is no `SiftScan` or `SiftCount`, and
  `sift.score()` raises `sift.score() is not available on "t" while row-level security or a security barrier view applies`. A query that
  ranked under a policy before now fails; rank as a role that the policy does not apply to. `sift.facets` already refused tables with
  row-level security. Test: `rls_stats` (plans and `EXPLAIN` costs of two tables that differ only in their hidden rows are identical;
  no scan, count pushdown or score under a policy or a security barrier view; the owner keeps the scan).
- Performance: `count(*)` over vacuumed documents (segments with dirty superblocks, until a merge rewrites them) costs a fifth to a half of
  what it did in 2.0.0. User instructions per query, interleaved A/B against 2.0.0, 200,000 documents with 10, 25 and 45% deleted and
  vacuumed: the count of a term 0.21x, of two terms 0.49x, of three with OR 0.30x; ranked AND 0.87x to 0.88x, ranked OR 0.84x to 0.85x, the
  filtered scan 0.91x to 0.94x. Against the same live rows rebuilt, the count of a term went from 18.5 to 18.6 times the clean index's to 6.6
  to 6.7 times, two terms from 3.5 to 4.4 to 1.7 to 2.3, three with OR from 4.5 to 6.2 to 1.5 to 2.1. A dirty superblock now cuts a group out
  of its liveness bits in one pass instead of one expansion per page; a clean superblock of a segment that has dirty ones is counted without
  decoding its documents; and the page index of a superblock is built only by what reads a page's rows. **Known limit:** a count over vacuumed
  documents still costs several times a clean index's (6.6 to 6.7 times for a term) until a merge rewrites the segment; closing that gap needs
  a change of the on-disk format and is not part of this release. Clean indexes are no slower: the clean count of a term takes 0.57x of the
  instructions (a group counted in bulk no longer walks its pages), of two terms 0.95x to 0.97x and of three with OR 0.87x to 0.91x, and the
  instruction gate (`bench/steps/cpu_gate.sh`, 16 query cells on 61 segments, one segment and after `VACUUM`) shows no regression (0.98x to
  1.00x). No change of the format; counts, rows and scores are the same. The `expand` kernel takes the mask and the start bit and returns the
  popcount (the digest pinned by `kernels_simd` changes; every level stays bit-identical). `bench/dirty-query.sh` also times the count of two
  terms, of three with OR and the parallel count.
- CI: `make check` keeps its scratch small (admission of jobs by the scratch they need, 1 MB WAL segments and a small WAL for the throwaway
  clusters, the postings comparison of `index_build_postings` in four batches over the same rows); no change for users.

## 2.0.0

A major version because several behaviors change (listed first); the on-disk format is unchanged, so no `REINDEX`.

### Upgrade

- **Run `ALTER EXTENSION pg_sift UPDATE` in every database after installing the new files.** The extension SQL goes from 1.1 (what 1.4.0 shipped) to 1.2:
  `sift.segments` gains `extents` and `dead_docs` (it is dropped and recreated, so a view that selects from it must be dropped before the update and
  recreated after), and `sift.index_info(index)` and `sift.version()` are new. The packages' smoke test installs 1.0, updates it to the packaged
  version and runs the queries again. Existing indexes keep working: no `REINDEX`.

### Breaking / behaviour changes

- `sift.max_expansions` and `sift.prefix_cache_size` can only be set by a superuser (`PGC_SUSET`). A role that sets them in a session gets
  `permission denied to set parameter`. Set them in `postgresql.conf`, `ALTER SYSTEM`, `ALTER DATABASE ... SET` or `ALTER ROLE ... SET`.
- `sift.promote`, `sift.merge`, `sift.verify`, `sift.segments` and `sift.segment_postings` need the owner of the index's table or the `MAINTAIN`
  privilege on it (PostgreSQL 17 and 18: `GRANT MAINTAIN ON table TO role`), as `VACUUM` and `REINDEX` do. Monitoring roles that called them need the grant.
- A regular expression leaf cannot have back-references (`/(a+)\1/`); the query fails at parse time like any invalid expression. Rewrite it without one.
- `sift.score()` in `UPDATE`, `DELETE`, `MERGE` and `FOR UPDATE` fails when the statement is planned, saying so.
- `VACUUM` merges until the policy asks for no more, down to the new reloption `vacuum_segments` (default 2), instead of one step: a `VACUUM` of an index
  left in many segments takes about 4 times as long as before (0.08 s to 0.33 s at 60 segments of 50,000 documents) and queries after it are faster.
  Raise `vacuum_segments` (`ALTER INDEX ... SET`, no `REINDEX`) to keep `VACUUM` short. A segment more than 8 times the size of the ones before it is
  left alone, so an index may keep more segments than the target.
- `VACUUM` returns the free blocks at the end of the index file to the operating system (when they are 1000 blocks or a 16th of the index, and only if it
  gets `AccessExclusiveLock` on the index without waiting); a hot standby cancels the queries that hold the index when it replays the lock, like for a
  heap. `ALTER INDEX ... SET (vacuum_truncate = false)` turns it off.
- The maintenance worker stays up to `sift.maintenance_naptime` (default 2000 ms) after it started, holding a background worker slot for that time;
  leave room in `max_worker_processes`.
- A wildcard, regex or fuzzy leaf that expands to more than `sift.max_expansions` (512) terms now matches every word it matches and scores as one term
  instead of failing; `sift.max_expansions` is now the number of terms a leaf scores individually.
- `CREATE INDEX` at a small `maintenance_work_mem` writes more, smaller segments instead of growing its memory or failing (at the default budget the
  layout is the same), and a merge reads at most 16 GB of segments at a time.
- Source builds link without LTO unless `SIFT_LTO=1` (at most 4 link jobs); the deb, rpm and AUR packages set it, so packages are built with LTO as before.

### Changes

- Fix: a ranked scan with `LIMIT` plus `OFFSET` of 67,108,864 or more failed with `invalid memory alloc request size` (so did
  `LIMIT 2147483647` meaning "all"), and so did a ranked scan drained without a limit once it had more than 16.7 million matches. The
  top-k heap now grows with the hits found, in huge allocations, and counts in 64 bits: a huge limit costs what the matches take.

- Fix: the score upper bounds that prune ranked scans were derived for up to 256 scored terms, but a plan has a term for every distinct
  term of its phrases (up to 256 phrases of 128 terms). Their slack now grows with the number of terms above 256 (nothing changes below
  it: same bounds, same pruning), so a document matching thousands of terms can no longer score above its block's bound and be pruned.
  No wrong result was found at 300 terms (`rank_leaves` compares the top k with an exact reference).

- Fix: a wildcard, regex or fuzzy leaf that expands to more than `sift.max_expansions` (512) terms failed with `the leaf at position N
  expands to more than 512 terms` in execution (`*ing`, `/.*tion/` or `~2` on a natural vocabulary; `EXPLAIN` worked). Like a prefix, it
  now matches every word it matches and scores as one term (the documents with such a word, tf the number of those words); a leaf with
  512 terms or fewer is unchanged. Cost on 100,000 articles (456,027 terms): `*ing` (72,478 documents) 90-170 ms, 42 ms with the blob in
  the backend's cache; `/.*tion/` 320 ms. `sift.max_expansions` is now the number of terms a leaf scores individually, for every kind.
- Changed: `sift.max_expansions` and `sift.prefix_cache_size` are superuser settings (`PGC_SUSET`) instead of settable by any role: with
  up to 1,000,000 expansions and 1 GB of cache per backend they let any role exhaust the server's memory. Set them in `postgresql.conf`,
  `ALTER SYSTEM`, `ALTER DATABASE ... SET` or `ALTER ROLE ... SET`; a role that sets them in a session gets `permission denied to set parameter`.
- Fix: a wide prefix gathered four bytes of memory per posting of every term it expands to, without bound (`t*` on a billion postings is
  4 GB). A pass now gathers at most 128 MB, then counts the postings of every superblock and builds the superblocks in windows that fit
  it (a superblock that alone does not fit sums the term frequencies as they come, 128 kB); the result is the same blob, bit for bit. A
  prefix that fits the budget (every one before) takes the same single pass as before. The scan of the dictionary and of the postings
  of the expansion serves cancel and `statement_timeout` every 16 dictionary blocks and at every group (a buffer lock held during the scan held them off).
- Changed: a regular expression leaf cannot have back-references (`/(a+)\1/`): they make a match exponential in the length of the term.
  The query fails at parse time like any invalid expression.

- Fix: a cursor over a sparse term (fewer than 8192 postings, or inline in the dictionary) allocated the 32 kB buffer a group of a frequent
  term needs; it now allocates one the size of its rows (at most one per posting, 64 bytes each). A 200-term `OR` over six segments
  allocated 39 MB for its cursors, now 1.1 MB. Frequent terms are unchanged. The copy of a frequent term's whole blob (a term in every
  row of a billion-row index is hundreds of MB per segment and query) is not windowed yet.

- Fix: a term lookup in a dictionary whose terms share their first 8 bytes (fixed-width ids such as `ord-0000` plus digits, order
  numbers, hashes with a common prefix) read every block of the run of terms with that prefix, linearly: 625 blocks per lookup in 40,000
  ids. It finds the block by bisection (10 blocks); other dictionaries read the same blocks as before.

- Fix: every query that reads the documents of a segment (ranked scans, filters) read and summed the whole SB directory of its docmap,
  24 bytes per superblock: more than 300 kB per segment and query on a billion-row index. A backend now keeps the directories of
  segments of 128 superblocks or more (16 MB, least recently used out; a segment never changes, so its generation names the directory
  for good) and a repeated query reads none of it. Smaller segments are read each time, as before.

- Planner: the estimate of a query is computed once per statement instead of three or four times (the selectivity of the clause, the cost of each
  path, the index path). The last 8 estimates of a backend are kept under the index file, the version of its directory, the expansion cap and the
  query; a promotion, a merge, VACUUM or REINDEX changes the key. Slim tier, one backend, the same 1500 statements, user cycles per statement
  (mean +- standard error of 5 interleaved runs): count 867k +- 31k to 761k +- 15k (-12%), filter 1.86M to 1.77M (-5%), mixed 1.52M to 1.45M
  (-4%), and 846k to 811k (-4%).
- Ranker: a join (or any consumer) that keeps a small share of the rows and pulls past the limit hint makes the scan search again for 16 times
  as many hits (4 times beyond 4096) instead of 4 times, so a join that keeps a tenth of the rows searches twice instead of three times. Rows,
  scores and ties are those of the full ranking. Slim tier join class, one backend, the same 1500 statements, user cycles per query: 4.74M +- 0.02M
  to 3.90M +- 0.05M (-18%).
- Planner: a prepared statement (a generic plan, whose query the planner cannot read) now gets the filter pushdown like the literal query,
  decided when the query is known: the plan keeps the filter's bitmap as a child and the scan builds it only when the bound query matches
  enough documents for it to pay (`EXPLAIN VERBOSE` shows the number); a rare query is answered by checking the filter after the ranker.
  The plan is priced as before, so the plan cache keeps reusing it. Rows, scores and ties are unchanged. Slim tier, one backend, the same
  1500 statements, user cycles per query, prepared statements: filter class 3.23M +- 0.12M to 1.66M +- 0.06M (-49%), join class 5.93M to 4.68M,
  rare AND with a 2.3% filter 772k to 658k; with the filter keeping 0.1% of the rows the ranked or class goes from 18.7M to 2.5M and the
  filter class from 4.2M to 0.95M; at 2.3% 8.5M to 3.8M and 2.7M to 1.5M; with a filter of 10% or 50% of the rows nothing changes
  (`bench/steps/ab.sh filter_matrix`: three query classes, four filters, automatic, custom and generic plans).
- Planner: the filter pushdown is chosen for a filter on a value stored in one late stretch of the heap. The planner placed the rows of
  an equality filter on a btree column from the column's statistics, which say nothing about rows loaded by group (correlation near
  zero), and priced the scan that checks the filter after the ranker as if they were spread over the heap; it now reads where the
  entries of the key lie from the btree (a few entries, at planning time) and verifies every match before them. A filter at the start
  of the heap keeps the post-filter scan, so no plan that was fast changes. Slim places, one backend, user cycles per query: filtered
  one-letter prefix (p1) 3.52M +- 0.10M to 0.67M +- 0.01M, two-letter prefix (p2) 1.05M +- 0.03M to 0.68M +- 0.02M; the top and count workloads
  of p1 and the other article shapes are unchanged within noise. Rows, scores and ties are unchanged.
- Fix: the final aggregate of a parallel `count(*)` plan started from uninitialized costs, so the planner sometimes priced the parallel count at
  thousands more and chose a partial aggregate over a parallel scan instead (found by the sanitizer run of `parallel_scan`).
- Performance: queries over an index with vacuumed documents (segments with dirty superblocks, until a merge rewrites them) are 3 to 36
  times faster. The liveness of a page was read one document at a time, a buffer pin, lock and bit test each, which was most of the
  query; the bits of a superblock are now read once and spread over the page's offsets. 200,000 documents with a quarter deleted and
  vacuumed, ms per query (the same rows rebuilt: clean): ranked AND 1.36 to 0.26 (clean 0.26), ranked OR 2.39 to 0.70 (0.64),
  unranked filtered scan 2.11 to 0.65 (0.57), `count(*)` 1.82 to 0.05 (0.019). Clean indexes are unchanged (interleaved A/B on the same
  data, 0.93x to 1.23x with 7 to 10% error; slim bench: 0 regressions). N, df and avgdl still count vacuumed documents until a merge
.
  `bench/dirty-query.sh` measures latency and score drift against the dead fraction (deleted and vacuumed, deleted from the oldest rows,
  or rolled back).
- Fix: the memory of a merge (`sift.merge`, the maintenance worker, VACUUM's rewrite of a segment with dead documents) no longer grows with
  the size of the segments: it streams them. The documents of a segment are read a superblock at a time (a cache of the rows that
  fits `maintenance_work_mem`, in place of arrays of every key and length), a term is written from the postings of its runs without being
  collected first, and the dictionary index, and the postings of a large term (at least half of what a segment's budget holds, and the
  grouped threshold), are staged in temporary files and copied into the segment. The segments are byte for byte the ones the merge wrote
  before, and the format is unchanged. Merging 4 segments of 3M documents (690 MB) at 16MB of `maintenance_work_mem` took 140 MB of
  anonymous memory and 17.4 s, and takes 16 MB and 8.3 s now (0.4M documents: 22 to 10 MB). `scripts/merge-bytes.sh` compares the
  index files of two builds page by page, and the `mergebig` group of `scripts/perf-gate-maint.sh` measures the peak and the time.

- Fix: the memory of `CREATE INDEX` grows no more with the table at a small `maintenance_work_mem`. The runs a term in most rows makes were
  merged level after level into runs that held the term whole when written as a segment (1MB budget, a term in 9 of 10 rows: 6.4 to 31.5MB
  from 0.4M to 6.4M rows). Runs now stop merging where a segment's largest term fits the budget, and the build writes the table as more,
  smaller segments instead (about 20 for 400,000 such rows at 20,000 postings a term; at the default budget the layout is the same).

- Fix: `CREATE INDEX` no longer fails with `segment dictionary exceeds 4 GB` or `postings of one term exceed 4 GB` (the lengths and offsets
  of the format are 32 bits): the runs of a build merge, and make a segment, only while their largest term and their dictionary stay
  under those limits (estimated high, so a segment holds up to a third of them), and the build writes more segments instead.

- Fix: `CREATE INDEX` on documents of a heap page that together passed 1 GB (a few large, out-of-line values: three of 360 MB) failed with
  `string buffer exceeds maximum allowed length`: a page's documents are no longer all copied before they are analyzed, a large one is
  analyzed at once. A `maintenance_work_mem` of 4 GB or more (or a document with very many distinct terms) could fail with `inverted buffer
  pool exhausted`: the buffer is now spilled before a document that could exhaust its pool.

- Fix: a keyword value of more than 255 bytes failed an `INSERT`, a `COPY` or the end of a long `CREATE INDEX` with `a keyword value is longer
  than 255 bytes`. It is now indexed as its first 236 bytes and a hash of all of it (the same term from a build, an insert and a query),
  so two long values that share their start stay two values and `@@>` finds exactly the rows with that value. A facet over such a value
  counts it under its start and hash instead of failing.

- `CREATE INDEX` reports its progress in `pg_stat_progress_create_index`: the phases `scanning table`, `spilling runs`, `merging runs` and
  `writing segments`, and the documents analyzed and then written (`tuples_done` of `tuples_total`), parallel workers included. A parallel
  build that gets fewer workers than requested logs it at `DEBUG1`.

- Security: `sift.promote`, `sift.merge`, `sift.verify`, `sift.segments` and `sift.segment_postings` were executable by every role with `USAGE`
  on schema `sift`, which could force rewrites of any index or hold its maintenance lock. They now need the owner of the index's table or
  the `MAINTAIN` privilege on it (as `VACUUM` and `REINDEX` do); `ALTER EXTENSION pg_sift UPDATE` to 1.2 is not needed for this.
- Fix: a query with `col @@> ANY (array)` failed in `EXPLAIN` and at run time with `invalid byte sequence for encoding "UTF8": 0x00`. It now
  plans as a bitmap scan over one index scan per element (an OR), prepared or not; `ALL` and `NOT ANY` use a sequential scan as before.
- Planner: materialized views get the ranked `SiftScan` and the `SiftCount` pushdown like tables. A prepared `LIMIT $1 [OFFSET $2]` gives the
  ranked scan of a generic plan its limit hint (a broad term with `LIMIT 10` took 1.38x less time per query on 50,000 articles; custom plans
  and other shapes are unchanged). `sift.score()` in `UPDATE`, `DELETE`, `MERGE` and `FOR UPDATE` now fails when it is planned, saying so.
- Extension 1.2 (`ALTER EXTENSION pg_sift UPDATE`): `sift.segments` gains `extents` and `dead_docs`; new `sift.index_info(index)` (relation
  size, high-water mark, free and reusable blocks, directory entries, maintenance lock held) and `sift.version()` (the library reported 1.0
  for a 1.1 extension).
- Operations: promotions, merges and maintenance worker runs of at least `sift.log_min_duration` (default 1 s, `-1` off) are logged, with
  the inserts that wait for room in the directory and a maintenance worker that could not be registered; errors of the worker name the index.
  `ALTER INDEX ... SET/RESET` of an option that changes the analyzer warns that the table's queries and inserts fail until `REINDEX`.
  `sift.highlight` no longer parses its query for every row.
- Docs: [docs/SIZING.md](docs/SIZING.md) (disk, memory and time of large indexes, dead documents in BM25 statistics, `SERIALIZABLE`, row-level
  security, updates). `bench/ab-query.sh` times one statement on two builds, `bench/sql/dead_docs.sql` measures the effect of rolled-back
  loads on scores. `bench/cpu-query.sh` counts the instructions and CPU time of a backend per transaction on two builds, which a loaded host
  does not change.
- Tests: `scripts/ci.sh quick` also runs the regression tests that need no test-only function against the release library
  (`SIFT_TESTING` off).
- New: reloption `merge_fanin` (default 8, 2 to 32): the segments of one size tier that trigger a merge, so far the constant 8. It is read when
  a merge is chosen: `ALTER INDEX ... SET` applies it with no `REINDEX`, and the default is today's behavior. The other limits follow it and
  stay consistent for any value: the merge of the smallest segments starts above `greatest(32, 2 * merge_fanin)` directory entries (32 up to a fanin of 16, as
  before), a merge takes at most twice the fanin of segments, and a build-time check proves that the largest fanin still merges before inserters must make room
  (112 of 128 entries). Tests: `merge_fanin` (every fanin from 2 to 32) and the TAP test `maint_fanin` (inserts against the worker at 2 and 32).
- New: setting `sift.maintenance_cost_delay` (milliseconds, 0 to 100, default 0): the maintenance worker sleeps that long after every 16 pages (128 kB)
  its merges read or write, like `vacuum_cost_delay`. At 0 nothing changes. `VACUUM`, `sift.merge()` and the inserters' merges are not delayed.
- New: reloption `dead_rewrite_pct` (default 50, 1 to 100): the percent of dead documents above which maintenance rewrites a lone segment, so far
  the constant 50. `ALTER INDEX ... SET` applies it to the next `VACUUM` or worker step, with no `REINDEX`; 100 never rewrites. The default is today's
  behavior: against 5 (400,000 documents, 10 rounds of 6% updated or deleted rows) it took `VACUUM` 1.9x (update) and 6.9x (delete) as long, wrote 5.3x
  the WAL on deletes and left a 1.3-1.4x larger file, for faster queries only after deletes. Tests: `dead_rewrite_pct`.
- Measured, no change: `mutable_limit` stays 4096 kB. At 1024 the 16-client insert rate was 18% lower, `INSERT .. SELECT` 17% and `COPY` 47% lower, and the
  index grew 37% larger in 4.5 times the segments, for ranked queries 2 to 4 times faster; promotions and merges were 3 times shorter but 3.7 times as many
  (`scripts/perf-gate-maint.sh --cells mut`, `--cells dead`; numbers in [docs/USAGE.md](docs/USAGE.md#maintenance-and-operations)).
- Fix: the file of an index never shrank. The space `VACUUM` and the merges free at the end of the file stayed allocated (a table that kept 5%
  of 200,000 documents: 722 blocks after the bulk delete, a `VACUUM`, `sift.merge(index, true)` and another `VACUUM`, 639 of them free).
  `VACUUM` now returns the free blocks at the end of the file to the operating system, as it does for a heap: when they are 1000 blocks or a
  16th of the index, and only if it gets `AccessExclusiveLock` on the index without waiting (a query or an insert that holds the index, or the
  maintenance worker in a step, keeps it from truncating until the next `VACUUM`); the lock is released as soon as the file is cut. That
  example: 722 blocks before, 83 after. A hot standby cancels the queries that hold the index when it replays the lock, like for a heap.
  A `VACUUM` with parallel workers (several large indexes) cannot take the lock, which needs a transaction id, and leaves it to the next
  one; `VACUUM (PARALLEL 0)` and autovacuum truncate. No on-disk change. Only blocks the horizon already lets the allocator reuse are
  given back. `ALTER INDEX ... SET (vacuum_truncate = false)` turns it off for an index (no `REINDEX`).

- Performance: `CREATE INDEX` takes fewer instructions per document and posting: the checks of the memory bounds left that path (the order of the
  documents is checked once per page, the pool once per large document, a page is sorted only when its HOT chains made it unsorted), and a
  table with few runs writes no extra runs at 1MB. Runs are merged only when the run to come does not fit a segment, not before: a level that
  already fits is read by the merge that writes it, with no pass of its own (at 1 MB that pass cost 46% more temporary bytes on a table whose
  largest term just fits).
- Fix: an index with damaged offsets in a sparse term (inline keys or a LIST blob with more than 512 offsets) reports a corrupt term, as
  `sift.verify` does, instead of crashing; a damaged docmap directory no longer shares the cache entry of an intact one.
- Build: `make` links without LTO unless `SIFT_LTO=1` (at most 4 link jobs): PGXS inherits `-flto=auto` from some distributions' `pg_config`, which
  started one link job per core in every development, CI and bench build. The deb, rpm and AUR recipes set `SIFT_LTO=1`.

- Fix: promoting a mutable segment no longer takes memory in proportion to the segment. The records were all copied to memory, about six
  times the size of the segment, whatever `maintenance_work_mem` (a segment of 138 MB at `mutable_limit = 1048576`: 827 MB at a 64 MB budget, 797 MB
  at 16 MB); they are now sorted in a tuplesort that spills to disk (191 MB and 155 MB; a segment of 46 MB at a 4 MB budget: 54 MB, where it took
  268 MB). A segment that fits the budget promotes as fast as before (1.0 to 1.2 s against 1.2 to 1.7 s at 64 MB); one that does not is slower than with
  all of it in memory (3.8 s against 1.7 s at 16 MB), the price of staying within the budget.
- Fix: a long merge no longer stalls the inserts. Promotion and merge steps now claim the segments they replace instead of holding one
  exclusive lock per index, so the background worker's merge, an inserter that finds the directory nearly full (112 of 128 entries) and
  the inserters that promote a backlog of sealed segments work on different segments side by side. Before, every such inserter waited
  for the whole merge, and the throughput of 8 bulk writers equalled that of one. VACUUM, `sift.promote()` and `sift.merge()` still run
  alone. The inserter waits only when everything left to merge is being merged by others.
- New: `sift.standby_conflicts` (superuser, default off). On a hot standby without `hot_standby_feedback`, the primary logs a recovery conflict before it reuses blocks released by a merge, so a standby scan is waited for (up to `max_standby_streaming_delay`) or cancelled by the usual conflict instead of failing with "segment recycled during scan". Default behavior is unchanged.
- Fix: a promotion or merge that fails (out of disk, a corrupt segment, an error of the host) now returns the blocks it had taken to the
  free list, where it used to leave them allocated until the next orphan sweep, and the background retry doubled its delay after each
  failure (2 s up to an hour) instead of failing again every 2 s and growing the index file each time. `VACUUM` and the inserts that
  merge for room no longer fail with the step: the error is logged as a `WARNING` and counted for the delay. An explicit `sift.promote()` or
  `sift.merge()` still raises it. A background worker that cannot be registered is logged. The wrapper and the delay add nothing to the
  insert path or to a step that succeeds.
- Fix: the file of an index grew to 2.6 times its live segments while the background worker merged a backlog of segments, and kept that size.
  The worker's one transaction held back the horizon the free extents are judged by, and the free space the merges released was too
  fragmented for the long runs a merged segment asked for. The worker now runs every promotion or merge in a transaction of its own, a
  step reuses what the previous one freed (the horizon is computed anew at each allocation), and the streams of a step take shorter runs of
  free space, down to a 256th of the size they are expected to need. 60 small segments merged by the worker: file 2.6 times the live
  segments, now 1.6. An old snapshot elsewhere (an idle transaction) still keeps what is released after it from being reused.
- Fix: a merge or promotion of a segment above about 8 GB failed with `sift segment stream needs more than 1002 extents` and, with it,
  `VACUUM`'s cleanup and the inserts that merge for room, when other writers allocated between the runs of its stream. The runs of a
  stream now grow with it (up to 64 MB), 16 GB take about 300 extents, and a merge reads at most 16 GB of segments, so a segment past
  that size is left alone (the policy takes the smaller ones of its run, `sift.merge(index, true)` too). Nothing changes below 1 GB.
- `VACUUM` merges until the policy asks for no more, instead of one step: an index that a build with a small `maintenance_work_mem` left in
  60 to 128 segments (queries 2.2x slower at 11 segments than merged) needed one VACUUM per 8 segments less; now one merges down to
  `vacuum_segments` (below). A `VACUUM` of such a backlog takes about 4 times as long as the single step it did before (0.08 s to 0.33 s at 60
  segments of 50,000 documents): deliberate maintenance work, which leaves 2 segments instead of 46 and makes the queries after it faster
  (`count(*)` 1.20 ms to 0.68 ms, ranked 2.16 ms to 1.95 ms). Queries that run while `VACUUM` merges are 6 to 7% slower, from the
  extra merging itself: the same queries with the same segments and no merge running, and the same maintenance on the old build, measure the same.
- Fix: after a COPY or bulk insert with `sift.async_maintenance` on, up to three sealed segments could stay unpromoted until the next seal or a
  `VACUUM` (queries read them as mutable data: 8 to 16 ms instead of 1.3 ms): the worker was started once per 2 s and exited when it had
  nothing left, so the seals of the rest of that window had none. The worker now stays, and looks for work, until
  `sift.maintenance_naptime` (new, default 2000 ms, 200 to 60000) after it started. It holds one worker slot for that time.
- Fix: with the default `vacuum_segments = 2`, adding a small segment to a large index made every `VACUUM` rewrite the whole index (a 0.1 MB
  segment next to 11 MB: 0.8 to 0.97 s and 11 MB of WAL at each, against 5 ms with `vacuum_segments = 3`), because the two smallest segments
  were merged whatever their sizes. A segment more than 8 times the size of the ones before it now stays, so an index may keep more than
  the target when sizes differ that much (`sift.merge(index, true)` merges them). The rewrite of a mostly dead segment was also never reached at
  a large `vacuum_segments`: the merges the target discarded hid it; it is now chosen alone.
- New: reloption `vacuum_segments` (default 2, 1 to 128): the number of segments `VACUUM` leaves an index with. A higher value is shorter
  `VACUUM` and more segments to query (60 segments of 50,000 documents: 0.19, 0.15, 0.12 and 0.10 s at 2, 4, 8 and 16, `count(*)` 0.08, 0.14,
  0.22 and 0.35 ms). `ALTER INDEX ... SET` applies it with no `REINDEX`; the worker's and the inserters' merges are unchanged.

## 1.4.0

- No change to the PostgreSQL extension's behavior, on-disk format or SQL: install the new files, no `ALTER EXTENSION` and no `REINDEX`. Indexes, scores and plans are the same as 1.3.1; the benchmark queries run at the same speed.
- The engine now builds as a portable core shared with the SQLite extension and the new MariaDB extension (`src/portable.list`, one host header `src/sift_host.h`, checked by `scripts/check-portable.sh` in `make check`). The sorted-run and merge code moved from `src/sift_build.c` to `src/sift_runs.c`, the merge policy to `src/sift_merge.c`; the compiled query path is instruction for instruction the same.
- `make check` takes about 1.5 minutes on 8 cores (it took about 4): one job queue over up to 8 lanes instead of four fixed lanes, faster differential and TAP tests, same coverage.
- `scripts/build-bench.sh` compares `CREATE INDEX` time, memory and temporary files of two builds at several budgets; the runs merged into each segment are logged at `debug2`.

## 1.3.1

- Fix: `CREATE INDEX` failed with `sift parallel build worker needs more than 256 runs` (`HINT: Increase maintenance_work_mem or build
  without parallel workers`) when a parallel worker spilled more than 256 runs, i.e. on a large table with a `maintenance_work_mem` small for
  it; a build of enough runs for more than 128 segments failed with `sift index ... would need more than 128 segments`. The build now
  merges runs in passes (the last fan-in runs of a level into one of the next, a fan-in bounded by the memory budget), so any table builds
  at any `maintenance_work_mem` (down to 1MB), in parallel or not: a small budget only makes the build slower, and when it would need more
  than 128 segments the build writes fewer, larger ones. The index answers and scores as one built with more memory. Builds at the default
  or a generous budget take as long as before; at 1MB they take up to 1.5x longer (second merge pass), where they used to fail or exceed
  the budget. `sift` logs the runs merged into each segment at `debug2`, and `scripts/build-bench.sh` compares build time, memory and temporary
  files of two builds at several budgets.

## 1.3.0

- Planner: `@@>` on a key whose index has `fields` (or any analyzer other than the default) is estimated from the index like any other. It is
  the call `sift.match_idx`, which had no estimate, so the planner expected a third of the rows for every query (a term of 44 of 1M names as
  a third of them) and costed the scan and the joins above it as for a broad term. A rare term of such an index, searched among the
  sites of a tenant with 10% of the names (`id IN (SELECT ...)` through a left join that repeats them): hash join 2.6 s, now a
  nested loop semi join under the ranked scan, 18 ms.
- Planner: a ranked scan with a `LIMIT` is costed for what it does when drained. Past its limit hint it ranks again for four times as many
  rows, to the last match, as a hash join or a sort above it makes it; its total cost was its startup plus a hundredth per row, so hash
  plans over the whole scan looked far cheaper than they are and won against the nested loop semi join that stops at the limit. 1M names,
  152k matches of `honey`, a tenant with 0.3% of the names: 190 ms, now 68 ms (`h*`: 247 to 69 ms; a rare term 9.1 to 1.6 ms); tenants
  with 2% to 50% keep their nested loop (17 and 3 ms), and one with 0.05% its hash join (about 110 ms).
- Removed: reloption `phrase_index = pairs`; phrases use word positions. `CREATE INDEX ... WITH (phrase_index = ...)` fails like any unknown
  parameter, and an index built with it is refused ("built with other analyzer options") until `ALTER INDEX ... RESET (phrase_index)` and
  `REINDEX`.
- Benchmarks: the engines are measured one at a time, each with every CPU of the host, interleaved in rounds (the CPUs are no longer shared out or capped);

## 1.2.0

- Word positions are stored by default (reloption `positions`, default on): phrase queries are decided from the index instead of
  re-tokenizing the heap text of each candidate. A phrase that every match must contain (not under `OR` or `NOT`) is verified in the
  index, and a phrase under `OR` (a CJK word becomes one) filters its candidates the same way; counts and ranked scans of such phrases
  read no heap tuple. On the 218k-article benchmark phrase queries run about 4.1x faster than with `positions = off` and 3.3x faster than pg_search,
  counts 7.3x faster. The price is a default index about 2.5x larger (52.5 MiB to 130.9 MiB on the 218k articles, pg_search: 183.6 MiB) and a
  build about 30% slower (1.7 to 2.2 s): `positions = off` restores the small index and the heap recheck of phrases. Indexes built by earlier
  versions keep working but have no positions: `REINDEX` them to get phrases decided by the index. Terms with at most two postings
  keep their positions in the dictionary entry (up to 256 bytes). Documents inserted after the build (mutable segments, and the
  segments promoted and merged from them) and phrases under `NOT` keep the heap recheck.
- On-disk format 1.1 (metapage minor version): indexes created or reindexed by this version are 1.1 and must not be opened by earlier
  releases, which write 1.0 and do not check the minor version (`REINDEX` the index there to go back). Format 1.0 indexes stay readable and work without positions until
  `REINDEX`. A build that meets a newer minor version than it knows refuses the index with "REINDEX the index".
- Counts and filters: a query with a phrase rechecks in the heap only the candidates the bitmaps and positions do not decide (an
  `OR` branch without the phrase, a `NOT` of a superset and every segment with positions are not rechecked); `count(*)` counts the
  all-visible pages of a group straight from its bitmaps; the planner no longer overcharges a pushed-down filter, so a ranked scan
  takes it when it is cheaper (a ranked `OR` with a 2% filter 6.0 -> 4.8 ms, plan choice wrong for 2 of 100 queries instead of 70);
  terms from 8K postings (was 64K) are bitmap groups with block bounds: ranked `AND` and `OR` about 20% faster and `count(*)` of
  `OR`s about 27% faster for about 10% more index (applies to new builds and `REINDEX`).
- Top-k: ranked blocks accumulate leaves by descending upper bound and score only the documents that can still reach the threshold
  (new `bm25_sel` and `prune_gt` kernels in AVX-512, AVX2 and scalar), and the bounds of dense LIST terms are read without decoding
  their postings. Scores and ties are bit-identical. Ranked `AND` and `OR` use about 27% fewer cycles, prefix top-k and boosted
  place queries 8-34% fewer.
- Fix: `sift.score()` through joins. A ranked scan under a join tree of depth two or more (a join plus a semi join, a lateral join, outer, anti and self joins, aggregates over a join) raised "must be computed by a sift scan" although the plan scanned the table with a ranked SiftScan: the joins now pass the scan's score on, also from a partitioned or inheritance parent. Regression `rank_join`.
- Prefix queries never fail on the expansion limit: a prefix with more than `sift.max_expansions` terms (`h*` or `ho*` on a large
  table without `prefix_index`) matches all the words that start with it and scores as one term, the same score `prefix_index` gives it
  (below the limit a prefix still scores as the max over its terms). `prefix_index` is now a speed and size choice, not a requirement
  (a million names: 10-15 ms for `h*` the first time in a backend, under 1 ms cached, `sift.prefix_cache_size`, against under 1 ms and
  a 2-4x larger index with it). Wildcard, regex and fuzzy leaves still raise the error above the limit. No index format change.
- PostgreSQL 17 support, with 19 (release candidate) experimental: `make` accepts 17 and newer, and the version differences live in
  `src/sift_compat.h`. PostgreSQL 17 has no `extension_control_path`, so `scripts/pgtest.sh install` copies sift into the server's own
  directories there (`SIFT_SUDO=sudo` when they are not writable) and the update-path checks of the lifecycle test are skipped. CI
  runs 17 and 18, and 19 built from git without failing the workflow.
- Fix: `name:(a b)` and `name:{a b}` lost their qualifier when the group was unboosted and the same operator as its parent (the leaves
  searched the whole text, on the index and on a seq scan alike), and `name:(NOT a)` dropped it too. Without `fields` a colon is text
  in groups as well (no more "unknown field (the index has none)"); a name an index with fields lacks lists its fields.
- Fix: with several sift indexes on one key, an index with another analyzer (one with `fields` next to a plain one) could serve `@@>`, so
  a query answered differently depending on the cheapest plan, or failed while the planner costed the index. `@@>` now means one analyzer
  (the default one when an index has it) and only indexes that answer the same serve it.
- Fix: a ranked scan with a broad match and a selective filter on a column that follows the heap order (an id range or id list at the
  end of the table) was planned as filter-after-ranker, which verifies every match before the filter's first row (tens to hundreds of
  ms against under 1 ms pushed down). The post-filter estimate now counts the matches before and after the rows the filter's btree
  quals can keep, from the column's histogram and correlation, and the bitmap's tuple count no longer includes the planner's lossy-page
  inflation. The estimate also prices the blocks the ranker scores (a broad term ties its superblock bounds, so the scan runs in heap
  order) and the descents of a `= ANY(const array)` bitmap by the leaf pages they touch rather than one random page per element, so
  1000 to 10000 scattered ids with a broad term are pushed down (418 ms against 5 ms in the report) and ids at the head of the heap
  stay on the plain scan only when it is faster.
- Fix: assertion failures on a server built with `--enable-cassert` (a stray `foreach_node(Node/Path/Plan)` cast, the GlobalVis horizon
  taken from the index instead of its heap, and a page special area read before the page layout check in `sift.verify`).
- Packages: a `.deb` (`postgresql-18-pg-sift`, PGDG on Debian 13 and Ubuntu 24.04) and an `.rpm` (`pg_sift_18`, PGDG on RHEL, Rocky and
  Alma 9 and 10), x86-64, are attached to each release with `SHA256SUMS`. `scripts/package.sh` builds them from a tag and smoke-tests
  them in containers (install, create, upgrade from 1.0, remove); the `release` workflow runs it on every `v*` tag.

## 1.1.0

- Filter pushdown: a ranked scan can take a bitmap of the other quals (btree, BRIN, GIN) as a child and score only the matches it keeps;
  `sift.filter_pushdown = off | on | force`, default `on`; EXPLAIN shows `Filter Superblocks`, `Filter Pages Exact`, `Filter Pages Lossy`.
- Keyword facets: `analyzer = keyword` indexes each whole value as one term; `sift.facets(index, facet_index, query, lim, nulls)` counts
  the matches of a query per value.
- Parallel scan: parallel-aware SiftScan and SiftCount with a shared ranked threshold, behind `sift.parallel_scan` (default off). It is
  never planned together with a pushed-down filter.
- Partitioned tables: `ORDER BY sift.score(ctid) DESC LIMIT k` through a partitioned parent ranks each partition with a ranked scan under a
  MergeAppend (scores use each partition's own statistics); `sift.max_score()` is not available through a parent.
- Fix: a crash rechecking a phrase with `count(*)` or a ranked scan on an expression index (also affects 1.0.0).
- Upgrade from 1.0: install the new files and run `ALTER EXTENSION pg_sift UPDATE`.

## 1.0.0

- First release: the `sift` index access method (`USING sift`), `text @@> text` queries (boolean, phrase, prefix, wildcard, regex, fuzzy),
  exact BM25 ranking with `sift.score()`, `count(*)` from the index, mutable segments with background maintenance, parallel `CREATE INDEX`,
  generic WAL and replica support.