# pgColumnar roadmap

Status of the forward-looking work items and what remains. Every item preserves
the clean-room discipline in [../PROVENANCE.md](../PROVENANCE.md) and must land
with differential coverage against the heap oracle and pass the PostgreSQL 15-19
matrix. Gap specifications are in [gaps/](gaps/).

## Done

| Item | Where |
| --- | --- |
| Format 2.1 encodings (RLE, FOR, delta, delta-of-delta, Gorilla, dictionary) | I1-I8 |
| Vectorized scan, filter, and aggregate; late materialization | I3-I6 |
| Bloom filters, including collatable/text columns | gap 25 |
| Parallel scan | gap 23 |
| Covering `count(*)` from metadata | gap 28 (slice) |
| Sorted single-projection (`pgcolumnar.vacuum_sorted`) | gap 26 piece 1 |
| Arrow IPC export (`pgcolumnar.export_arrow`) | gap 27 |
| Parquet export (`pgcolumnar.export_parquet`) | gap 27 |
| Read stream / AIO in the scan (`pgcolumnar.enable_read_stream`) | gap 29 |
| Corrupt-input decode/reader hardening | SECURITY_AUDIT.md |
| Arrow/Parquet export type coverage (date/time/timestamp/uuid/numeric/json) | gaps/27-IMPL-export-type-coverage.md |
| Arrow IPC import (`pgcolumnar.import_arrow`) | gap 27 |
| PG18/19 coverage: generated columns, temporal constraints. REPACK investigated, and the conclusion was wrong: it does not work (#399) | POSTGRESQL_VERSION_ADOPTION.md |
| Full index-only scan (visibility-map fork, lazy vacuum, default on) | gap 28 |
| Multiple projections (C-Store): catalog, write fan-out, planner scan, vacuum, back-fill | gap 26 piece 2 |
| Arrow/Parquet nested export (arrays → List, composite → Struct/group) | gap 27 |
| Parquet import (`pgcolumnar.import_parquet`): Snappy, dictionary, data page v1/v2 | gap 27 |
| Arrow nested import (List → array, Struct → composite) | gap 27 |
| Parquet nested import (LIST → array, group → composite; Dremel level assembly) | gap 27 |
| **Gap 27 complete** — Arrow/Parquet interop: export + import, flat + nested, both formats | gap 27 |
| Read external Parquet in place: `read_parquet`, `parquet_schema`, `pgcolumnar_parquet` FDW | Phase G |
| Parquet FDW predicate pushdown (row-group skipping) and column projection pushdown | Phase G |
| Parquet read codecs: GZIP, ZSTD, LZ4_RAW (added to uncompressed and Snappy) | Phase G |
| Parquet read type coverage: uuid, numeric via DECIMAL, fixed binary, ms/us/ns time units | Phase G |
| Multi-file reads: a directory of `*.parquet` or a glob read as one relation | Phase G |
| Reader hardening against crafted files (scale/size/chunk-count guards, NaN and inverted stats) | Phase G |
| Streaming Parquet reads: footer plus one page at a time, no whole-file load, no size ceiling | Phase G |
| DECIMAL read from INT32/INT64 as well as fixed and variable byte arrays | Phase G |
| Recursive directory walk, depth-bounded, not following directory symlinks | Phase G |
| Hive-style partition pruning on the FDW: declared partition_columns, whole-file pruning | Phase G |
| **Phase G read surface complete** — external Parquet read, pushdown, multi-file, streaming, all matrix-gated | Phase G |

## Remaining

**Object storage, read and write.** Parquet and Iceberg data normally live on S3, GCS
or ADLS, and we read local files only. This is a prerequisite for #388 rather than a
parallel feature. Reads are #393, writes are #394. It was previously recorded only in
`PHASE_G_EXTERNAL_PARQUET_PLAN.md` under a heading reading "open decisions", which is
why nobody found it.

Ordered by value-to-effort. **Gap 27 (Arrow/Parquet interop) is fully complete**:
export and import, flat and nested, for both Arrow and Parquet, all self-contained
(no libarrow/libparquet dependency) and matrix-gated. See
[gaps/27-arrow-parquet-interop.md](gaps/27-arrow-parquet-interop.md).

The concrete remaining list was complete as of 2026-07-23, and a July 2026
external audit plus the follow-on work has since added and closed another round.
Closed since: `ANALYZE` collecting no column statistics (#154), fetch by row
number being quadratic within a row group (#143), the metadata aggregate losing
its fast path to a single deleted row (#149), imports not maintaining indexes or
enforcing unique and exclusion constraints (#153), and a wide table falling off
the fetch cache into per-row group decode (#157). The audit record is
[EXTERNAL_AUDIT_2026_07.md](EXTERNAL_AUDIT_2026_07.md).

Closed since that list was written, on 2026-07-31 and 2026-08-01: bulk load throughput (#155),
deferrable unique constraints on the import path (#168), the `ANALYZE` cost and
point-lookup plan regression (#171), sorted layouts decaying with nothing to measure it (#301), the online
recluster not recording its ordered extent (#311), bloom filters read for every
candidate group and every column (#314), the decode path having no
interrupt-correctness gate that could fail (#254), and the untested column cache,
which was resolved by the file being deleted (#282), the dead column cache (#303), the per-column bloom read (#314), and the C23 standard flag spelling (#294).

**Open, in the order they are worth taking.**

This list deliberately carries no measurements. Four successive rewrites of this
file were each wrong because they restated numbers that live in the issues, and a
restated number drifts the moment the issue moves. Each entry says what the work
is and where the current numbers are; follow the link for figures.

1. **Full-scan aggregate performance** (#289).
   The read-path decode dispatch was inlined in #307 (merged), and the grouped
   single-pass aggregate landed in #321, behind
   `pgcolumnar.enable_group_vectorization` (default off).
   Before planning from #321's numbers, read its body: by its own account the
   grouped node is a foundation, and the larger lever for that query shape is
   dictionary-coded grouping on the high-cardinality text key. The shapes where
   pgColumnar is furthest behind, and the two where it is slower than heap, are
   q6 and q8 in #289's table; nothing in flight addresses either.
2. **Code comment audit** (#291) to ISO 24495-1:2023, the plain-language
   standard the project's documentation follows. The documentation half landed
   in #298, which added `test/docs_style.sh` to the matrix as a durable gate.
   The code comments remain, and no gate covers them: most of ISO 24495-1 is not
   mechanical (whether a reader got what they needed, whether they could act on
   it), so any claim of conformity there is unverifiable by construction.

Bulk ingest is done and #300 is closed. `pgcolumnar.parallel_copy` fans core COPY,
unchanged, across background workers: partition-parallel in #323, single-table in
#324, both atomic via two-phase commit. The parser bypass the issue was named for
was rejected on its own measurement, since parse is a minority of the load. Read
the #300 thread for the profile and the two write-path findings it produced.

Selective-scan page reads are done and #310 is closed. A query skipping to 1 of
667 chunk groups used to fault about 85 percent of the table's pages; it now
touches 3.5 percent. Both causes were bloom filters being read when nothing would
consult them: for groups a scan skips (#315) and for columns no predicate probes
(#317). Confirmed on the real 100M TSBS-cpu dataset with the same on-disk table
and only the library swapped.

Formerly deferred, now built and on main. All three items this paragraph used to
list as "not yet built" are built: end-truncation ships as
`pgcolumnar.truncate(regclass)` behind `pgcolumnar.enable_end_truncation`, reclaim
free-list splitting and coalescing landed via #90, and the F1 delete-vector
catalog rename is done, which the schema shows directly:
`pgcolumnar.delete_vector` is the catalog's name today. The previous revision of
this paragraph corrected the first two and still called the third outstanding.

## Future directions

Candidate directions beyond the item above, drawn from a survey of published
columnar-engine techniques (deep-research pass, 2026-07-21). Each notes rough
effort and a primary citation. The speedup figures are self-reported by each
system's authors on their own hardware and workloads; they indicate the value of
a technique, not a guaranteed pgColumnar gain. Anything adopted still lands with
differential coverage and the PostgreSQL 15-19 matrix, and clean-room provenance
is preserved.

Already implemented on the native engine (this list predates that work; kept for
provenance, but do not treat these as open): ALP for floats and FSST for strings
(Phase E, including the chunk-shared FSST table); Z-order multi-dimensional
clustering (Phase F2/F3c, eager and online; Hilbert is still open); and richer
zone maps / SMA, i.e. per-chunk sum, value_count, and null_count are stored and
sum/count aggregates are answered from metadata when a group has no deletes
(falling back to a scan when it does, see `columnar_vector.c`). The genuinely open
directions below are the large ones: morsel-driven parallelism, data-centric JIT,
join/aggregate acceleration, delete-vector merge-on-read for MERGE, per-tier block
compression defaults, and the FastLanes on-disk format generation.

### Execution

- Adaptive, sample-based cascade encoding selection. Move from one fixed encoding
  per column to a recursive cascade (encode the output of one lightweight scheme
  with another) chosen per block by sampling a small fraction of tuples. pgColumnar
  already has the primitives (RLE, FOR, delta, dictionary); the missing piece is
  the sampling selector. High value, low-to-medium effort.
  Source: BtrBlocks, SIGMOD 2023, https://dl.acm.org/doi/10.1145/3589263 .
- Morsel-driven parallelism. Schedule small fixed-size row fragments to a worker
  pool running whole pipelines, so the degree of parallelism is elastic at runtime
  rather than fixed in the plan. The morsel unit maps onto the existing per-chunk
  parallel scan, but PostgreSQL's parallel-worker model is process-based and
  plan-fixed, so this is a large effort.
  Source: Leis et al., SIGMOD 2014, https://dl.acm.org/doi/10.1145/2588555.2610507 .
- Data-centric JIT with adaptive execution. Compile hot pipelines to machine code
  (push-based produce/consume, tuples in registers), and start each query in an
  interpreter, switching per-morsel to compiled code from runtime feedback to
  avoid the compile latency that penalizes short queries. Large effort; PostgreSQL
  already ships LLVM JIT infrastructure to build on.
  Sources: Neumann, VLDB 2011, https://www.vldb.org/pvldb/vol4/p539-neumann.pdf ;
  Kohn/Leis/Neumann, ICDE 2018, https://db.in.tum.de/~leis/papers/adaptiveexecution.pdf .
- Join and aggregate acceleration (open). Bloom or runtime join filters with
  sideways information passing, or hash-join and hash-GROUP-BY pushdown into the
  columnar scan. The research pass found no surviving primary source scoped to a
  table access method, so this needs its own investigation before a spec.

### Storage and data skipping

- Richer zone maps (Small Materialized Aggregates). Extend the per-chunk minimum
  and maximum to also carry sum, count, and null count, so aggregates can be
  answered from metadata and pruning improves on low-selectivity scans where
  indexes do not help. Low-to-medium effort, on top of the existing zone-map
  catalog. Sources: Moerkotte, VLDB 1998, https://vldb.org/conf/1998/p476.pdf ;
  Databricks data skipping, https://docs.databricks.com/aws/en/tables/data-skipping .
- Multi-dimensional clustering. Order rows by a space-filling curve (Z-order, and
  preferably Hilbert) so existing data skipping improves across several columns at
  once, with incremental background reclustering rather than one-time sorting.
  Medium effort, building on `vacuum_sorted`. Sources: Databricks (above); Delta
  Lake 3.1 Liquid Clustering, https://delta.io/blog/delta-lake-3-1/ .
- Delete vectors and merge-on-read. Mark deleted and updated rows in a side
  structure and reconcile at read time, with background compaction to reclaim,
  reducing write amplification and underpinning an efficient `MERGE`. Builds on
  the existing visibility-map fork and row mask. Medium effort.
  Source: Delta Lake 3.1, https://delta.io/blog/delta-lake-3-1/ .

### Compression and layout

- ALP for floats and decimals, and FSST for strings. ALP encodes doubles that
  originated as decimals losslessly as integers and vector-compresses genuinely
  real values, decoding faster than Gorilla and Zstd; FSST compresses short
  strings while keeping random access. Both are per-column codec upgrades. Low
  effort. Sources: ALP, SIGMOD 2024, https://duckdb.org/science/alp ; FSST is used
  by BtrBlocks and FastLanes (below).
- Reconsider default block compression. On fast local NVMe, general-purpose block
  compression (pglz, lz4, zstd) can cost more in CPU than it saves in I/O; make it
  opt-in per storage tier, and apply dictionary encoding aggressively, including on
  float columns. This finding is scoped to fast local storage and reverses for
  high-latency or remote (object-store) storage, so keep block compression the
  default there. Low effort (defaults and per-table options).
  Source: Zeng et al., VLDB 2024, https://www.vldb.org/pvldb/vol17/p148-zeng.pdf .
- FastLanes-style expression encoding. For a future on-disk format generation,
  cascade lightweight encodings over fixed 1024-value vectors with multi-column
  compression and partial bottom-up decode, so the executor receives compressed
  vectors and runs directly on them. Large effort (a new format generation) that
  targets the run-at-a-time compressed executor pgColumnar already has.
  Source: FastLanes, PVLDB vol.18, 2025, https://www.vldb.org/pvldb/vol18/p4629-afroozeh.pdf .
- Asynchronous write and background compaction (wishlist). Commit inserts in a
  fast, lightly encoded or uncompressed write-optimized form and return to the
  transaction immediately, then have a background worker rewrite the row groups
  with the full encoding cascade afterward. This hides encoder latency from the
  foreground for the heaviest encoders at scale. The classic write-optimized to
  read-optimized store split is the prior art. It must be toggleable on or off per
  table or per storage tier, since it trades foreground latency for write
  amplification, transient extra space, and slower interim reads until compaction.
  It shares the MVCC-safe row-group rewrite machinery with the mutation and
  clustering work, so it belongs with that phase. Gated on a measurement showing
  synchronous encoding cost is still a problem after the per-chunk shared FSST
  table (E3b) lands. Large effort.

### Interoperability and PostgreSQL integration

The research pass returned few surviving primary sources in this area, so these
are directions to investigate and spec, not validated recommendations:

- External Parquet read with predicate and projection pushdown is done (Phase G,
  see above). What remains here: ORC, open table formats (Apache Iceberg, Delta
  Lake, Hudi). Within Parquet the remaining item is partition inference; streaming
  reads, INT32/INT64-backed
  DECIMAL reads, recursive directory walks, and Hive-style partition pruning are
  done (see Done above).
- Arrow C Data Interface zero-copy export, and Arrow Flight SQL or ADBC access.
- New PostgreSQL 17-19 integration points: read stream and asynchronous IO (partly
  used), `MERGE`, incremental materialized views (pg_ivm), logical decoding of
  columnar changes, optimizer-statistics injection, and TOAST or large-value
  handling.

### Open questions to resolve before starting

- Which join-acceleration technique returns the most inside a table access method,
  and how does it interact with the planner and executor hooks.
- The concrete design for external file and open-table-format access (native
  reader versus foreign data wrapper) and whether it reuses the existing pruning
  metadata.
- Which PostgreSQL 17-19 APIs are the highest-leverage integration points.
- Whether the next format is a full FastLanes-style vector rewrite (larger) or an
  incremental BtrBlocks-style cascade selector on the current format 2.2 encodings
  (lower), given the run-at-a-time compressed executor.

## PostgreSQL 18/19 adoption

Features new in PostgreSQL 17-19 that pgColumnar can use, all version-gated to
preserve the 15-19 matrix. Detail and sources in
[POSTGRESQL_VERSION_ADOPTION.md](POSTGRESQL_VERSION_ADOPTION.md):

- Read stream / AIO in the scan — shipped, see the Done table.
- Virtual generated columns (PostgreSQL 18) — done, covered by
  `test/generated_columns.sh`. Stored and virtual columns both read correctly on a
  columnar table, and a virtual column has no chunk written for it at all
  (`columnar_write_state.c` skips `attgenerated == 'v'`), which the same suite pins.
- Temporal constraints (`WITHOUT OVERLAPS` in 18, `FOR PORTION OF` in 19) — done,
  covered by `test/temporal.sh` against the heap oracle, `FOR PORTION OF` gated to 19.
- REPACK (PostgreSQL 19) — investigated, and the answer was no. It does not work on a
  columnar table, and the error now says so (#399, #409). Not a remaining item.
- Optimizer statistics injection (PostgreSQL 18) and a btree skip-scan benchmark
  line: smaller follow-ups, still open with no measurement.

## Test-harness follow-up

The suites are bash driving a differential oracle (heap mirror vs columnar,
compared by order-independent result hash) plus C property tests for the codecs.
One structural improvement is worthwhile when the harness is next touched: assert
EXPLAIN output from `FORMAT JSON` fields rather than text grep, and make a failed
step report which assertion failed instead of aborting under `set -e`. A wider
move to a Python/pyunit differential harness is possible later but is not
required; it should be specced before starting.