name: ci # Every pull request runs the fast tier: lint, and one PostgreSQL 18 job on x86 with the regression tests its diff can break (scripts/areas.tsv, `scripts/ci.sh affected`) # and the tests it touches, a few minutes in all. A pull request from an `integration/` branch, or one labeled `ci-full` (one at a time: the Actions minutes of a private # repository are finite), adds the fast tier on PostgreSQL 17, 19 and AArch64, macOS, and the SQLite, MariaDB and replication jobs its diff can break. A push to main or to an # `integration/` branch runs everything the diff can break, PostgreSQL 17, 18 and 19, the TAP, differential and # release-library stages and the other jobs; its base is the last green run of that branch, so a red or cancelled push is not forgotten by the next one. # The nightly run and a dispatch (of main, or of a branch before the merge of a risky change) run every job of the quick tier ("Full scope" in the first job is # the mark the release tool looks for); the nightly run, and a dispatch with heavy set, add the full tier (sanitizers, long soaks and fuzz runs). `full_pg` (default `17 18`) narrows the full tier to one major, so its 4 shards run as a dispatch of their own (the longest, on 18, took 55 minutes); `versions` still filters every job, so dispatch with `versions` and `full_pg` both set to that major. Only the full tier installs pg_repack (PGDG ships 1.5.3 for 17, 18 and 19), so test/t/col_pg_repack.sh runs there and skips elsewhere. Every PostgreSQL 18 job installs pg_squeeze (PGDG ships 1.9.4, a few seconds), so test/t/col_pg_squeeze.sh runs in the shard that holds it and skips on 17 and 19. # PostgreSQL 19 (a server built from the git tree, cached: .github/actions/pg19) runs whatever PostgreSQL 17 and 18 run, in the same shards, and fails the run like them. # The full tier is six shards on each of PostgreSQL 18 and 17 (five took 60 minutes or more on two of them, hosted run 42800648290) (the deep one nine and five, cut by the weights of scripts/weights-deep.tsv that scripts/ci-times.py writes from a run, the full entries first so that the account's 20 concurrent jobs go to the long ones; the jobs that install a library flavor end the last shard of 17), about 100 lane minutes a shard. # A deep run does not repeat the x86 quick shards on 17 and 18: its full tier has the same jobs at full scale; the quick shards of AArch64 and 19 stay, with the release-library list on 18. # Linux on AArch64 (`arm: true` entries, a hosted ubuntu-24.04-arm runner) runs the PostgreSQL 18 entries again: the fast tier of a `ci-full` pull request, the quick tier # of a push, the nightly run and a dispatch. # The memory-debugging lanes (assert, valgrind) are assert.yml, weekly. # A pull request of any kind runs one more job, PostgreSQL 18 with assertions (`cassert: true`), on the regression tests that run workers (`ASSERT_JOBS` of scripts/ci/affected.sh), when it # changes a source that starts workers or handles snapshots (`ASSERT_MARK`) or carries the label `assertions`; the push, nightly and dispatch cassert entry runs the family too. # No scheduled run is enabled in this repository: dispatch the workflow to run it. on: push: branches: [main, 'integration/**'] pull_request: types: [opened, reopened, synchronize, labeled] workflow_dispatch: inputs: heavy: description: Also run the full tier (make check-full) type: boolean default: false deep: description: The full tier at its long sizes (the nightly run has them); without it a job of the full tier fits 45 minutes and the run 75 type: boolean default: false versions: description: 'The PostgreSQL majors of the quick tier, e.g. 18 for a branch with a large diff' default: '17 18 19' full_pg: description: 'The PostgreSQL majors of the full tier (heavy), e.g. 18 for one run of its own; `versions` still applies to every job, so the major must be in both' default: '17 18' permissions: contents: read actions: read # a pull request run is cancelled by the next push to the same pull request. A push to main or to an integration branch never cancels the run of the one before it: that run finishes, and of the # runs waiting behind it GitHub keeps the newest, which scopes its diff from the last green run (see `changes`), so every batch of merges gets one complete run and # the head of main is tested. A nightly or dispatch run has a group of its own, so a push never cancels it (main-verify.yml dispatches one when the head of main has none). # A dispatch with heavy has a group of its own per run, so it never waits behind another run of the same ref (a long deep run held a lean one pending for six minutes); two # heavy dispatches of one ref can therefore run together, so check `gh run list` before dispatching. No dispatch is cancelled by another. concurrency: group: ci-${{ github.event_name }}-${{ github.ref }}${{ github.event.action == 'labeled' && github.event.label.name != 'ci-full' && format('-label-{0}', github.run_id) || github.event_name == 'workflow_dispatch' && inputs.heavy && format('-run-{0}', github.run_id) || '' }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} # `changes` and `lint` start the work and carry the condition; the other jobs skip through `needs`. Any label other than `ci-full` starts a run of its own that skips # everything, in a group of its own, so it does not cancel the run of the pull request. jobs: changes: if: ${{ github.event_name != 'pull_request' || github.event.action != 'labeled' || github.event.label.name == 'ci-full' || github.event.label.name == 'assertions' }} runs-on: ubuntu-24.04 timeout-minutes: 5 outputs: jobs: ${{ steps.scope.outputs.jobs }} fast: ${{ steps.scope.outputs.fast }} fastjobs: ${{ steps.scope.outputs.fastjobs }} sqlite: ${{ steps.scope.outputs.sqlite }} mariadb: ${{ steps.scope.outputs.mariadb }} replication: ${{ steps.scope.outputs.replication }} macos: ${{ steps.scope.outputs.macos }} matrix: ${{ steps.matrix.outputs.matrix }} steps: - uses: actions/checkout@v6 with: fetch-depth: 0 - id: scope env: GH_TOKEN: ${{ github.token }} CIPR: ${{ startsWith(github.head_ref, 'integration/') || contains(github.event.pull_request.labels.*.name, 'ci-full') }} run: | base=all case "$GITHUB_EVENT_NAME" in pull_request) base=origin/$GITHUB_BASE_REF ;; push) # the newest green run of main whose commit is an ancestor of this one: what it checked is not checked again for sha in $(gh run list --workflow ci.yml --branch "$GITHUB_REF_NAME" --status success --limit 20 --json headSha --jq '.[].headSha'); do git merge-base --is-ancestor "$sha" HEAD 2>/dev/null && { base=$sha; break; } done ;; esac echo "base: $base" scripts/ci.sh affected --list "$base" | tee -a "$GITHUB_STEP_SUMMARY" # a pull request that is neither from an integration branch nor labeled `ci-full` runs the fast tier on x86 and the assertions family (below), not the other platforms if [ "$GITHUB_EVENT_NAME" = pull_request ] && [ "$CIPR" != true ]; then scripts/ci.sh affected --github "$base" | grep -vE '^(sqlite|mariadb|replication|macos)=' | tee -a "$GITHUB_OUTPUT" else scripts/ci.sh affected --github "$base" | tee -a "$GITHUB_OUTPUT" fi # the mark of a run that covered everything (the release tool reads it from the API) - name: Full scope if: ${{ steps.scope.outputs.full == 'true' }} run: echo "every job of the quick tier ran" - id: matrix env: NSHARDS: ${{ steps.scope.outputs.nshards }} PG: ${{ steps.scope.outputs.pg }} JOBS: ${{ steps.scope.outputs.jobs }} FAST: ${{ steps.scope.outputs.fast }} FASTJOBS: ${{ steps.scope.outputs.fastjobs }} FASTPG: ${{ steps.scope.outputs.fastpg }} VERSIONS: ${{ inputs.versions }} FULL_PG: ${{ inputs.full_pg }} EVENT: ${{ github.event_name }} CIPR: ${{ startsWith(github.head_ref, 'integration/') || contains(github.event.pull_request.labels.*.name, 'ci-full') }} CASSERT: ${{ steps.scope.outputs.cassert }} CASSERTJOBS: ${{ steps.scope.outputs.cassertjobs }} LABEL_ASSERTIONS: ${{ contains(github.event.pull_request.labels.*.name, 'assertions') }} HEAVY: ${{ github.event_name == 'schedule' || inputs.heavy == true }} DEEP: ${{ github.event_name == 'schedule' || inputs.deep == true }} run: | # a pull request: one PostgreSQL 18 job, the regression tests of the areas the diff touches (`jobs`, narrower than the push's: scripts/areas.tsv `fast:` rows) and the tests it touches (`only`, SIFT_CI_JOBS) # PostgreSQL 17, 18 and 19 (assertions make its shards the slowest: 22 to 30 minutes with three, hosted run 42800632926) have a shard more: the jobs of 17 that install a library flavor run one after the other, apart from the pool (scripts/ci.sh), and the 4 cores of a hosted x86 runner # take 16 to 18 minutes for a third of the 18 quick tier (hosted run 42152410447; an arm64 runner 11) set -o pipefail # a jq error must fail the step: an empty matrix skips every test job and the gate passes d=; [ "$DEEP" = true ] && d=1 jq -nc --argjson full17 "$(scripts/ci.sh --shards 17 $d)" --argjson full18 "$(scripts/ci.sh --shards 18 $d)" --argjson n "${NSHARDS:-1}" --arg cipr "$CIPR" --arg cassert "$CASSERT" --arg cassertjobs "$CASSERTJOBS" --arg label "$LABEL_ASSERTIONS" --argjson heavy "$HEAVY" --argjson deep "$DEEP" --arg pg "$PG" --arg fastpg "$FASTPG" --arg jobs "$JOBS" --arg fast "$FAST" --arg fastjobs "$FASTJOBS" --arg event "$EVENT" --arg versions "$VERSIONS" --arg fullpg "$FULL_PG" ' if $event == "pull_request" then (if $fastpg == "true" then ([{pg: 18, arm: false}] + (if $cipr == "true" then [{pg: 18, arm: true}, {pg: 17}, {pg: 19, git: true}] else [] end)) | map({shard: 1, n: 1, tier: "fast", jobs: $fastjobs, only: $fast} + .) else [] end) + (if $cassert == "true" or $label == "true" then [{pg: 18, shard: 1, n: 1, tier: "quick", jobs: $cassertjobs, cassert: true}] else [] end) else (if $heavy then [(17, 18) as $v | select($fullpg == "" or (($fullpg | split(" ") | index($v | tostring)) != null)) | (if $v == 17 then $full17 else $full18 end) as $m | range(1; $m + 1) as $s | {pg: $v, shard: $s, n: $m, tier: "full", jobs: "."}] else [] end) + (if $pg == "true" then (if $deep then [] else [(17, 18) as $v | (if $n > 1 then $n + 1 else $n end) as $m | range(1; $m + 1) as $s | {pg: $v, shard: $s, n: $m, tier: "quick", jobs: $jobs}] end) + [(if $n > 1 then $n + 1 else $n end) as $m | range(1; $m + 1) as $s | {pg: 19, shard: $s, n: $m, tier: "quick", jobs: $jobs, git: true}] + [range(1; $n + 1) as $s | {pg: 18, shard: $s, n: $n, tier: "quick", jobs: $jobs, arm: true}] + [range(1; 3) as $s | {pg: 18, shard: $s, n: 2, tier: "quick", jobs: ("^installcheck-agg_(cte_|.*_cost$|.*_scope$|.*lateral|.*subplan|plan_|nested_ctes|.*placeholder|statement_cte)|" + $cassertjobs), cassert: true}] else [] end) end | map(select($versions == "" or ((.pg | tostring) as $p | ($versions | split(" ") | index($p)) != null)))' | sed 's/^/matrix=/' | tee -a "$GITHUB_OUTPUT" # no PostgreSQL: the syntax of every script, the portable core, the map of areas against the tests, the workflows lint: if: ${{ github.event_name != 'pull_request' || github.event.action != 'labeled' || github.event.label.name == 'ci-full' }} runs-on: ubuntu-24.04 timeout-minutes: 10 steps: - uses: actions/checkout@v6 # the python3 of macOS is 3.9: the scripts and generators lint runs must work on it - uses: actions/setup-python@v6 with: python-version: '3.9' - run: scripts/lint.sh - name: actionlint run: bash <(curl -fsSL https://raw.githubusercontent.com/rhysd/actionlint/v1.7.7/scripts/download-actionlint.bash) 1.7.7 && ./actionlint postgresql: name: PostgreSQL ${{ matrix.pg }}${{ matrix.arm && ' arm64' || '' }}${{ matrix.tier == 'full' && ' full tier' || matrix.tier == 'fast' && ' (fast)' || '' }}${{ matrix.n > 1 && format(' ({0}/{1})', matrix.shard, matrix.n) || '' }}${{ matrix.git && ' (git)' || '' }}${{ matrix.cassert && ' (assertions)' || '' }} needs: changes if: ${{ needs.changes.outputs.matrix != '[]' }} runs-on: ${{ matrix.arm && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }} timeout-minutes: ${{ matrix.tier == 'full' && (github.event_name == 'schedule' || inputs.deep == true) && 120 || matrix.tier == 'full' && 60 || 30 }} strategy: fail-fast: false matrix: include: ${{ fromJSON(needs.changes.outputs.matrix) }} steps: - uses: actions/checkout@v6 with: # The compatibility tests build released libraries from their tags. fetch-depth: 0 - uses: ./.github/actions/apt-limits # one cache for each major and flavour of the entry, renewed every week (a package that left the mirror is no longer asked for); the first job of the entry to finish saves it # the packages apt downloads (scripts/apt-install.sh) are kept there: a mirror that stalls holds an install for minutes, and the server with its development files is 90 MB of it - id: week run: | echo "week=$(date +%G-%V)" >> "$GITHUB_OUTPUT" echo "APT_ARCHIVES=$HOME/apt-debs" >> "$GITHUB_ENV" - name: Cache the apt packages uses: actions/cache@v5 with: path: ~/apt-debs key: apt-pg${{ matrix.pg }}${{ matrix.git && '-git' || '' }}${{ matrix.cassert && '-assert' || '' }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.week.outputs.week }} restore-keys: | apt-pg${{ matrix.pg }}${{ matrix.git && '-git' || '' }}${{ matrix.cassert && '-assert' || '' }}-${{ runner.os }}-${{ runner.arch }}- # PGDG also serves the previous major of the pg_upgrade test (below) to the entry that builds PostgreSQL 19 itself - name: PGDG repository timeout-minutes: 5 run: | scripts/pgdg-repo.sh - name: Install PostgreSQL ${{ matrix.pg }} if: ${{ !matrix.git && !matrix.cassert }} timeout-minutes: 10 run: | scripts/apt-install.sh postgresql-${{ matrix.pg }} postgresql-server-dev-${{ matrix.pg }} libipc-run-perl liblz4-dev libzstd-dev libicu-dev ccache ${{ matrix.tier == 'full' && format('postgresql-{0}-repack', matrix.pg) || '' }} ${{ matrix.pg == 18 && 'postgresql-18-squeeze' || '' }} echo /usr/lib/postgresql/${{ matrix.pg }}/bin >> "$GITHUB_PATH" echo /usr/lib/ccache >> "$GITHUB_PATH" # the compiler cache of the build and of the 8-page flavour of the scaling gate (58 s cold, 3 s warm). The key is the compiler and the sources the objects come from, so a commit that # changes none of them (tests, docs, workflows) finds its cache and saves nothing new, and any other restores the newest cache of the branch or of main (caches of the default branch # are read by every branch) and saves its own. The first job of the entry to finish saves it, a failed run included: the objects are the cost, not the tests - name: Key of the compiler cache id: ccache if: ${{ !matrix.git && !matrix.cassert }} run: | prefix="ccache-pg${{ matrix.pg }}-$RUNNER_OS-$RUNNER_ARCH-gcc$(gcc -dumpfullversion)-" echo "prefix=$prefix" >> "$GITHUB_OUTPUT" echo "key=${prefix}${{ hashFiles('src/**', 'Makefile', 'scripts/harden.mk') }}" >> "$GITHUB_OUTPUT" - name: Cache the compiler id: ccache-cache if: ${{ !matrix.git && !matrix.cassert }} uses: actions/cache/restore@v5 with: path: ~/.cache/ccache key: ${{ steps.ccache.outputs.key }} restore-keys: | ${{ steps.ccache.outputs.prefix }} ccache-pg${{ matrix.pg }}-${{ runner.os }}-${{ runner.arch }}- # the cached build of the newest commit of the branch (the nightly and dispatch runs build the head of it when the cache lacks it) - name: Get PostgreSQL 19 if: ${{ matrix.git }} uses: ./.github/actions/pg19 with: exact: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} # the planner asserts of the server (a foreach_node cast of the wrong node type is an Assert) exist only in a build with --enable-cassert, which the # distribution packages lack: the aggregate planner tests run on a PostgreSQL 18 built from source with it (scripts/pg-assert.sh, cached with the build of assert.yml) - name: Cache PostgreSQL ${{ matrix.pg }} with assertions if: ${{ matrix.cassert }} uses: actions/cache@v5 with: path: ~/.cache/sift/pg key: pg-assert-${{ matrix.pg }}-${{ hashFiles('scripts/pg-assert.sh') }} - name: Build PostgreSQL ${{ matrix.pg }} with assertions if: ${{ matrix.cassert }} run: | scripts/apt-install.sh bison flex libipc-run-perl liblz4-dev libzstd-dev prefix=$(PG_ASSERT_MAJOR=${{ matrix.pg }} scripts/pg-assert.sh) echo "$prefix/bin" >> "$GITHUB_PATH" - name: Use PostgreSQL 19 if: ${{ matrix.git }} timeout-minutes: 10 run: | scripts/apt-install.sh libipc-run-perl liblz4-dev libzstd-dev libicu-dev libxml2 echo "$HOME/pg19/bin" >> "$GITHUB_PATH" # test/t/pg_upgrade.sh upgrades from the major before the server's when its binaries are in /usr/lib/postgresql/N-1 (17 to 18, 18 to 19) and builds # pg_sift for it, so it needs the server and its headers. Only the entries whose share of the queue has the job install them; a major PGDG lacks # is a warning, and the test then upgrades within one major - name: Install the previous PostgreSQL major (pg_upgrade) if: ${{ matrix.pg > 17 }} timeout-minutes: 10 env: SIFT_CI_AFFECTED: ${{ matrix.jobs }} SIFT_CI_JOBS: ${{ matrix.only }} SIFT_CI_SHARD: ${{ matrix.shard }}/${{ matrix.n }} run: | plan=$(scripts/ci.sh --plan ${{ matrix.tier == 'full' && 'full' || 'quick' }}) grep -q '^tap-shell-pg_upgrade ' <<<"$plan" || { echo "no pg_upgrade job in this entry"; exit 0; } prev=$(( ${{ matrix.pg }} - 1 )) scripts/apt-install.sh postgresql-$prev postgresql-server-dev-$prev || echo "::warning::PostgreSQL $prev is not available from PGDG: pg_upgrade runs within one major" # PostgreSQL 17 has no extension_control_path: the tests install sift into the server's own directories - name: Install into the server (PostgreSQL 17) if: ${{ matrix.pg == 17 }} run: echo "SIFT_SUDO=sudo env PATH=$PATH" >> "$GITHUB_ENV" # the older releases the compatibility tests build (tags do not move; about 70 CPU seconds each, seven of them); PostgreSQL 17 cannot hold two builds, and the # git build of the server changes with every run. A root is by commit (scripts/old-lib-check.sh), so what an earlier cache holds stays valid: the key follows # the compiler and the tags (a release adds a library), the tests that build them change it too, and the shard of the same key prefix restores the # last cache and builds only what it lacks. The cache is saved after a failed run as well: the builds are the cost, not the tests - name: Key of the older releases id: oldlib if: ${{ matrix.pg != 17 && !matrix.git && !matrix.cassert }} run: | prefix="old-lib-$RUNNER_OS-$RUNNER_ARCH-pg${{ matrix.pg }}-gcc$(gcc -dumpfullversion)-shard${{ matrix.shard }}-" echo "prefix=$prefix" >> "$GITHUB_OUTPUT" echo "key=$prefix$(git tag --list 'v*' | sha256sum | cut -c1-12)-${{ hashFiles('scripts/old-lib-check.sh', 'test/t/compat_*.sh', 'test/t/old_lib_refusal.sh', 'test/t/pg_upgrade.sh') }}" >> "$GITHUB_OUTPUT" - name: Cache the older releases id: oldlib-cache if: ${{ matrix.pg != 17 && !matrix.git && !matrix.cassert }} uses: actions/cache/restore@v5 with: path: ~/.cache/sift/old-lib/*/root key: ${{ steps.oldlib.outputs.key }} restore-keys: ${{ steps.oldlib.outputs.prefix }} # the last stage of make check is the leak gate (scripts/leakcheck.sh): leak warnings, crashes and sanitizer # reports in the server logs of every suite, and the cluster checks recorded at each shutdown # includes the regression tests against the release library (SIFT_TESTING off, stage release-library-regress), with its own leak gate # the jobs: the ones scripts/areas.tsv picks for the diff (all of them in the nightly entries), and this runner's share of them (shard k of n) # the checks of the harness (lint's scripts, the selftests) run in `lint`, once for every run, not in a shard - name: ${{ matrix.tier == 'full' && 'Full test tier (make check-full)' || 'Quick test tier (make check, with the leak gate)' }} run: make ${{ matrix.tier == 'full' && 'check-full' || 'check' }} env: SIFT_CI_TMP: ${{ runner.temp }}/sift-ci SIFT_CI_AFFECTED: ${{ matrix.jobs }} SIFT_CI_JOBS: ${{ matrix.only }} SIFT_CI_NO_HARNESS: 1 SIFT_CI_STATS: ${{ matrix.tier == 'quick' && '1' || '' }} SIFT_CI_ASSERT: ${{ matrix.cassert && '1' || '' }} SIFT_CI_SHARD: ${{ matrix.shard }}/${{ matrix.n }} SIFT_CI_DEEP: ${{ (github.event_name == 'schedule' || inputs.deep == true) && '1' || '' }} # a job that hangs is stopped with the end of its log after 30 minutes (15 in the quick tier, scripts/ci/config.sh): the limit of the workflow ends the whole shard, and its logs are lost CI_JOB_TIMEOUT: ${{ matrix.tier == 'full' && !(github.event_name == 'schedule' || inputs.deep == true) && '1800' || '' }} # the first hosted run says whether the compiler wrappers took: a cache of 0 hits on a warm key means the build did not go through them - name: Compiler cache statistics if: ${{ !cancelled() && !matrix.git && !matrix.cassert }} run: ccache -s - name: Keep the compiler cache if: ${{ !cancelled() && steps.ccache-cache.outcome == 'success' && steps.ccache-cache.outputs.cache-hit != 'true' }} uses: actions/cache/save@v5 with: path: ~/.cache/ccache key: ${{ steps.ccache.outputs.key }} - name: Keep the older releases if: ${{ !cancelled() && steps.oldlib-cache.outcome == 'success' && steps.oldlib-cache.outputs.cache-hit != 'true' }} uses: actions/cache/save@v5 with: path: ~/.cache/sift/old-lib/*/root key: ${{ steps.oldlib.outputs.key }} - name: Keep the logs of a failed run if: failure() || cancelled() uses: actions/upload-artifact@v6 with: name: pg_sift-test-logs-pg${{ matrix.pg }}${{ matrix.arm && '-arm64' || '' }}${{ matrix.cassert && '-cassert' || '' }}-${{ matrix.tier }}-${{ matrix.shard }} path: | ${{ runner.temp }}/sift-ci/logs ${{ runner.temp }}/sift-ci/lane*/**/tap/**/*.log ${{ runner.temp }}/sift-ci/lane*/**/postgres.log ${{ runner.temp }}/sift-ci/lane*/**/sanitizer ${{ runner.temp }}/sift-ci/lane*/**/leakcheck.* ${{ runner.temp }}/sift-ci/lane*/regress/*/regression.diffs test/regression.diffs if-no-files-found: ignore # one container per node (primary, standby, subscriber): separate filesystems and installs of the extension replication: name: Replication in containers (${{ matrix.image }}) needs: changes if: ${{ needs.changes.outputs.replication == 'true' }} runs-on: ubuntu-24.04 timeout-minutes: 15 strategy: fail-fast: false matrix: image: [postgres:17, postgres:18] steps: - uses: actions/checkout@v6 - run: PG_IMAGE=${{ matrix.image }} scripts/replication.sh --container # the SQLite extension (sqlite/): independent of PostgreSQL, built against the system SQLite development package sqlite: name: SQLite extension needs: changes if: ${{ needs.changes.outputs.sqlite == 'true' }} runs-on: ubuntu-24.04 timeout-minutes: 30 steps: - uses: actions/checkout@v6 - uses: ./.github/actions/apt-limits - name: Install build dependencies timeout-minutes: 10 run: | scripts/apt-install.sh --update build-essential libsqlite3-dev sqlite3 python3 sqlite3 --version # the portable-core check (the files of src/portable.list include only src/core/host.h), then: build, smoke test, baseline instruction check, 59 unit tests, # FTS5 differential on a synthetic corpus, all again with the scalar # kernels forced and for the vector-free build; then ASan+UBSan on both test files and the stress test, and TSan on the thread stress # test. The tests that need the 218k-article database or a pg_sift run are skipped, with a message, when their variables are unset. # vm.mmap_rnd_bits: the runner's address randomization is too wide for TSan's shadow memory. - name: Tests (sanitizers included) run: | sudo sysctl -w vm.mmap_rnd_bits=28 SIFT_SANITIZE=1 sqlite/test.sh # the MariaDB extension (mariadb/): the portable-core check, then the release recipe for two series: build the library in the stock # mariadb: image against the headers of its server, and load it into a server of that image (scripts/package-mariadb.sh, # mariadb/tests/smoke.sh). The full suite, mariadb/test.sh (218k-article corpus, pg_sift reference, InnoDB FULLTEXT, about 30 minutes # on a quiet machine), needs a PostgreSQL and the corpus and is run by hand before a release. mariadb: name: MariaDB extension (${{ matrix.series }}) needs: changes if: ${{ needs.changes.outputs.mariadb == 'true' }} runs-on: ubuntu-24.04 timeout-minutes: 30 strategy: fail-fast: false matrix: series: ['11.8', '13.0'] steps: - uses: actions/checkout@v6 - run: scripts/check-portable.sh - name: Build and smoke-test the library of the series run: SERIES=${{ matrix.series }} scripts/package-mariadb.sh all HEAD dist-mariadb # Homebrew on macOS arm64, the one place that is not Linux: the formula of this checkout builds and passes its own test on PostgreSQL 17 and 18, and the # test harness runs on brew's bash 5 (a few regression tests and the TAP tests that failed on the BSD tools; the stages that need GNU tools or /proc stay on Linux) while `make check` under the stock # /bin/bash 3.2 must stop with a message instead of exiting 0. It runs where scripts/areas.tsv says (gh:macos: the build, the harness and Formula/ are # `.`, so a core or harness change, the nightly run and a dispatch), never on a pull request that touches only sources or docs. Hosted macOS minutes are # free for a public repository (10x the Linux rate for a private one), hence the time limit. macos: name: macOS arm64 (Homebrew) needs: changes if: ${{ needs.changes.outputs.macos == 'true' }} runs-on: macos-15 timeout-minutes: 45 env: HOMEBREW_NO_AUTO_UPDATE: 1 HOMEBREW_NO_INSTALL_CLEANUP: 1 HOMEBREW_NO_ANALYTICS: 1 steps: # the whole history and the tags: the tests that build an older release (git archive of v3.2.x) skip without them - uses: actions/checkout@v6 with: fetch-depth: 0 - name: Install bash 5 and PostgreSQL 17 and 18 run: brew install bash postgresql@17 postgresql@18 lz4 zstd # the README install (a tap, brew install, brew test) on this checkout: the formula is copied into a tap of its own, in the runner's prefix. The stable formula is # installed from the real archive of its tag (the url and sha256 of the file are what a user gets), the HEAD one from this checkout; style and audit read the formula # as it is in the file - name: Formula (brew style, brew audit, brew install, brew test) run: | git branch -f macos-ci HEAD brew tap-new --no-git local/ci # Homebrew refuses to load a formula from a tap it was not told to trust (brew audit and brew install fail with "untrusted tap") brew trust local/ci mkdir -p "$(brew --repository local/ci)/Formula" cp Formula/pg_sift.rb "$(brew --repository local/ci)/Formula/pg_sift.rb" brew style local/ci/pg_sift brew audit --strict --online local/ci/pg_sift brew install local/ci/pg_sift brew test local/ci/pg_sift brew uninstall local/ci/pg_sift sed "s|^ head .*| head \"file://$PWD\", using: :git, branch: \"macos-ci\"|" Formula/pg_sift.rb >"$(brew --repository local/ci)/Formula/pg_sift.rb" brew install --HEAD local/ci/pg_sift brew test local/ci/pg_sift # the report this job answers: no brew bash on PATH, so `env bash` is /bin/bash 3.2 - name: make check under the stock bash 3.2 fails with a message run: | /bin/bash --version | head -1 if out=$(env PATH=/usr/bin:/bin:/usr/sbin:/sbin PG_CONFIG="$(brew --prefix postgresql@18)/bin/pg_config" make check 2>&1); then echo "$out"; echo "make check exited 0 under bash 3.2"; exit 1 fi echo "$out" grep -q 'needs bash 4+' <<<"$out" # the BSD tools and the clang of macOS: the portability reports of `make check` were these (paste, stat, wc, seq, the module suffix) - name: lint run: | PATH="$(brew --prefix)/bin:$(brew --prefix postgresql@18)/bin:$PATH"; export PATH scripts/lint.sh # the formula above leaves its pg_sift in the server's directories, as a user's install does: the upgrade tests must use their own - name: make check (regression tests, the tests that tripped on macOS) under brew bash 5 env: SIFT_CI_TMP: ${{ runner.temp }}/sift-ci SIFT_CI_JOBS: '^(installcheck-(facets|filter_pushdown)|tap-shell-(col_auto_merge_concurrent|col_refresh_crash|col_rollup_ondemand|col_wbuf_batch|leakcheck_gate|pg_upgrade|pg_upgrade_32|replica_mixed|scaling_slope))$' SIFT_CI_NO_HARNESS: 1 run: | PATH="$(brew --prefix)/bin:$(brew --prefix postgresql@18)/bin:$PATH"; export PATH bash --version | head -1 make check # PostgreSQL 17 has no extension_control_path: sift is installed into the server's own directories (the runner is throwaway). replica_mixed and # scaling_slope need PostgreSQL 18 and skip themselves, and the older releases are not built on 17 either (scripts/old-lib-check.sh) - name: make check (the same tests) on PostgreSQL 17 env: SIFT_CI_TMP: ${{ runner.temp }}/sift-ci17 SIFT_CI_JOBS: '^(installcheck-(facets|filter_pushdown)|tap-shell-(col_auto_merge_concurrent|col_refresh_crash|col_rollup_ondemand|col_wbuf_batch|leakcheck_gate|pg_upgrade|pg_upgrade_32|replica_mixed|scaling_slope))$' SIFT_CI_NO_HARNESS: 1 run: | PATH="$(brew --prefix)/bin:$(brew --prefix postgresql@17)/bin:$PATH"; export PATH make check - name: Keep the logs of a failed run if: failure() uses: actions/upload-artifact@v6 with: name: pg_sift-test-logs-macos path: | ${{ runner.temp }}/sift-ci*/logs ${{ runner.temp }}/sift-ci*/lane*/**/postgres.log ${{ runner.temp }}/sift-ci*/lane*/regress/*/regression.diffs if-no-files-found: ignore # the one name for a branch ruleset to require: the shard names of the jobs above change with the diff and the event, this one does not. # A job that did not run (skipped) passes; a failed or cancelled one fails the gate gate: needs: [changes, lint, postgresql, replication, sqlite, mariadb, macos] if: ${{ always() }} runs-on: ubuntu-24.04 timeout-minutes: 2 steps: - run: '! ${{ contains(needs.*.result, ''failure'') || contains(needs.*.result, ''cancelled'') }}'