# CI setup notes

## GitHub Actions (mirror at `gburd/pg_turbovec`)

The mirror automatically syncs from Codeberg. Workflows live at
`.github/workflows/test.yml` and run on every push to `main` plus
every PR.

If the workflow fails with `cargo-pgrx` complaining about `edition2024`,
bump the toolchain version in `setup-rust-toolchain` — `cargo-pgrx
0.17.0` requires Cargo ≥ 1.85.0 (the version that stabilised
`edition2024`).

Trigger a manual run: <https://github.com/gburd/pg_turbovec/actions>
→ "test" workflow → Run workflow.

## Codeberg Actions (canonical CI at `gregburd/pg_turbovec`)

**No application required — Codeberg's hosted Forgejo Actions is open to
all users now.** The (outdated)
<https://codeberg.org/Codeberg-CI/request-access> page predates the
current setup; ignore it. The current docs are at
<https://codeberg.org/actions/meta>.

What you DO need:

1. **Enable Actions** on the repo: Settings → Units → "Enable Actions".
   `has_actions: True` confirms it. (Already done.)

2. **Use a Codeberg-provided runner label.** Hosted runners have these
   labels with a 10-minute job-runtime cap each:

   | Label | CPU | RAM | Runtime cap |
   |---|---|---|---|
   | `codeberg-tiny`   | 1 | 2 GB | 2 min |
   | `codeberg-small`  | 2 | 4 GB | 5 min |
   | `codeberg-medium` | 4 | 8 GB | 10 min |

   Plus `*-lazy` variants for jobs that can wait. `runs-on: docker`
   (the GitHub Actions default) doesn't match any Codeberg runner;
   pushes that use it sit in `status: waiting` forever.

3. **Live with the runtime cap.** `cargo pgrx test pg<N>` jobs take
   7–15 min each (cold cache builds PG from source); they can't fit
   `codeberg-medium`. So `.forgejo/workflows/test.yml` is intentionally
   slim — it runs only `drift-check` (3 s) on `codeberg-tiny`. The
   full 6-PG-version test matrix runs only on the GitHub mirror at
   `.github/workflows/test.yml`.

   If you want the full matrix on Codeberg too, register a self-hosted
   runner with no time cap:

   ```bash
   # 1. Visit https://codeberg.org/gregburd/pg_turbovec/settings/actions/runners
   # 2. Click "Create new runner" → copy the token.
   # 3. ssh to a runner host (meh and arnold both work; meh has 24 cores)
   #    and run:
   export FORGEJO_RUNNER_TOKEN=<paste-the-token>
   bash scripts/install-forgejo-runner.sh
   ```

   Idempotent. Drops a static `forgejo-runner` binary into
   `~/.local/share/forgejo-runner/`, registers it against the
   repo, creates a systemd user unit so it auto-starts on boot.
   Once registered, change `.forgejo/workflows/test.yml` to also
   include the test matrix and add `runs-on: self-hosted` to it.

## Status

- **GitHub mirror CI:** ✅ green on every push (drift-check + 6 PG
  versions). The pg14 job in run `26428026790` (commit `f270120`)
  hung 6h in `apt-get install` — transient apt-mirror flakiness;
  the other 5 PG versions succeeded on the same commit. Re-running
  pg14 clears it; the next push will re-trigger CI.
- **Codeberg Actions:** ✅ enabled (`has_actions: True`). After
  the workflow split, `drift-check` will run on `codeberg-tiny`
  on every push. The full test matrix lives only on the GitHub
  mirror per the runtime-cap analysis above.

## What the workflows do

Both run the same two-stage pipeline:

1. **`drift-check` job** — runs `bash scripts/drift-check.sh` to
   verify version numbers, PG version matrix, bench-result
   references, and markdown links are consistent across the
   tree. Fails fast on any drift.
2. **`test` matrix** — `cargo pgrx test pg<N>` for N in
   `[13, 14, 15, 16, 17, 18, 19]`, each under the `native` scoring
   kernel, plus one `classic`-kernel job on pg16. The `cargo pgrx init
   --pgN download` step builds PostgreSQL N from source the first
   time the workflow runs in a given runner image, then caches
   it across runs.

Cache keys include `Cargo.lock` so a dependency bump invalidates
both the pgrx install and the cargo target dir.

## Both turbovec scoring kernels are exercised (the `layout` axis)

turbovec 1.0.0 picks its blocked-codes scoring kernel at **runtime** by
CPU feature: `avx512vnni && avx512vbmi` selects the **vector-major**
permute-dot kernel, otherwise the **classic LUT** kernel. Their 4-bit
quantized scores differ enough to flip a near-tie, so a host that only
ever runs one kernel can pass while the other returns a different top-k.
This actually shipped: the v2.0.0 port's EC2 qualification ran only on
an AVX-512-VNNI host (vector-major) and reported green, but three KNN
tests then failed on the non-VNNI GitHub runners (classic). It was fixed
by making `knn()` exact-cosine-rerank the true f32 vectors
(layout-independent), but the coverage gap is now closed structurally:
the `test` job carries a `layout` matrix axis (`native` for the full pg
range; `classic` on pg16) that sets turbovec's `TURBOVEC_NO_VECTOR_MAJOR`
env (`1` forces classic, `0` = whatever the runner supports), so **both
kernels are tested deterministically regardless of the runner's CPU**.
`cargo pgrx test` inherits the env into the postgres backend it spawns.

## CI hardware is AVX2-only for the *native* kernel selection

The GitHub `ubuntu-latest` runners are AVX2-capable, and turbovec
selects its SIMD kernel with a **runtime** `is_x86_feature_detected!`
check. So the test matrix always exercises the AVX2 path and **never**
the pre-AVX2 scalar fallback.

This matters because turbovec once shipped a silent wrong-results bug
on non-AVX2 CPUs (it returned the same TID N times instead of the
top-N). `pg_turbovec` CI would not have caught it, and still can't:

- You cannot force the scalar path from `pg_turbovec` — turbovec's
  `FORCE_SCALAR_FALLBACK` is `pub(crate)`.
- Compile-time `-C target-feature=-avx2` does **not** help: it changes
  what the compiler emits, not what the runtime feature-detect picks
  on an AVX2 machine. A CI job built that way would still run the
  AVX2 path. We deliberately do **not** add such a job — it would be
  coverage theatre.

The real mitigations are turbovec's own upstream
`x86_scalar_fallback_tests::scalar_fallback_matches_simd_topk` test
plus **validating turbovec git-rev bumps on a pre-AVX2 host** (or
`qemu-x86_64 -cpu Nehalem`) before tagging. Treat a `turbovec` rev
bump in `Cargo.toml` as the trigger for that validation. The scale +
distinct-ids regression guards that catch the *symptom* of this bug
class regardless of SIMD path live in
`index_am_recall_floor_{2,3,4}bit` and the `assert_distinct_ids`
assertions on every ANN-scan test. Full rationale: `docs/TESTING.md`.

## Common failures + fixes

| Symptom | Cause | Fix |
|---|---|---|
| `cargo-pgrx` install fails: "feature `edition2024` is required" | Rust toolchain < 1.85 | Bump `toolchain:` in setup-rust-toolchain |
| `cargo pgrx init` fails on `apt-get`-missing package | New PG version added without updating apt deps | Add the missing `lib*-dev` to the install step |
| `cargo pgrx test` flakes on `postmaster.pid` | Stale state from prior failed run, cache hit | Add `pkill -9 -f test-pgdata; mv target/test-pgdata /tmp/orph-$$` before the test step |
| GitHub Actions runs but Codeberg doesn't | Actions disabled on the Codeberg repo | See "Manual one-time step" above |

## Drift between GitHub mirror + Codeberg

Drift between the two CI workflows is tracked by
`scripts/drift-check.sh` step "10. CI matrix": both workflow files
must have the same `matrix.pg` value. Drift-check fails the
build if they disagree.