# Design notes: the Datomic-review issues (#325–#332)

Each entry: what it is, the decision taken, the trade-offs, and what's left.
Issues #329, #331, #332 (core) shipped in 1.13.0; the rest land in 1.14.0.

---

## #325 — user-defined transaction functions (PostgreSQL, `script` builds)

**What.** An ident carrying `:db/fn` (bootstrap attribute entid 20, a string
holding mino source `(fn [db & args] ...)`) is a transaction function. A
transaction invokes it as `[:my/fn arg ...]`; it returns tx-data that is spliced
into the same transaction.

**Decision.** Expand invocations *before* the transactor's two passes
(`functions/txfn.rs::expand_user_tx_fns`), recursively, in one sandboxed
interpreter per transaction. The expansion is plain tx-data, so tempids,
upserts, built-ins (`:db.fn/cas`, `:db/retractEntity`) and cardinality are
handled by the existing code unchanged. Full design in
`325-user-defined-transaction-functions.md`.

| Trade-off | Chosen | Alternative | Why |
|---|---|---|---|
| Language | mino (sandboxed Clojure dialect) | JVM Clojure | No JVM in the backend; mino is the reviewed in-backend runner. Bodies need a mechanical rewrite from Datomic `:db/fn`. |
| Where expansion happens | Pre-pass over the tx vector | Inside the transactor's entity loop (work-queue) | The loop iterates a fixed vector; a pre-pass needed no transactor restructuring and can't disturb tempid/upsert ordering. |
| `db` the fn sees | Basis at the start of the tx | In-progress db including earlier datoms in the same tx | Matches Datomic, and avoids exposing half-applied state. A fn that needs an atomic read-modify-write uses `:db.fn/cas` in its output. |
| Storage of the fn | `:db/fn` string datom | Opaque/compiled blob | Transparent (pull, pg_dump), versioned in history like any datom. Re-parsed per tx; cheap. |
| Recursion bound | `mentat.txfn_max_depth` (16) + the `mentat.script_max_*` limits | Unbounded | A self-calling fn must fail cleanly, not loop. |
| Availability | `script` feature only | Always on | The interpreter is behind `script`; default builds pay nothing. |

**Security.** Same sandbox and limits as `edn_eval` (no filesystem/network
prims; steps/heap/depth GUCs; interrupt hook); runs as the calling role; never
`SECURITY DEFINER`. Installing a fn is transacting a string, i.e. the same trust
as defining any attribute.

**Not done (yet).** The embedded SQLite and DuckDB backends: they have the
same `ScriptBackend` seam, so the same pre-pass could run there; tracked as a
follow-up. A role gate on *installing* `:db/fn` (only if someone asks).

---

## #326 — `:db/retractEntity` retracts inbound references

**What.** Datomic's retractEntity retracts every datom whose *value* is a ref to
the entity, and cascades through `:db/isComponent`. All three backends did the
component cascade but left inbound refs dangling.

**Decision.** Retract `[?referrer ?a e]` for every live ref datom pointing at
`e` (pg: `current_datoms` where `v_ref = e`; SQLite: `datoms` via the VAET
index; DuckDB: the `v_i` mirror column). Only the ref datom goes; the referring
entity survives. New `MentatStoring::referrers` on SQLite/DuckDB.

| Trade-off | Chosen | Alternative |
|---|---|---|
| Behavior change | Always (Datomic semantics) | Opt-in flag | It's a correctness fix to match the documented model; a flag would keep the surprising default. |
| Cascade referrers? | No — retract only the ref datom | Delete referring entities | Datomic doesn't; it would destroy unrelated data. |
| Read source (pg) | Current-state projection | Append-only log `WHERE added` | The log keeps superseded assertions; reading it re-retracts history (found while testing; the outbound read moved to the projection too). |

**Cost.** One indexed lookup per retracted entity (VAET / `v_ref` index).
History is unchanged in shape: the extra retractions are ordinary datoms.

---

## #327 — lookup refs in `:db.fn/cas` / `:db/retractEntity`

**What.** Datomic accepts `[:unique/attr value]` wherever an entity id goes.

**State.** PostgreSQL already did (1.13.0 added a regression test). The embedded
SQLite (and DuckDB, which shares the transactor) rejected it.

**Decision.** `resolve_tx_fn_entid` resolves a lookup ref through the same
`resolve_avs` the transactor uses for ordinary lookup refs; the attribute must be
`:db/unique` and the entity must exist (a tx-fn can't upsert).

| Trade-off | Chosen | Alternative |
|---|---|---|
| Missing entity | Error | Allocate (upsert) | cas/retractEntity of a non-existent entity is meaningless; erroring is what pg_mentat does. |

---

## #328 — scripting `q` against an as-of / since db

**What.** `(mentat.store/q (mentat.store/as-of db t) query)` errored on the
embedded backend ("Mentat has no as-of query rewriting").

**Finding.** The refusal was stale: the embedded engine has had temporal
Datalog (`Store::q_once_as_of` / `q_once_since`) since 1.8.0, DuckDB's backend
already used it, and pg passes `asOf`/`since` to `edn_q`.

**Decision.** Route an as-of/since db to the temporal query entry points, with
or without `:in` inputs. A shared model test now pins it on all three backends.

| Trade-off | Chosen | Alternative |
|---|---|---|
| `pull` on a historical db (SQLite) | Still refused | Reconstruct via datoms | Separate change; `entity`/`read`/`datoms` already work on a historical basis. |

---

## #330 — `mentat_explain` / `mentat_query_sql` / `query_view` ignore collection `:in`

**What.** These compiled the query through a wrapper that passed *no* enriched
(collection / tuple / relation) bindings, so a `[?x ...]` input was silently
dropped: the SQL had no filter and EXPLAIN showed a plan edn_q never runs.

**Decision.** Build the enriched bindings exactly as the executing path does
(`parse_input_bindings_v2`) in all three entry points, and delete the
binding-dropping wrapper so it can't recur.

| Trade-off | Chosen | Alternative |
|---|---|---|
| Missing declared input | Error (same as edn_q) | Silently unfiltered SQL | The old behavior hid the bug. |

---

## #332 follow-up — `(bm25-match ...)` and `(turbovec-near ...)` where-fns

**What.** 1.13.0 shipped the pg_fts / pg_turbovec helper functions; queries
couldn't use the indexes from Datalog.

**Decision.**
- `[(bm25-match $ :attr "query" k) [[?e ?score]]]` → a derived table over
  `datoms_text_new` with `to_ftsdoc(lang, v) @@@ to_ftsquery(lang, q)` and
  `ORDER BY ... <=> ...` (BM25, served by the pg_fts index), `LIMIT k`.
- `[(turbovec-near $ :attr "[...]" k [:cosine|:l2|:inner|:l1]) [[?e ?dist]]]`
  → top-K from the per-attribute `turbovec.vector` aux table, using
  schema-qualified `OPERATOR(turbovec.<op>)`.

| Trade-off | Chosen | Alternative |
|---|---|---|
| bm25 attr/store predicate | Inlined integer literals | Bind parameters | `create_fts_index` builds a **partial** index (`WHERE a = <entid> AND store_id = 0 AND added`); the planner only uses it if the query predicate provably implies that, which a parameter doesn't. Both values are integers we resolve, never user text. |
| Score direction | Raw `<=>` (smaller = better), as pg_fts orders | Negate to "higher is better" | Matches pg_fts and pgvector-style distance semantics already used by `vector-near`. |
| Vector dimension | `CHECK (vector_dims(v) = N)` | `turbovec.vector(N)` typmod | pg_turbovec's type takes no typmod. Note pg_turbovec also requires `dim % 8 == 0` to build an index. |