ADR 0045/0046 — Search in core via D1 FTS5

Search is a built-in capability in Cusp, not a plugin you install. It works on first boot, searching published content for the public site and drafts for the admin. Two decisions sit behind it: which engine backs the index, and where the index gets written.

Decision

Keyword search lives in core, backed by a single FTS5 virtual table named search_index in the same D1 database everything else uses. There is no new binding and no new billable service. The index write is inlined as a retryable workflow step inside the publish and unpublish flows rather than running as a best-effort plugin hook.

Why FTS5 in D1

Editor-first is the load-bearing product invariant, and making someone install a plugin to get search is friction against it. Search is too core to leave one indirection away.

• FTS5 is built into the SQLite that D1 already runs — no new binding, no new service, no marginal cost per publish.
• It preserves the one-Worker, one-D1, one-R2, one-KV invariant that an external search service would break.
• A JS-side index shipped to the browser was rejected: it bloats the bundle, cannot search drafts behind auth, and does not scale past a few hundred documents.
• Vectorize and semantic search were rejected for core: a separate billable binding plus embedding cost on every publish, and semantic ranking is overkill for keyword lookup. That remains plugin territory — the line to cross is when search means asking the site a question rather than finding the post with these words.

The trade-off accepted: keyword-only matching, no semantic ranking, no typo tolerance. That is acceptable for the personal and small-team sites Cusp targets.

How the index is shaped

The search_index table is a derived projection of documents and revisions. It is not a versioned entity — it has no entity_versions rows and no R2 spillover. If it is ever lost or corrupted, the recovery path is to rebuild it from the corpus with the RebuildSearchIndex workflow; history is owned by documents.

• Tokenizer — unicode61 remove_diacritics 2, so accented Latin scripts match their unaccented forms.
• Prefix indexes — 2- and 3-character prefixes, cheap to maintain, and they power the admin Cmd+K type-ahead with the same engine rather than a second one.
• Ranking — BM25 with the title column weighted 4x the body, so a match in the title outranks the same match in the body.
• Indexed columns — title and body are tokenised; document_id, collection_slug, status, and updated_at are stored UNINDEXED for filtering and tie-break ordering.

Surfaces: a public GET /api/v1/search endpoint that filters to published rows, and an admin GET /api/admin/search endpoint that also returns drafts and backs the Cmd+K command palette. There is no core-rendered /search page — presenting results is a theme's concern, driven off the public endpoint. Both API endpoints sanitise the raw query into a safe FTS5 MATCH expression — FTS5 query-string handling is a core security surface, so reserved operators are stripped and the token count is capped before anything reaches the parser.

Why index writes run in the workflow

Plugin hooks are async and fire-and-forget — failures are swallowed by design, so third-party code can never break a core flow. That posture is right for plugins but wrong for a core capability, where a failed index write is a real defect that should be retried and surfaced rather than vanishing.

So the write is inlined where it can use the workflow's per-step retry budget and structured observability:

• PublishDocument — an index-for-search step upserts the row. It runs after the document and its listings are on R2, so a search hit always lands on a coherent edge, and a failed index write never blocks the publish itself.
• UnpublishDocument — a deindex-from-search step removes the document from the index, and the admin delete path runs through this same workflow so a removed document leaves no orphaned search row.

The plugin hook bus and its degrade-to-noop contract are unchanged; this decision is specifically about core-owned derived projections, not plugins.

Consequences

• Search works on first boot with no plugin install and no extra binding.
• Failed index writes are visible on the admin Health page and recoverable with a Rebuild search index action backed by the RebuildSearchIndex workflow.
• Index retry posture matches the rest of publish — predictable, not silent.
• Results are keyword-only; index size grows linearly with body text, so very large long-form sites will see query latency creep up over time.

Search shipped in core from the start, so there was never a migration off a search plugin. A separately-scoped search plugin was designed but never released; the automatic copy-rows-and-uninstall migration that would have retired it was consequently never implemented. Every deploy has had core FTS5 search from its first boot.