Search and media in core

Two capabilities that other systems leave to plugins ship in Cusp core: full-text search and image handling. On a fresh deploy an editor can search every document and put a picture in a post without installing anything. Both build on Cloudflare primitives Cusp already depends on — D1 for search, R2 for media — so neither adds a binding or an external service.

Search: D1 FTS5 in core

Search is a SQLite FTS5 virtual table, search_index, defined in migration 0015_search_index.sql. It is a derived projection of the document corpus, not a versioned entity — there are no entity_versions rows for it and no R2 spillover. Rebuilding it is cheap, and the authoritative history lives in documents plus their revisions.

• Tokeniser — unicode61 with remove_diacritics 2, which handles accented Latin scripts. Trigram and typo tolerance are explicit non-goals.
• Prefix index — prefix = "2 3" adds 2- and 3-character prefix tokens, which powers the admin Cmd+K type-ahead without a separate engine or a LIKE fallback.
• Ranking — BM25 with title weighted four times the body. The weight is applied per query (bm25(search_index, 0.0, 0.0, 4.0, 1.0)), not stored.
• Status column — every row carries 'published' or 'draft'. Public search filters to 'published'; admin search returns both. The publish and unpublish workflows manage the published rows the public corpus is built from.

Index writes live in the workflows

Core features run inline in the workflow steps rather than on the plugin hook bus. The index write is a step in PublishDocument, placed after the page goes live so a failed index write never blocks publication. The step's retry budget covers transient D1 errors; a permanent failure is surfaced on the admin Health page and resolved by the rebuild workflow.

• Publish — PublishDocument upserts the document into search_index with status 'published' after the page is live.
• Unpublish — UnpublishDocument removes the document from public results.
• Delete — the admin-API delete handler removes the row in the same sequence as the documents row delete; a stray row left behind is reaped by the next rebuild.

Rebuilding the index

The RebuildSearchIndex workflow (binding REBUILD_SEARCH) regenerates the whole index. Its first step counts the existing rows: unless the caller passes force, a rebuild over an already-populated index is a no-op, so first-boot enqueues are cheap. When it does run, it clears the index, then paginates the published corpus and re-indexes a page per step — a step that fails mid-page retries from the top of the same page. The setup flow kicks one rebuild when a site is first installed, and an operator can trigger one from the Health page. A brief window where results are incomplete during a rebuild is acceptable; lazy per-request rebuild was rejected because a partial index lying about result counts is not.

Search endpoints

The public endpoint is GET /api/v1/search, mounted outside the bearer gate so a visitor can search without a token. The query is required and 2 to 100 characters after trim; fewer than two returns 400. perPage caps at 50 — lower than the standard 100 because search is heavier. Responses carry public cache headers (Cache-Control: public, max-age=60, stale-while-revalidate=600) and a Surrogate-Key, and snippets wrap hits in <mark> tags. Raw input is never passed straight into FTS5 MATCH: the query helper extracts tokens and re-joins them, which closes the FTS5 syntax-injection surface (parentheses, quotes, NEAR, column prefixes).

{
  "data": [
    {
      "documentId": "01J...",
      "collectionSlug": "recipes",
      "title": "Spiced buns",
      "snippet": "...warm <mark>spiced buns</mark>...",
      "rank": 1.42
    }
  ],
  "meta": { "page": 1, "perPage": 20, "total": 7, "took_ms": 4 }
}

The admin endpoint is GET /api/admin/search, open to the administrator, editor, and author roles. It returns drafts and published documents and highlights both title and a body snippet so the Cmd+K palette feels instant — the client debounces 120 ms and the prefix index handles prefix queries directly.

Media: content-addressable R2 in core

Images ship in core too. A media table (migration 0014_media.sql) backs a first-party admin Media library, and the editor can upload, pick, caption, replace, and delete images on a fresh install. The original bytes live in R2 under a content-addressed key, and published pages can serve correctly-sized variants through Cloudflare Image Transformations — the /cdn-cgi/image URL transformer that works against R2-backed originals.

• R2 layout — media/{yyyy}/{mm}/{content_hash}.{ext}. Because the bytes under a key never change, the upload sets Cache-Control public, max-age=31536000, immutable honestly.
• Dedup by hash — a unique index on content_hash where deleted_at is null means two uploads of the same bytes share one R2 object and one library row. A duplicate upload returns the existing row with X-Cusp-Dedup: hit.
• Versioned metadata — media_metadata is in the versioned-entity registry, so every alt edit, rename, or soft-delete goes through versionedWrite and accumulates entity_versions. Site-wide time travel and restore work on media the same way they work on other entities.

Upload validation

Validation happens at the Worker boundary, in order: a 25 MB Content-Length cap, multipart parse, a MIME sniff from magic bytes (allowlist PNG, JPEG, WebP, GIF, AVIF), SHA-256, dedup lookup, a best-effort dimensions probe, the R2 PUT, then the versioned insert. The MIME check reads the actual bytes rather than trusting the Content-Type header. SVG, HTML, and JS are rejected even if the header claims an image type, because the <svg><script> XSS surface is too easy to forget. The allowlist is image-only — video and audio return 400.

Soft-delete and the SweepMedia workflow

Deleting a media row sets deleted_at and leaves the R2 bytes in place, so an accidental delete is recoverable from version history. The SweepMedia workflow (binding SWEEP_MEDIA, src/worker/workflows/sweep-media.ts) is the reaper. It wakes nightly at 02:00 UTC — an hour before the backup workflow so the nightly backup captures the post-sweep state and never archives media the sweep just hard-deleted — using the long-lived step.sleepUntil pattern, and persists its instance id under the _sweep_media_workflow_instance_id setting.

1. Find soft-deleted rows older than the 30-day retention window, oldest first, capped at 500 per pass.
2. For each row, run the reference scan — mediaRepo.referencesIn hydrates every live document's blocks and data (following R2 spillover) and walks the JSON for a matching media id.
3. If any reference remains, leave the row alone. A document that still uses a deleted image renders the missing-source fallback, and the row stays in the candidate set until the references are cleared.
4. Otherwise hard-delete the media row. Its entity_versions tombstone remains, so time travel can still see that the row used to exist.
5. Refcount the R2 object — only delete the bytes when no other live row shares the content_hash. This prevents deleting bytes a dedup-sharing row still needs.

Reference safety and the content-hash refcount are the load-bearing checks: hard-deleting a referenced image would surface as a broken image to readers, and deleting shared bytes would be silent data loss. The 30-day window plus the 500-row cap keeps each pass to seconds on a personal site and minutes on a large one. EXIF, including GPS, is preserved on the original bytes; stripping it is not current behaviour.

SweepMedia and RebuildSearchIndex are two of the long-lived Cloudflare Workflows that keep these features correct. Async work covers the self-scheduling pattern they share with the nightly backup, prune, and scheduled-publish jobs.