core/docs/archive/v0171-sqlite-backend.md

v0.17.1 — SQLite Backend

Overview

Adds SQLite as an alternative database backend alongside PostgreSQL. Enables single-binary deployments, air-gapped environments, edge nodes, and developer laptops without requiring a PostgreSQL instance.

Architecture

DB_DRIVER env var
    │
    ├─ "postgres" (default) → database.DialectPostgres → postgres.NewStores()
    └─ "sqlite"             → database.DialectSQLite   → sqlite.NewStores()

Both implement the same 19 store.* interfaces unchanged.

Key Design Decisions

Decision	Rationale
Parallel store/sqlite/ package	Clean separation, no runtime branching in SQL
? placeholders (not $N)	SQLite native, no conversion overhead
JSON text arrays ('["a","b"]')	Replace TEXT[]/UUID[] + pq.Array
json_each() for array membership	Replace = ANY(array_col)
App-side store.NewID() UUIDs	Replace DEFAULT gen_random_uuid()
datetime('now')	Replace NOW() / CURRENT_TIMESTAMP
LIKE fallback for search	Replace tsvector / ts_rank
Feature-gated vector search	pgvector → returns error hint on SQLite
excluded.col in ON CONFLICT	Replace Postgres $N reuse in SET clause
WAL mode + single writer	Optimal SQLite concurrency for web apps

Files Delivered

Infrastructure

File	Purpose
server/config/config.go	Added DBDriver field + DB_DRIVER env var
server/database/dialect.go	Dialect type, IsPostgres(), IsSQLite()
server/database/database.go	Dual-driver Connect(): postgres + sqlite
server/database/migrate.go	Dialect-aware migration runner
server/database/migrations/sqlite/001_v017_schema.sql	Full SQLite schema
server/store/id.go	store.NewID() — app-side UUID generation

Store Layer (19 stores)

File	Lines	Key Transforms
store/sqlite/helpers.go	Query builders with ?, JSON helpers
store/sqlite/stores.go	NewStores() constructor
store/sqlite/user.go	store.NewID(), datetime('now')
store/sqlite/provider.go	JSONB→TEXT, store.NewID()
store/sqlite/catalog.go	JSONB→TEXT scanning
store/sqlite/persona.go	json_each() for group grants
store/sqlite/policy.go	excluded.col in upsert
store/sqlite/user_settings.go	COALESCE with excluded.col
store/sqlite/channel.go	Tags as JSON text array
store/sqlite/message.go	store.NewID()
store/sqlite/note.go	LIKE search fallback, JSON tags
store/sqlite/usage.go	Dynamic ? filters
store/sqlite/pricing.go	excluded.col upserts
store/sqlite/extension.go	store.NewID()
store/sqlite/attachment.go	store.NewID()
store/sqlite/knowledge_bases.go	IN(?) replaces ANY(), vector search gated
store/sqlite/groups.go	store.NewID()
store/sqlite/resource_grants.go	json_each() for array membership
store/sqlite/team.go	store.NewID(), excluded.col
store/sqlite/audit.go	store.NewID()
store/sqlite/global_config.go	excluded.col upsert

Integration

File	Purpose
server/MAIN_GO_PATCH.md	Shows exact main.go changes needed

Pattern Conversion Reference

Placeholders

-- Postgres:  WHERE id = $1 AND name = $2
-- SQLite:    WHERE id = ?  AND name = ?

UUID Generation

// Postgres: RETURNING id  (gen_random_uuid() DEFAULT)
// SQLite:   obj.ID = store.NewID()
//           INSERT INTO table (id, ...) VALUES (?, ...)
//           RETURNING created_at  (id already set)

Array Types

// Postgres: pq.Array(teamIDs)  →  team_id = ANY($3)
// SQLite:   for _, tid := range teamIDs { args = append(args, tid) }
//           team_id IN (?,?,?)
//
// Postgres: pq.Array(&tags)    →  stored as TEXT[]
// SQLite:   ArrayToJSON(tags)  →  stored as '["a","b"]'
//           ScanArray(jsonStr) →  read back

Array Membership (granted_groups)

-- Postgres: gm.group_id = ANY(rg.granted_groups)
-- SQLite:   JOIN json_each(rg.granted_groups) je ON je.value = gm.group_id

ON CONFLICT Upserts

-- Postgres: ON CONFLICT (key) DO UPDATE SET value = $2, updated_by = $3
-- SQLite:   ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_by = excluded.updated_by

Timestamps

-- Postgres: NOW(), CURRENT_TIMESTAMP, TIMESTAMPTZ
-- SQLite:   datetime('now'), TEXT (ISO 8601)

Full-Text Search

-- Postgres: search_vector @@ to_tsquery('english', $1)
-- SQLite:   (title LIKE ? OR content LIKE ?)  -- per word

Vector Search

-- Postgres: c.embedding <=> $1::vector  (pgvector cosine distance)
-- SQLite:   Feature-gated. Returns error hint.
--           KB ingestion works (stores text chunks).
--           Future: sqlite-vec extension support.

Configuration

# PostgreSQL (default — no changes needed)
DB_DRIVER=postgres
DATABASE_URL=postgres://user:pass@host:5432/switchboard

# SQLite
DB_DRIVER=sqlite
DATABASE_URL=switchboard.db          # file path
DATABASE_URL=:memory:                # in-memory (testing)
DATABASE_URL=/data/switchboard.db    # absolute path

Auto-detection: if DB_DRIVER is empty, inferred from DATABASE_URL format.

SQLite Pragmas (set automatically)

Pragma	Value	Purpose
journal_mode	WAL	Concurrent readers
busy_timeout	5000ms	Retry on lock
foreign_keys	ON	Enforce FK constraints

Limitations vs PostgreSQL

Feature	PostgreSQL	SQLite
Vector similarity search	✅ pgvector	❌ Feature-gated
Full-text search	✅ tsvector/tsquery	⚠ LIKE fallback
Concurrent writes	✅ MVCC	⚠ Single writer (WAL)
Array columns	✅ Native	⚠ JSON text arrays
NUMERIC(12,6) precision	✅ Exact	⚠ REAL (float64)
GIN indexes	✅ Native	❌ Not available
Partial indexes	✅ Full support	✅ Supported
RETURNING clause	✅ Full support	✅ SQLite 3.35+

go.mod Addition

require modernc.org/sqlite v1.34.5  // pure-Go, no CGO required

TODO (follow-up)

• Wire main.go dialect switch (see MAIN_GO_PATCH.md)
• Add modernc.org/sqlite to go.mod
• CI matrix: run integration tests against both drivers
• SQLite-specific integration test suite
• sqlite-vec extension support for vector search (optional)
• Benchmark: SQLite vs PostgreSQL for typical workloads