Changeset 0.17.1 (#76)
This commit is contained in:
186
docs/v0171-sqlite-backend.md
Normal file
186
docs/v0171-sqlite-backend.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# v0.17.1 — SQLite Backend
|
||||
|
||||
## Overview
|
||||
|
||||
Adds SQLite as an alternative database backend alongside PostgreSQL. Enables
|
||||
single-binary deployments, air-gapped environments, edge nodes, and developer
|
||||
laptops without requiring a PostgreSQL instance.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
DB_DRIVER env var
|
||||
│
|
||||
├─ "postgres" (default) → database.DialectPostgres → postgres.NewStores()
|
||||
└─ "sqlite" → database.DialectSQLite → sqlite.NewStores()
|
||||
|
||||
Both implement the same 19 store.* interfaces unchanged.
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Parallel `store/sqlite/` package | Clean separation, no runtime branching in SQL |
|
||||
| `?` placeholders (not `$N`) | SQLite native, no conversion overhead |
|
||||
| JSON text arrays (`'["a","b"]'`) | Replace `TEXT[]`/`UUID[]` + `pq.Array` |
|
||||
| `json_each()` for array membership | Replace `= ANY(array_col)` |
|
||||
| App-side `store.NewID()` UUIDs | Replace `DEFAULT gen_random_uuid()` |
|
||||
| `datetime('now')` | Replace `NOW()` / `CURRENT_TIMESTAMP` |
|
||||
| `LIKE` fallback for search | Replace `tsvector` / `ts_rank` |
|
||||
| Feature-gated vector search | pgvector → returns error hint on SQLite |
|
||||
| `excluded.col` in ON CONFLICT | Replace Postgres `$N` reuse in SET clause |
|
||||
| WAL mode + single writer | Optimal SQLite concurrency for web apps |
|
||||
|
||||
## Files Delivered
|
||||
|
||||
### Infrastructure
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `server/config/config.go` | Added `DBDriver` field + `DB_DRIVER` env var |
|
||||
| `server/database/dialect.go` | `Dialect` type, `IsPostgres()`, `IsSQLite()` |
|
||||
| `server/database/database.go` | Dual-driver `Connect()`: postgres + sqlite |
|
||||
| `server/database/migrate.go` | Dialect-aware migration runner |
|
||||
| `server/database/migrations/sqlite/001_v017_schema.sql` | Full SQLite schema |
|
||||
| `server/store/id.go` | `store.NewID()` — app-side UUID generation |
|
||||
|
||||
### Store Layer (19 stores)
|
||||
| File | Lines | Key Transforms |
|
||||
|------|-------|---------------|
|
||||
| `store/sqlite/helpers.go` | Query builders with `?`, JSON helpers |
|
||||
| `store/sqlite/stores.go` | `NewStores()` constructor |
|
||||
| `store/sqlite/user.go` | `store.NewID()`, `datetime('now')` |
|
||||
| `store/sqlite/provider.go` | JSONB→TEXT, `store.NewID()` |
|
||||
| `store/sqlite/catalog.go` | JSONB→TEXT scanning |
|
||||
| `store/sqlite/persona.go` | `json_each()` for group grants |
|
||||
| `store/sqlite/policy.go` | `excluded.col` in upsert |
|
||||
| `store/sqlite/user_settings.go` | COALESCE with `excluded.col` |
|
||||
| `store/sqlite/channel.go` | Tags as JSON text array |
|
||||
| `store/sqlite/message.go` | `store.NewID()` |
|
||||
| `store/sqlite/note.go` | LIKE search fallback, JSON tags |
|
||||
| `store/sqlite/usage.go` | Dynamic `?` filters |
|
||||
| `store/sqlite/pricing.go` | `excluded.col` upserts |
|
||||
| `store/sqlite/extension.go` | `store.NewID()` |
|
||||
| `store/sqlite/attachment.go` | `store.NewID()` |
|
||||
| `store/sqlite/knowledge_bases.go` | `IN(?)` replaces `ANY()`, vector search gated |
|
||||
| `store/sqlite/groups.go` | `store.NewID()` |
|
||||
| `store/sqlite/resource_grants.go` | `json_each()` for array membership |
|
||||
| `store/sqlite/team.go` | `store.NewID()`, `excluded.col` |
|
||||
| `store/sqlite/audit.go` | `store.NewID()` |
|
||||
| `store/sqlite/global_config.go` | `excluded.col` upsert |
|
||||
|
||||
### Integration
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `server/MAIN_GO_PATCH.md` | Shows exact main.go changes needed |
|
||||
|
||||
## Pattern Conversion Reference
|
||||
|
||||
### Placeholders
|
||||
```sql
|
||||
-- Postgres: WHERE id = $1 AND name = $2
|
||||
-- SQLite: WHERE id = ? AND name = ?
|
||||
```
|
||||
|
||||
### UUID Generation
|
||||
```go
|
||||
// Postgres: RETURNING id (gen_random_uuid() DEFAULT)
|
||||
// SQLite: obj.ID = store.NewID()
|
||||
// INSERT INTO table (id, ...) VALUES (?, ...)
|
||||
// RETURNING created_at (id already set)
|
||||
```
|
||||
|
||||
### Array Types
|
||||
```go
|
||||
// Postgres: pq.Array(teamIDs) → team_id = ANY($3)
|
||||
// SQLite: for _, tid := range teamIDs { args = append(args, tid) }
|
||||
// team_id IN (?,?,?)
|
||||
//
|
||||
// Postgres: pq.Array(&tags) → stored as TEXT[]
|
||||
// SQLite: ArrayToJSON(tags) → stored as '["a","b"]'
|
||||
// ScanArray(jsonStr) → read back
|
||||
```
|
||||
|
||||
### Array Membership (granted_groups)
|
||||
```sql
|
||||
-- Postgres: gm.group_id = ANY(rg.granted_groups)
|
||||
-- SQLite: JOIN json_each(rg.granted_groups) je ON je.value = gm.group_id
|
||||
```
|
||||
|
||||
### ON CONFLICT Upserts
|
||||
```sql
|
||||
-- Postgres: ON CONFLICT (key) DO UPDATE SET value = $2, updated_by = $3
|
||||
-- SQLite: ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_by = excluded.updated_by
|
||||
```
|
||||
|
||||
### Timestamps
|
||||
```sql
|
||||
-- Postgres: NOW(), CURRENT_TIMESTAMP, TIMESTAMPTZ
|
||||
-- SQLite: datetime('now'), TEXT (ISO 8601)
|
||||
```
|
||||
|
||||
### Full-Text Search
|
||||
```sql
|
||||
-- Postgres: search_vector @@ to_tsquery('english', $1)
|
||||
-- SQLite: (title LIKE ? OR content LIKE ?) -- per word
|
||||
```
|
||||
|
||||
### Vector Search
|
||||
```
|
||||
-- Postgres: c.embedding <=> $1::vector (pgvector cosine distance)
|
||||
-- SQLite: Feature-gated. Returns error hint.
|
||||
-- KB ingestion works (stores text chunks).
|
||||
-- Future: sqlite-vec extension support.
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```bash
|
||||
# PostgreSQL (default — no changes needed)
|
||||
DB_DRIVER=postgres
|
||||
DATABASE_URL=postgres://user:pass@host:5432/switchboard
|
||||
|
||||
# SQLite
|
||||
DB_DRIVER=sqlite
|
||||
DATABASE_URL=switchboard.db # file path
|
||||
DATABASE_URL=:memory: # in-memory (testing)
|
||||
DATABASE_URL=/data/switchboard.db # absolute path
|
||||
```
|
||||
|
||||
Auto-detection: if `DB_DRIVER` is empty, inferred from `DATABASE_URL` format.
|
||||
|
||||
## SQLite Pragmas (set automatically)
|
||||
|
||||
| Pragma | Value | Purpose |
|
||||
|--------|-------|---------|
|
||||
| `journal_mode` | WAL | Concurrent readers |
|
||||
| `busy_timeout` | 5000ms | Retry on lock |
|
||||
| `foreign_keys` | ON | Enforce FK constraints |
|
||||
|
||||
## Limitations vs PostgreSQL
|
||||
|
||||
| Feature | PostgreSQL | SQLite |
|
||||
|---------|-----------|--------|
|
||||
| Vector similarity search | ✅ pgvector | ❌ Feature-gated |
|
||||
| Full-text search | ✅ tsvector/tsquery | ⚠ LIKE fallback |
|
||||
| Concurrent writes | ✅ MVCC | ⚠ Single writer (WAL) |
|
||||
| Array columns | ✅ Native | ⚠ JSON text arrays |
|
||||
| `NUMERIC(12,6)` precision | ✅ Exact | ⚠ REAL (float64) |
|
||||
| GIN indexes | ✅ Native | ❌ Not available |
|
||||
| Partial indexes | ✅ Full support | ✅ Supported |
|
||||
| RETURNING clause | ✅ Full support | ✅ SQLite 3.35+ |
|
||||
|
||||
## go.mod Addition
|
||||
|
||||
```
|
||||
require modernc.org/sqlite v1.34.5 // pure-Go, no CGO required
|
||||
```
|
||||
|
||||
## TODO (follow-up)
|
||||
|
||||
- [ ] Wire main.go dialect switch (see MAIN_GO_PATCH.md)
|
||||
- [ ] Add `modernc.org/sqlite` to go.mod
|
||||
- [ ] CI matrix: run integration tests against both drivers
|
||||
- [ ] SQLite-specific integration test suite
|
||||
- [ ] sqlite-vec extension support for vector search (optional)
|
||||
- [ ] Benchmark: SQLite vs PostgreSQL for typical workloads
|
||||
Reference in New Issue
Block a user