switchboard/core
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Releases 9 Wiki Activity

Changeset 0.17.1 (#76)

Browse Source
This commit is contained in:
xcaliber
2026-02-28 01:40:31 +00:00
parent c9141a6896
commit 856dc9b0ac
64 changed files with 8037 additions and 1657 deletions
Download Patch File Download Diff File Expand all files Collapse all files

63
docs/ROADMAP.md
View File

@@ -60,7 +60,7 @@ v0.16.0 User Groups v0.17.0 Persona-KB Binding
│
┌───────┴──────────────┐
│ │
v0.17.1 SQLite Backend v0.17.2 CodeMirror 6
v0.17.1 SQLite Backend ✅ v0.17.2 CodeMirror 6
(dual DB) (editor bundle, chat
│ input, ext editor)
└───────┬──────────────┘
@@ -244,46 +244,45 @@ Depends on: knowledge bases (v0.14.0), user groups (v0.16.0).
---
## v0.17.1 — SQLite Backend
## v0.17.1 — SQLite Backend ✅
Dual-database support. Adding a second backend now means every subsequent
schema change (memory, projects, channels restructure, auth) is developed
against both Postgres and SQLite from day one — no retrofit. SQLite enables
single-binary deployment for dev, demo, and single-user scenarios.
Dual-database support. Every subsequent schema change is developed
against both Postgres and SQLite from day one — no retrofit. SQLite
enables single-binary deployment for dev, demo, edge, and single-user.
Depends on: v0.17.0 DB debt cleanup (store layer is sole DB interface,
no raw `database.DB` calls remain).
**Rationale: Why here, not later**
- v0.17.0 cleans up the last raw `ExecContext` bypass
- v0.18.0 adds another `vector()` column (memory) — better to have the
vector abstraction in place first
- v0.19.0–v0.24.0 each add schema — every one benefits from dual-backend CI
Depends on: v0.17.0 DB debt cleanup (store layer is sole DB interface).
**Store Layer**
- [ ] SQLite implementations for all store interfaces
- [ ] `DB_DRIVER` env var: `postgres` (default) | `sqlite`
- [ ] Shared migration runner with dialect-aware SQL (or separate migration files per driver)
- [ ] Connection pooling / WAL mode for SQLite concurrency
- [ ] CI matrix: run integration tests against both Postgres and SQLite
- [x] 19 SQLite store implementations covering all domain interfaces
- [x] `DB_DRIVER` env var: `postgres` (default) | `sqlite`
- [x] Separate migration files per driver (`migrations/sqlite/`)
- [x] WAL mode + single-writer connection pooling for SQLite concurrency
- [x] CI matrix: full handler integration tests against both Postgres and SQLite
**Vector Search Abstraction**
- [ ] `VectorStore` interface extracted from `KnowledgeBaseStore` + `NoteStore`
- [ ] Postgres implementation: pgvector cosine similarity (existing)
- [ ] SQLite implementation: sqlite-vec extension, or feature-gated (KB search disabled without it)
- [ ] Graceful degradation: if vector search unavailable, `kb_search` tool returns error hint, KBs still store/serve documents
**Vector Search**
- [x] App-level cosine similarity in Go (pure Go, no CGO/sqlite-vec required)
- [x] Embeddings stored as JSON text in `kb_chunks.embedding` column
- [x] Full feature parity with pgvector: `SimilaritySearch`, `InsertChunks`
- [x] Graceful: adequate performance for SQLite-scale deployments (single-user, not millions of chunks)
**Schema Compatibility**
- [ ] UUID generation: `gen_random_uuid()` → application-side UUID generation (Go `uuid.New()`)
- [ ] JSONB columns → JSON text columns with application-side marshaling
- [ ] `ARRAY` types (e.g. `granted_groups UUID[]`) → junction tables or JSON arrays
- [ ] Timestamp handling: `TIMESTAMPTZ` → SQLite `TEXT` with ISO 8601
- [ ] `ON CONFLICT` / upsert syntax alignment
- [ ] Trigger syntax differences (PostgreSQL `EXECUTE FUNCTION` vs SQLite `BEGIN...END`)
- [x] UUID generation: application-side `store.NewID()` (Go `uuid.New()`)
- [x] JSONB → JSON text columns with `ToJSON()`/`ScanJSON()` helpers
- [x] `UUID[]` arrays → JSON arrays with `ToJSONArray()`/`ScanJSONArray()`
- [x] `TIMESTAMPTZ` → SQLite `TEXT` with `datetime('now')`
- [x] `ON CONFLICT excluded.*` pattern (works in both dialects)
- [x] `INSERT RETURNING id` → separate query pattern for SQLite
**Test Infrastructure**
- [x] `dialectSQL()` converts `$N` → `?` and strips `::jsonb` at runtime
- [x] `database.PH(n)` returns dialect-appropriate placeholder
- [x] `database.TruncateAll()` dialect-aware (DELETE + PRAGMA vs TRUNCATE CASCADE)
- [x] Generic provider config: `PROVIDER`/`PROVIDER_KEY`/`PROVIDER_URL` (replaces `VENICE_API_KEY`)
- [x] Model selection prefers non-reasoning models in live tests
**What SQLite mode skips** (feature-gated, not broken):
- [ ] pgvector-dependent features (KB similarity search, semantic note search) unless sqlite-vec available
- [ ] `LISTEN/NOTIFY` (WebSocket event fan-out uses in-process EventBus instead — already works)
- [x] `LISTEN/NOTIFY` — WebSocket event fan-out uses in-process EventBus (already works)
- [x] Full-text search `tsvector` — conversation_search uses LIKE fallback
---

186
docs/v0171-sqlite-backend.md Normal file
View File

@@ -0,0 +1,186 @@
# v0.17.1 — SQLite Backend
## Overview
Adds SQLite as an alternative database backend alongside PostgreSQL. Enables
single-binary deployments, air-gapped environments, edge nodes, and developer
laptops without requiring a PostgreSQL instance.
## Architecture
```
DB_DRIVER env var
│
├─ "postgres" (default) → database.DialectPostgres → postgres.NewStores()
└─ "sqlite" → database.DialectSQLite → sqlite.NewStores()
Both implement the same 19 store.* interfaces unchanged.
```
### Key Design Decisions
| Decision | Rationale |
|----------|-----------|
| Parallel `store/sqlite/` package | Clean separation, no runtime branching in SQL |
| `?` placeholders (not `$N`) | SQLite native, no conversion overhead |
| JSON text arrays (`'["a","b"]'`) | Replace `TEXT[]`/`UUID[]` + `pq.Array` |
| `json_each()` for array membership | Replace `= ANY(array_col)` |
| App-side `store.NewID()` UUIDs | Replace `DEFAULT gen_random_uuid()` |
| `datetime('now')` | Replace `NOW()` / `CURRENT_TIMESTAMP` |
| `LIKE` fallback for search | Replace `tsvector` / `ts_rank` |
| Feature-gated vector search | pgvector → returns error hint on SQLite |
| `excluded.col` in ON CONFLICT | Replace Postgres `$N` reuse in SET clause |
| WAL mode + single writer | Optimal SQLite concurrency for web apps |
## Files Delivered
### Infrastructure
| File | Purpose |
|------|---------|
| `server/config/config.go` | Added `DBDriver` field + `DB_DRIVER` env var |
| `server/database/dialect.go` | `Dialect` type, `IsPostgres()`, `IsSQLite()` |
| `server/database/database.go` | Dual-driver `Connect()`: postgres + sqlite |
| `server/database/migrate.go` | Dialect-aware migration runner |
| `server/database/migrations/sqlite/001_v017_schema.sql` | Full SQLite schema |
| `server/store/id.go` | `store.NewID()` — app-side UUID generation |
### Store Layer (19 stores)
| File | Lines | Key Transforms |
|------|-------|---------------|
| `store/sqlite/helpers.go` | Query builders with `?`, JSON helpers |
| `store/sqlite/stores.go` | `NewStores()` constructor |
| `store/sqlite/user.go` | `store.NewID()`, `datetime('now')` |
| `store/sqlite/provider.go` | JSONB→TEXT, `store.NewID()` |
| `store/sqlite/catalog.go` | JSONB→TEXT scanning |
| `store/sqlite/persona.go` | `json_each()` for group grants |
| `store/sqlite/policy.go` | `excluded.col` in upsert |
| `store/sqlite/user_settings.go` | COALESCE with `excluded.col` |
| `store/sqlite/channel.go` | Tags as JSON text array |
| `store/sqlite/message.go` | `store.NewID()` |
| `store/sqlite/note.go` | LIKE search fallback, JSON tags |
| `store/sqlite/usage.go` | Dynamic `?` filters |
| `store/sqlite/pricing.go` | `excluded.col` upserts |
| `store/sqlite/extension.go` | `store.NewID()` |
| `store/sqlite/attachment.go` | `store.NewID()` |
| `store/sqlite/knowledge_bases.go` | `IN(?)` replaces `ANY()`, vector search gated |
| `store/sqlite/groups.go` | `store.NewID()` |
| `store/sqlite/resource_grants.go` | `json_each()` for array membership |
| `store/sqlite/team.go` | `store.NewID()`, `excluded.col` |
| `store/sqlite/audit.go` | `store.NewID()` |
| `store/sqlite/global_config.go` | `excluded.col` upsert |
### Integration
| File | Purpose |
|------|---------|
| `server/MAIN_GO_PATCH.md` | Shows exact main.go changes needed |
## Pattern Conversion Reference
### Placeholders
```sql
-- Postgres: WHERE id = $1 AND name = $2
-- SQLite: WHERE id = ? AND name = ?
```
### UUID Generation
```go
// Postgres: RETURNING id (gen_random_uuid() DEFAULT)
// SQLite: obj.ID = store.NewID()
// INSERT INTO table (id, ...) VALUES (?, ...)
// RETURNING created_at (id already set)
```
### Array Types
```go
// Postgres: pq.Array(teamIDs) → team_id = ANY($3)
// SQLite: for _, tid := range teamIDs { args = append(args, tid) }
// team_id IN (?,?,?)
//
// Postgres: pq.Array(&tags) → stored as TEXT[]
// SQLite: ArrayToJSON(tags) → stored as '["a","b"]'
// ScanArray(jsonStr) → read back
```
### Array Membership (granted_groups)
```sql
-- Postgres: gm.group_id = ANY(rg.granted_groups)
-- SQLite: JOIN json_each(rg.granted_groups) je ON je.value = gm.group_id
```
### ON CONFLICT Upserts
```sql
-- Postgres: ON CONFLICT (key) DO UPDATE SET value = $2, updated_by = $3
-- SQLite: ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_by = excluded.updated_by
```
### Timestamps
```sql
-- Postgres: NOW(), CURRENT_TIMESTAMP, TIMESTAMPTZ
-- SQLite: datetime('now'), TEXT (ISO 8601)
```
### Full-Text Search
```sql
-- Postgres: search_vector @@ to_tsquery('english', $1)
-- SQLite: (title LIKE ? OR content LIKE ?) -- per word
```
### Vector Search
```
-- Postgres: c.embedding <=> $1::vector (pgvector cosine distance)
-- SQLite: Feature-gated. Returns error hint.
-- KB ingestion works (stores text chunks).
-- Future: sqlite-vec extension support.
```
## Configuration
```bash
# PostgreSQL (default — no changes needed)
DB_DRIVER=postgres
DATABASE_URL=postgres://user:pass@host:5432/switchboard
# SQLite
DB_DRIVER=sqlite
DATABASE_URL=switchboard.db # file path
DATABASE_URL=:memory: # in-memory (testing)
DATABASE_URL=/data/switchboard.db # absolute path
```
Auto-detection: if `DB_DRIVER` is empty, inferred from `DATABASE_URL` format.
## SQLite Pragmas (set automatically)
| Pragma | Value | Purpose |
|--------|-------|---------|
| `journal_mode` | WAL | Concurrent readers |
| `busy_timeout` | 5000ms | Retry on lock |
| `foreign_keys` | ON | Enforce FK constraints |
## Limitations vs PostgreSQL
| Feature | PostgreSQL | SQLite |
|---------|-----------|--------|
| Vector similarity search | ✅ pgvector | ❌ Feature-gated |
| Full-text search | ✅ tsvector/tsquery | ⚠ LIKE fallback |
| Concurrent writes | ✅ MVCC | ⚠ Single writer (WAL) |
| Array columns | ✅ Native | ⚠ JSON text arrays |
| `NUMERIC(12,6)` precision | ✅ Exact | ⚠ REAL (float64) |
| GIN indexes | ✅ Native | ❌ Not available |
| Partial indexes | ✅ Full support | ✅ Supported |
| RETURNING clause | ✅ Full support | ✅ SQLite 3.35+ |
## go.mod Addition
```
require modernc.org/sqlite v1.34.5 // pure-Go, no CGO required
```
## TODO (follow-up)
- [ ] Wire main.go dialect switch (see MAIN_GO_PATCH.md)
- [ ] Add `modernc.org/sqlite` to go.mod
- [ ] CI matrix: run integration tests against both drivers
- [ ] SQLite-specific integration test suite
- [ ] sqlite-vec extension support for vector search (optional)
- [ ] Benchmark: SQLite vs PostgreSQL for typical workloads