core/ROADMAP.md

# Switchboard Core — Roadmap

## Current: v0.6.2 — Docs Polish + Dynamic OpenAPI

Self-hosted extensible platform. Auth, identity, packages, Starlark sandbox,
storage, realtime, and ops are kernel primitives. Everything else is an extension.

**Kernel capabilities:** Auth (builtin/mTLS/OIDC) · Users/teams/groups/RBAC ·
Surfaces/extensions/libraries/workflows · Starlark sandbox (capability-gated) ·
Object storage (PVC/S3) + ext_data tables · WebSocket hub + realtime pub/sub ·
Audit log · Notifications · Scheduled tasks

**Completed history:** v0.2.x–v0.5.x fully documented in `CHANGELOG.md`.
Highlights: RBAC + settings cascade, event bus + triggers, SDK stabilization,
workflow engine (multi-stage, team roles, signoff gate, public entry, SLA),
package distribution, Notes surface (CM6, folders, tags, backlinks, graph),
realtime primitive, Chat surface (chat-core library + surface + polish),
upgrade test harness, cluster registry + HA.

---

## v0.6.0 — MVP

Extension, communication, and operations tracks converge. First
externally usable release.

Design docs: `docs/DESIGN-cluster-registry.md` — PG-backed cluster registry and self-assembling mesh.

### v0.6.0 — Cluster Registry + HA

PG is the consensus layer. Zero new infrastructure. `UNLOGGED` table + `LISTEN/NOTIFY` replaces etcd/Consul/Redis for homelab-to-small-team scale.

| Step | Status | Description |
|------|--------|-------------|
| `node_registry` table | ✅ | `UNLOGGED TABLE` — node_id, endpoint, seq, registered_at, heartbeat, stats JSONB. Postgres migration 013. |
| Node registration | ✅ | Self-registration on startup: `INSERT ... ON CONFLICT DO UPDATE`. `node_id` = `hostname-PID` or `CLUSTER_NODE_ID` env override. |
| Heartbeat tick | ✅ | Every 10s: update own heartbeat + collect runtime stats (goroutines, heap, GC, uptime, ws_clients). |
| Stale sweep | ✅ | Every heartbeat tick: `DELETE WHERE heartbeat < now() - 30s`. All nodes run it — idempotent, no ring topology. |
| Self-eviction | ✅ | If heartbeat UPDATE returns 0 rows: node was swept by peer → log error + `os.Exit(1)`. K8s restarts → re-register. |
| LISTEN/NOTIFY routing | ✅ | Durable events (messages, state changes) fan-out via `pg_notify`. All replicas receive, push to local WS subscribers, drop if irrelevant. Phase 1: ephemeral events (typing, presence) also via NOTIFY (8KB limit, ~60 bytes each). |
| Cluster API | ✅ | `GET /api/v1/admin/cluster` — returns `{data: [...]}` envelope with all registered nodes. |
| Admin cluster dashboard | ✅ | `cluster-dashboard` surface package renders one card per node: node_id, endpoint, uptime, ws_clients, heap, GC pause. JSONB stats — future keys render automatically, no schema migration. Auto-refresh every 10s. |
| Health endpoint | ✅ | `GET /health` includes `node_id` and `cluster: {size, peers, heartbeat_age_ms}`. |
| Config | ✅ | `CLUSTER_NODE_ID`, `CLUSTER_HEARTBEAT_INTERVAL` (default 10s), `CLUSTER_STALE_THRESHOLD` (default 30s), `CLUSTER_ENDPOINT` (Phase 2 mesh, auto-detect). |
| Single-node regression | ✅ | One-node behavior identical to pre-cluster: one registry row, NOTIFY delivers back to same instance. No special-casing. SQLite returns nil store — all cluster code guarded. |
| Multi-node integration test | ✅ | Docker Compose: 3 instances, shared PG. `ci/e2e-cluster-test.sh`: registration, stale sweep on stop, re-registration on restart. 3 unit tests + 2 handler tests. |

### v0.6.1 — Backup/Restore + Documentation

| Step | Status | Description |
|------|--------|-------------|
| Backup handler | ✅ | `POST /api/v1/admin/backup` streams `.swb` ZIP (JSONL core + ext_data tables + package assets). `POST /api/v1/admin/restore` wipes DB and restores from archive. Dialect-neutral (SQLite + Postgres). |
| Server-side backups | ✅ | `GET /api/v1/admin/backups` list, `GET /download`, `DELETE`. Store backups in `{STORAGE_PATH}/backups/`. |
| Admin backup section | ✅ | New "Backup" section under `/admin/backup`. Create (download or server-side), list, download, delete, restore with destructive confirmation. |
| Documentation API | ✅ | `GET /api/v1/docs` lists, `GET /api/v1/docs/:name` returns raw markdown. Authenticated (not admin-only). |
| Docs surface | ✅ | Builtin surface at `/docs/:section`. Sidebar navigation, client-side markdown renderer. 5 new docs: Getting Started, Extension Guide, API Reference, Deployment, Package Format. |
| Tests | ✅ | 6 handler tests (basic backup, ext_data backup, round-trip restore, schema mismatch, dump/restore table, list empty). E2E script `ci/e2e-backup-test.sh`. |

### v0.6.2 — Docs Polish + Dynamic OpenAPI

| Step | Status | Description |
|------|--------|-------------|
| Dark mode fix | ✅ | Added `--bg-code` to CSS variables (dark + light). Replaced all hardcoded light-mode fallbacks in docs CSS with theme-aware variables. Table styling, code blocks, nav items all readable in dark mode. |
| Loading & error handling | ✅ | Replaced borrowed `settings-placeholder` with docs-specific pulse animation. Added error state with retry button for failed doc list fetch. |
| Topbar navigation | ✅ | Imported `Topbar` + `UserMenu` into docs surface. Users can now navigate to other surfaces via the avatar menu. |
| Docs icon + menu entry | ✅ | Added `📖 Docs` entry to UserMenu standard items. Docs accessible from any surface's user menu. |
| `api_schema` manifest field | ✅ | Optional `api_schema` array in `manifest.json`. Parsed lazily by spec builder. Malformed entries logged and skipped — never blocks extension loading. |
| OpenAPI spec builder | ✅ | `BuildOpenAPISpec()` merges static kernel spec with extension routes. Tier 1: auto-generated stubs for all `api_routes`. Tier 2: `api_schema` replaces stubs with rich path items (params, body, response). |
| Dynamic spec endpoint | ✅ | `GET /api/docs/openapi.json` serves merged spec. Swagger UI updated to use JSON endpoint. Static YAML preserved for backward compat. |
| Tests | ✅ | 7 new handler tests: zero extensions, stubs, rich schema, multi-extension, malformed schema, disabled exclusion, required fields. |

---

## v0.6.x — Pre-Fork Hardening

Closes every audit finding before the public fork. No new features — only correctness, dead code elimination, and architectural cleanup. Sequence is fixed: each version is a gate for the next.

### v0.6.3 — Dead Code Sweep + Registry Fix

Pure cleanup. No behavior changes except fixing the broken registry install flow.

| Step | Status | Description |
|------|--------|-------------|
| Fix registry install | ☐ | SDK sends `{ url }`, handler expects `{ download_url }`. Fix `api-domains.js` to send `{ download_url: url }`. Every Install click currently returns 400. |
| Registry settings UI | ☐ | Add "Package Registry" section to admin settings with URL input field. Only way to configure registry today is a raw `PUT /api/v1/admin/settings/package_registry` — no user will find it. |
| Registry tooling + docs | ☐ | `scripts/generate-registry.sh` scans a directory of `.pkg` files and emits registry JSON. `docs/PACKAGE-REGISTRY.md` documents the format. |
| Delete dead kernel Go | ☐ | `store/interfaces.go:178–183` — orphaned ChannelListFilter comments. `pages/pages.go:922–930` — `roleFilterType()` + template registration (chat vestige, maps nonexistent roles). `main.go:67` — orphaned provider-type comment. |
| Delete dead vendor JS | ☐ | `vendor/marked.min.js` and `vendor/purify.min.js` — 62KB, zero production imports. Only referenced in test helpers. |
| Delete `dev.html` | ☐ | 676 lines, not imported by anything. Dead. |
| Remove `dashboard` from default bundle | ☐ | Requires `legacy-sdk` (doesn't exist), auto-installs as dormant on fresh installs. Confusing. Remove from `defaultBundledPackages`. |
| Remove `hello-dashboard` | ☐ | Proof-of-concept from early development. Move to `examples/` or delete. |
| Strip stale version comments | ☐ | 60+ files have `// v0.29.x:`, `// v0.33.x:` pre-fork comments. Single sed pass. |
| Narrow default bundle | ☐ | Default: `notes`, `chat`, `chat-core`, `mermaid-renderer`, `schedules`. Everything else available via registry or `BUNDLED_PACKAGES=*`. |

### v0.6.4 — Admin Health/Metrics Tab + Cluster Merge

Structural move: cluster dashboard becomes an Admin tab. Better home for health/metrics — shared context with other admin panels, no separate nav entry.

| Step | Status | Description |
|------|--------|-------------|
| "Health / Metrics" admin tab | ☐ | New tab in Admin surface. DB-agnostic metrics for all deployments. Cluster cards conditional on PG + multi-node detection. |
| Runtime metrics | ☐ | Per-node: goroutines, heap alloc/sys, stack in use, GC cycles, last GC pause, GC CPU %, uptime, WebSocket clients, extensions loaded, open FDs. |
| DB pool metrics | ☐ | All deployments: DB latency (`SELECT 1` round-trip), pool active/idle/max, wait count, wait duration. PG-only: table bloat (`n_dead_tup`), active backends (`pg_stat_activity`). |
| Cluster metrics | ☐ | PG multi-node only: cluster size, peer list with endpoint + uptime, heartbeat age per node, event bus publish/deliver rates. |
| Extension runtime metrics | ☐ | Starlark exec/min, errors/min, avg duration, HTTP outbound requests/min, trigger fires/min, schedule overruns. |
| Fatten heartbeat payload | ☐ | Heartbeat JSONB carries full metric set. `GET /api/v1/admin/metrics` for single-node SQLite fallback (same shape). |
| Retire `cluster-dashboard` | ☐ | Remove package once Admin Health tab ships. Update `defaultBundledPackages`. |
| Fix block renderer `requires` | ☐ | `mermaid-renderer`, `katex-renderer`, `csv-table`, `diff-viewer` all have `"requires": ["chat"]`. These are content renderers, not chat features. Remove constraint — they should activate without chat. |
| Health endpoint consolidation | ☐ | `/health` and `/api/v1/health` return near-identical JSON. Merge or clearly differentiate with docs. |

### v0.6.5 — Renderer Pipeline + Docs Rewrite

Most complex sub-version. Lifts block rendering to a kernel SDK primitive so all surfaces share it, then rewrites the docs for an external audience.

| Step | Status | Description |
|------|--------|-------------|
| SDK renderer primitive | ☐ | `sw.renderers.register(pattern, handler)` — kernel-level registration. Extensions call once; all surfaces consume. Decouples renderer discovery from chat surface. |
| Notes hooks SDK renderer pipeline | ☐ | Notes `renderMarkdown()` delegates fenced blocks to registered renderers instead of emitting raw `<pre>`. |
| Docs hooks SDK renderer pipeline | ☐ | Docs markdown renderer delegates fenced blocks to registered renderers. |
| Unify markdown renderer | ☐ | Three separate markdown implementations (Notes hand-rolled, Docs hand-rolled, Chat `marked`). Adopt `marked` (already vendored) across all three surfaces. Delete hand-rolled duplicates. |
| Sanitize HTML output | ☐ | DOMPurify is vendored but unused. Wire it as post-render step for user-generated markdown (Notes). Closes XSS surface. |
| Mermaid/KaTeX/CSV/Diff work everywhere | ☐ | With `requires: ["chat"]` removed (v0.6.4) and renderer pipeline lifted, these render in notes + docs + chat without surface-specific code. |
| Docs rewrite for external audience | ☐ | Remove all references to "chat-switchboard", "the fork", "the gut", "Phase 0", "scorched earth". Reframe from "we removed X" to "the kernel provides Y". Remove pre-v0.2.0 version references. |
| Add Mermaid architecture diagrams | ☐ | Diagrams in docs: system architecture overview, request flow, extension lifecycle, realtime event flow, settings cascade, cluster topology. Serves double duty: good docs + proof mermaid-renderer works in docs surface. |
| Surface alias decision | ☐ | `main.go:819–826` has six `/admin/surfaces/*` alias routes. Decision: keep permanently (document) or migrate SDK + ICD runner to `/admin/packages/` and delete. Ship one or the other, not both undocumented. |
| `CONTRIBUTING.md` + tutorial | ☐ | "Build your first extension" guide: manifest → Starlark → ext_data → API route → surface JS → install → test. Use `team-activity-log` as worked example. |

### v0.6.6 — Pre-Fork Hardening

Final pass before the hard fork. Security, correctness, and developer experience.

| Step | Status | Description |
|------|--------|-------------|
| Extension dependency auto-activation | ☐ | Installing a package with unmet `depends`/`requires` should auto-install dependencies from the bundled set, or reject with a clear error listing what's missing. Silent dormant is not acceptable for external users. |
| `ValidateManifest()` gate | ☐ | Single validation function called at install time. Catches malformed manifests early — same pattern as Unicode scan gate. |
| Package signing hook | ☐ | Optional `signature` field in manifest. `--verify` flag on install path (currently no-op). Five minutes now prevents a breaking manifest schema change post-fork. |
| OIDC state nonce validation | ☐ | `handlers/auth.go:221` — security TODO. Validate nonce on OIDC callback before accepting tokens. |
| Schema migration stub decision | ☐ | `handlers/starlark_helpers.go:93–97` — `RunSchemaMigrations` is a no-op stub called during updates. Implement real migrations or remove and document that only additive schema changes are supported. Don't ship a stub that pretends to work. |
| ICD/SDK runner update pass | ☐ | ICD runner and SDK runner are stale for cluster, backup, chat, realtime, dynamic OpenAPI. Update to cover current API surface. |
| Stale TODO resolution | ☐ | `main.go:867` (session middleware TODO, stale since v0.2.0). `auth.go:221` (OIDC nonce, covered above). `starlark_helpers.go:96` (migration stub, covered above). Resolve or delete. |

Then fork.

---

## Post-MVP

- LLM participation (`llm-bridge` extension: subscribes to `chat.message.created`, calls `provider.complete()`, streams response via `realtime.publish`, posts via `chat.send()`. Bot participants with persona config. Multi-model conversations.)
- Rich media extensions: image generation, code sandbox, STT/TTS
- Desktop app (Tauri or Electron)
- Sidecar tier: container-based extensions
- Federation: cross-instance package sharing
- Plugin marketplace with signing and review

---

## Design Decisions Log

| Decision | Rationale |
|----------|-----------|
| Tasks → extension | Scheduler was the most entangled kernel component (~3,400 lines). Rebuilding as extension validates the trigger system and removes the worst compilation debt. Three trigger primitives (time, webhook, event) replace the monolithic scheduler. |
| Sessions removed | Channel-based sessions coupled to deleted chat system. Workflow instances need new storage model — either ext_data tables or a dedicated kernel table. |
| `chat_only` → `custom` | Stage mode `chat_only` implied chat as a kernel concept. Renamed to `custom` which delegates to a surface package, proving extension composability. |
| Providers removed from kernel | Provider configs, model catalog, routing policies — all moved to extension track. Kernel provides credential storage (connections) and the Starlark `provider.complete` module as the interface. |
| Kernel permissions simplified | From 16 chat-centric permissions to 6 platform permissions. Extensions define their own capability requirements in manifests. |
| Preact+htm retained | 3KB runtime, no build step, works for extension authors without bundler config. KISS. |
| Single Docker image | Drop the frontend/backend split. Go binary + assets + migrations in one image. Simpler deployment, fewer moving parts. |
| Admin → RBAC group | The `role` column is pre-RBAC. v0.2.0 replaces it with a seeded "Admins" group + `surface.admin.access` grant. All users auto-join "Everyone" group. Admin middleware becomes a grant check, not a role check. |
| Settings cascade | RBAC controls scope auth (who can set at what level). `user_overridable` flag controls whether lower scopes can override higher. Two orthogonal axes, composes cleanly with extension manifests. |
| No new migrations pre-MVP | Edit existing migration SQL files in place. No migration chains until schema is in production. |
| Notes over Editor | First surface is Obsidian-style notes (rich text, folders, backlinks) instead of a code editor. Notes is a stronger E2E proof — it exercises ext_data, storage, and the SDK more fully than a pure-browser CM6 editor. |
| No built-in auto-install | Extensions ship in the repo but are not auto-installed. Distribution model TBD — explicit install only. Keeps the kernel clean and avoids opinionated defaults. |
| Chat as extension, not kernel | Human-to-human chat built entirely as library + surface packages. Zero kernel awareness of conversations, messages, or participants. Kernel gains one generic `realtime` module. Proves near-infinite extensibility. LLM participation layers on top via a separate bridge extension — the chat system doesn't know or care whether a participant is human or AI. |
| PG as consensus layer | Horizontal scaling uses PG as the sole coordinator (UNLOGGED node_registry + LISTEN/NOTIFY). No etcd, Consul, Redis, or Raft. Rationale: system is already tightly coupled to PG; adding a second consensus layer doubles operational complexity for zero benefit at homelab-to-small-team scale. UNLOGGED table is visible to all connections, survives session disconnect, and cleans up automatically on PG crash — correct behavior since all nodes are dead anyway. Sweep-all for health (every node deletes stale rows) is simpler than ring topology and has no edge cases. |
| Two trigger tiers | Event + webhook triggers are extension-declared (manifest contract, full sandbox). Scheduled tasks are user-created ad-hoc (restricted sandbox — no raw HTTP, no DB table creation, connections-only outbound). Separation keeps extension contracts static and user automation safe. |
| Scheduled task identity | Tasks run as their creator (RBAC-scoped). Admin-created tasks can opt into system context. Creator deactivation pauses the schedule. Ensures audit trail and permission boundaries. |
| Builtin package rationale | A builtin must enhance kernel surfaces or be required to demonstrate the platform's own capabilities. **notes** — primary content surface, exercises ext_data/storage/SDK/settings/realtime fully. **chat + chat-core** — primary communication surface, proves the extensibility thesis (100% extension, zero kernel awareness). **mermaid-renderer** — docs surface uses Mermaid diagrams to explain the architecture; without it the platform's own documentation doesn't render (self-bootstrapping). **schedules** — UI for the kernel's scheduled task system; without it users can't manage cron jobs (kernel primitive UI). All others are domain features, examples, dormant, or LLM-only tools — available via registry or `BUNDLED_PACKAGES=*`. |
| Cluster dashboard retired | `cluster-dashboard` shipped as a standalone surface package (v0.6.0) as an expedient. Health/metrics belong inside the Admin surface as a tab — shared context, no separate nav entry, no install required. Merged in v0.6.4. |
| Block renderers decoupled from chat | `mermaid-renderer`, `katex-renderer`, `csv-table`, `diff-viewer` shipped with `"requires": ["chat"]` because renderer discovery lived inside the chat surface. These are content renderers, not chat features. v0.6.4 removes the constraint; v0.6.5 lifts renderer registration to the kernel SDK so all surfaces share it without reimplementing discovery. |