core/ROADMAP.md

Switchboard Core — Roadmap

Current: v0.6.3 — Dead Code Sweep + Registry Fix

Self-hosted extensible platform. Auth, identity, packages, Starlark sandbox, storage, realtime, and ops are kernel primitives. Everything else is an extension.

Kernel capabilities: Auth (builtin/mTLS/OIDC) · Users/teams/groups/RBAC · Surfaces/extensions/libraries/workflows · Starlark sandbox (capability-gated) · Object storage (PVC/S3) + ext_data tables · WebSocket hub + realtime pub/sub · Audit log · Notifications · Scheduled tasks

Completed history: v0.2.x–v0.5.x fully documented in CHANGELOG.md. Highlights: RBAC + settings cascade, event bus + triggers, SDK stabilization, workflow engine (multi-stage, team roles, signoff gate, public entry, SLA), package distribution, Notes surface (CM6, folders, tags, backlinks, graph), realtime primitive, Chat surface (chat-core library + surface + polish), upgrade test harness, cluster registry + HA.

---

v0.6.0 — MVP

Extension, communication, and operations tracks converge. First externally usable release.

Design docs: docs/DESIGN-cluster-registry.md — PG-backed cluster registry and self-assembling mesh.

v0.6.0 — Cluster Registry + HA

PG is the consensus layer. Zero new infrastructure. UNLOGGED table + LISTEN/NOTIFY replaces etcd/Consul/Redis for homelab-to-small-team scale.

Step	Status	Description
node_registry table	✅	UNLOGGED TABLE — node_id, endpoint, seq, registered_at, heartbeat, stats JSONB. Postgres migration 013.
Node registration	✅	Self-registration on startup: INSERT ... ON CONFLICT DO UPDATE. node_id = hostname-PID or CLUSTER_NODE_ID env override.
Heartbeat tick	✅	Every 10s: update own heartbeat + collect runtime stats (goroutines, heap, GC, uptime, ws_clients).
Stale sweep	✅	Every heartbeat tick: DELETE WHERE heartbeat < now() - 30s. All nodes run it — idempotent, no ring topology.
Self-eviction	✅	If heartbeat UPDATE returns 0 rows: node was swept by peer → log error + os.Exit(1). K8s restarts → re-register.
LISTEN/NOTIFY routing	✅	Durable events (messages, state changes) fan-out via pg_notify. All replicas receive, push to local WS subscribers, drop if irrelevant. Phase 1: ephemeral events (typing, presence) also via NOTIFY (8KB limit, ~60 bytes each).
Cluster API	✅	GET /api/v1/admin/cluster — returns {data: [...]} envelope with all registered nodes.
Admin cluster dashboard	✅	cluster-dashboard surface package renders one card per node: node_id, endpoint, uptime, ws_clients, heap, GC pause. JSONB stats — future keys render automatically, no schema migration. Auto-refresh every 10s.
Health endpoint	✅	GET /health includes node_id and cluster: {size, peers, heartbeat_age_ms}.
Config	✅	CLUSTER_NODE_ID, CLUSTER_HEARTBEAT_INTERVAL (default 10s), CLUSTER_STALE_THRESHOLD (default 30s), CLUSTER_ENDPOINT (Phase 2 mesh, auto-detect).
Single-node regression	✅	One-node behavior identical to pre-cluster: one registry row, NOTIFY delivers back to same instance. No special-casing. SQLite returns nil store — all cluster code guarded.
Multi-node integration test	✅	Docker Compose: 3 instances, shared PG. ci/e2e-cluster-test.sh: registration, stale sweep on stop, re-registration on restart. 3 unit tests + 2 handler tests.

v0.6.1 — Backup/Restore + Documentation

Step	Status	Description
Backup handler	✅	POST /api/v1/admin/backup streams .swb ZIP (JSONL core + ext_data tables + package assets). POST /api/v1/admin/restore wipes DB and restores from archive. Dialect-neutral (SQLite + Postgres).
Server-side backups	✅	GET /api/v1/admin/backups list, GET /download, DELETE. Store backups in {STORAGE_PATH}/backups/.
Admin backup section	✅	New "Backup" section under /admin/backup. Create (download or server-side), list, download, delete, restore with destructive confirmation.
Documentation API	✅	GET /api/v1/docs lists, GET /api/v1/docs/:name returns raw markdown. Authenticated (not admin-only).
Docs surface	✅	Builtin surface at /docs/:section. Sidebar navigation, client-side markdown renderer. 5 new docs: Getting Started, Extension Guide, API Reference, Deployment, Package Format.
Tests	✅	6 handler tests (basic backup, ext_data backup, round-trip restore, schema mismatch, dump/restore table, list empty). E2E script ci/e2e-backup-test.sh.

v0.6.2 — Docs Polish + Dynamic OpenAPI

Step	Status	Description
Dark mode fix	✅	Added --bg-code to CSS variables (dark + light). Replaced all hardcoded light-mode fallbacks in docs CSS with theme-aware variables. Table styling, code blocks, nav items all readable in dark mode.
Loading & error handling	✅	Replaced borrowed settings-placeholder with docs-specific pulse animation. Added error state with retry button for failed doc list fetch.
Topbar navigation	✅	Imported Topbar + UserMenu into docs surface. Users can now navigate to other surfaces via the avatar menu.
Docs icon + menu entry	✅	Added 📖 Docs entry to UserMenu standard items. Docs accessible from any surface's user menu.
api_schema manifest field	✅	Optional api_schema array in manifest.json. Parsed lazily by spec builder. Malformed entries logged and skipped — never blocks extension loading.
OpenAPI spec builder	✅	BuildOpenAPISpec() merges static kernel spec with extension routes. Tier 1: auto-generated stubs for all api_routes. Tier 2: api_schema replaces stubs with rich path items (params, body, response).
Dynamic spec endpoint	✅	GET /api/docs/openapi.json serves merged spec. Swagger UI updated to use JSON endpoint. Static YAML preserved for backward compat.
Tests	✅	7 new handler tests: zero extensions, stubs, rich schema, multi-extension, malformed schema, disabled exclusion, required fields.

---

v0.6.x — Pre-Fork Hardening

Closes every audit finding before the public fork. No new features — only correctness, dead code elimination, and architectural cleanup. Sequence is fixed: each version is a gate for the next.

v0.6.3 — Dead Code Sweep + Registry Fix

Pure cleanup. No behavior changes except fixing the broken registry install flow.

Step	Status	Description
Fix registry install	✅	SDK sends { url }, handler expects { download_url }. Fix api-domains.js to send { download_url: url }. Every Install click currently returns 400.
Registry settings UI	✅	Add "Package Registry" section to admin settings with URL input field. Only way to configure registry today is a raw PUT /api/v1/admin/settings/package_registry — no user will find it.
Registry tooling + docs	✅	scripts/generate-registry.sh scans a directory of .pkg files and emits registry JSON. docs/PACKAGE-REGISTRY.md documents the format.
Delete dead kernel Go	✅	store/interfaces.go:178–183 — orphaned ChannelListFilter comments. pages/pages.go:922–930 — roleFilterType() + template registration (chat vestige, maps nonexistent roles). main.go:67 — orphaned provider-type comment.
Delete dead vendor JS	✅	vendor/marked.min.js and vendor/purify.min.js — 62KB, zero production imports. Only referenced in test helpers.
Delete dev.html	✅	676 lines, not imported by anything. Dead.
Remove dashboard from default bundle	✅	Requires legacy-sdk (doesn't exist), auto-installs as dormant on fresh installs. Confusing. Remove from defaultBundledPackages.
Remove hello-dashboard	✅	Proof-of-concept from early development. Move to examples/ or delete.
Strip stale version comments	✅	60+ files have // v0.29.x:, // v0.33.x: pre-fork comments. Single sed pass.
Narrow default bundle	✅	Default: notes, chat, chat-core, mermaid-renderer, schedules. Everything else available via registry or BUNDLED_PACKAGES=*.

v0.6.4 — Admin Health/Metrics Tab + Cluster Merge

Structural move: cluster dashboard becomes an Admin tab. Better home for health/metrics — shared context with other admin panels, no separate nav entry.

Step	Status	Description
"Health / Metrics" admin tab	☐	New tab in Admin surface. DB-agnostic metrics for all deployments. Cluster cards conditional on PG + multi-node detection.
Runtime metrics	☐	Per-node: goroutines, heap alloc/sys, stack in use, GC cycles, last GC pause, GC CPU %, uptime, WebSocket clients, extensions loaded, open FDs.
DB pool metrics	☐	All deployments: DB latency (SELECT 1 round-trip), pool active/idle/max, wait count, wait duration. PG-only: table bloat (n_dead_tup), active backends (pg_stat_activity).
Cluster metrics	☐	PG multi-node only: cluster size, peer list with endpoint + uptime, heartbeat age per node, event bus publish/deliver rates.
Extension runtime metrics	☐	Starlark exec/min, errors/min, avg duration, HTTP outbound requests/min, trigger fires/min, schedule overruns.
Fatten heartbeat payload	☐	Heartbeat JSONB carries full metric set. GET /api/v1/admin/metrics for single-node SQLite fallback (same shape).
Retire cluster-dashboard	☐	Remove package once Admin Health tab ships. Update defaultBundledPackages.
Fix block renderer requires	☐	mermaid-renderer, katex-renderer, csv-table, diff-viewer all have "requires": ["chat"]. These are content renderers, not chat features. Remove constraint — they should activate without chat.
Health endpoint consolidation	☐	/health and /api/v1/health return near-identical JSON. Merge or clearly differentiate with docs.

v0.6.5 — Renderer Pipeline + Docs Rewrite

Most complex sub-version. Lifts block rendering to a kernel SDK primitive so all surfaces share it, then rewrites the docs for an external audience.

Step	Status	Description
SDK renderer primitive	☐	sw.renderers.register(pattern, handler) — kernel-level registration. Extensions call once; all surfaces consume. Decouples renderer discovery from chat surface.
Notes hooks SDK renderer pipeline	☐	Notes renderMarkdown() delegates fenced blocks to registered renderers instead of emitting raw <pre>.
Docs hooks SDK renderer pipeline	☐	Docs markdown renderer delegates fenced blocks to registered renderers.
Unify markdown renderer	☐	Three separate markdown implementations (Notes hand-rolled, Docs hand-rolled, Chat marked). Adopt marked (already vendored) across all three surfaces. Delete hand-rolled duplicates.
Sanitize HTML output	☐	DOMPurify is vendored but unused. Wire it as post-render step for user-generated markdown (Notes). Closes XSS surface.
Mermaid/KaTeX/CSV/Diff work everywhere	☐	With requires: ["chat"] removed (v0.6.4) and renderer pipeline lifted, these render in notes + docs + chat without surface-specific code.
Docs rewrite for external audience	☐	Remove all references to "chat-switchboard", "the fork", "the gut", "Phase 0", "scorched earth". Reframe from "we removed X" to "the kernel provides Y". Remove pre-v0.2.0 version references.
Add Mermaid architecture diagrams	☐	Diagrams in docs: system architecture overview, request flow, extension lifecycle, realtime event flow, settings cascade, cluster topology. Serves double duty: good docs + proof mermaid-renderer works in docs surface.
Surface alias decision	☐	main.go:819–826 has six /admin/surfaces/* alias routes. Decision: keep permanently (document) or migrate SDK + ICD runner to /admin/packages/ and delete. Ship one or the other, not both undocumented.
CONTRIBUTING.md + tutorial	☐	"Build your first extension" guide: manifest → Starlark → ext_data → API route → surface JS → install → test. Use team-activity-log as worked example.

v0.6.6 — Pre-Fork Hardening

Final pass before the hard fork. Security, correctness, and developer experience.

Step	Status	Description
Extension dependency auto-activation	☐	Installing a package with unmet depends/requires should auto-install dependencies from the bundled set, or reject with a clear error listing what's missing. Silent dormant is not acceptable for external users.
ValidateManifest() gate	☐	Single validation function called at install time. Catches malformed manifests early — same pattern as Unicode scan gate.
Package signing hook	☐	Optional signature field in manifest. --verify flag on install path (currently no-op). Five minutes now prevents a breaking manifest schema change post-fork.
OIDC state nonce validation	☐	handlers/auth.go:221 — security TODO. Validate nonce on OIDC callback before accepting tokens.
Schema migration stub decision	☐	handlers/starlark_helpers.go:93–97 — RunSchemaMigrations is a no-op stub called during updates. Implement real migrations or remove and document that only additive schema changes are supported. Don't ship a stub that pretends to work.
ICD/SDK runner update pass	☐	ICD runner and SDK runner are stale for cluster, backup, chat, realtime, dynamic OpenAPI. Update to cover current API surface.
Stale TODO resolution	☐	main.go:867 (session middleware TODO, stale since v0.2.0). auth.go:221 (OIDC nonce, covered above). starlark_helpers.go:96 (migration stub, covered above). Resolve or delete.

Then fork.

---

Post-MVP

• LLM participation (llm-bridge extension: subscribes to chat.message.created, calls provider.complete(), streams response via realtime.publish, posts via chat.send(). Bot participants with persona config. Multi-model conversations.)
• Rich media extensions: image generation, code sandbox, STT/TTS
• Desktop app (Tauri or Electron)
• Sidecar tier: container-based extensions
• Federation: cross-instance package sharing
• Plugin marketplace with signing and review

---

Design Decisions Log

Decision	Rationale
Tasks → extension	Scheduler was the most entangled kernel component (~3,400 lines). Rebuilding as extension validates the trigger system and removes the worst compilation debt. Three trigger primitives (time, webhook, event) replace the monolithic scheduler.
Sessions removed	Channel-based sessions coupled to deleted chat system. Workflow instances need new storage model — either ext_data tables or a dedicated kernel table.
chat_only → custom	Stage mode chat_only implied chat as a kernel concept. Renamed to custom which delegates to a surface package, proving extension composability.
Providers removed from kernel	Provider configs, model catalog, routing policies — all moved to extension track. Kernel provides credential storage (connections) and the Starlark provider.complete module as the interface.
Kernel permissions simplified	From 16 chat-centric permissions to 6 platform permissions. Extensions define their own capability requirements in manifests.
Preact+htm retained	3KB runtime, no build step, works for extension authors without bundler config. KISS.
Single Docker image	Drop the frontend/backend split. Go binary + assets + migrations in one image. Simpler deployment, fewer moving parts.
Admin → RBAC group	The role column is pre-RBAC. v0.2.0 replaces it with a seeded "Admins" group + surface.admin.access grant. All users auto-join "Everyone" group. Admin middleware becomes a grant check, not a role check.
Settings cascade	RBAC controls scope auth (who can set at what level). user_overridable flag controls whether lower scopes can override higher. Two orthogonal axes, composes cleanly with extension manifests.
No new migrations pre-MVP	Edit existing migration SQL files in place. No migration chains until schema is in production.
Notes over Editor	First surface is Obsidian-style notes (rich text, folders, backlinks) instead of a code editor. Notes is a stronger E2E proof — it exercises ext_data, storage, and the SDK more fully than a pure-browser CM6 editor.
No built-in auto-install	Extensions ship in the repo but are not auto-installed. Distribution model TBD — explicit install only. Keeps the kernel clean and avoids opinionated defaults.
Chat as extension, not kernel	Human-to-human chat built entirely as library + surface packages. Zero kernel awareness of conversations, messages, or participants. Kernel gains one generic realtime module. Proves near-infinite extensibility. LLM participation layers on top via a separate bridge extension — the chat system doesn't know or care whether a participant is human or AI.
PG as consensus layer	Horizontal scaling uses PG as the sole coordinator (UNLOGGED node_registry + LISTEN/NOTIFY). No etcd, Consul, Redis, or Raft. Rationale: system is already tightly coupled to PG; adding a second consensus layer doubles operational complexity for zero benefit at homelab-to-small-team scale. UNLOGGED table is visible to all connections, survives session disconnect, and cleans up automatically on PG crash — correct behavior since all nodes are dead anyway. Sweep-all for health (every node deletes stale rows) is simpler than ring topology and has no edge cases.
Two trigger tiers	Event + webhook triggers are extension-declared (manifest contract, full sandbox). Scheduled tasks are user-created ad-hoc (restricted sandbox — no raw HTTP, no DB table creation, connections-only outbound). Separation keeps extension contracts static and user automation safe.
Scheduled task identity	Tasks run as their creator (RBAC-scoped). Admin-created tasks can opt into system context. Creator deactivation pauses the schedule. Ensures audit trail and permission boundaries.
Builtin package rationale	A builtin must enhance kernel surfaces or be required to demonstrate the platform's own capabilities. notes — primary content surface, exercises ext_data/storage/SDK/settings/realtime fully. chat + chat-core — primary communication surface, proves the extensibility thesis (100% extension, zero kernel awareness). mermaid-renderer — docs surface uses Mermaid diagrams to explain the architecture; without it the platform's own documentation doesn't render (self-bootstrapping). schedules — UI for the kernel's scheduled task system; without it users can't manage cron jobs (kernel primitive UI). All others are domain features, examples, dormant, or LLM-only tools — available via registry or BUNDLED_PACKAGES=*.
Cluster dashboard retired	cluster-dashboard shipped as a standalone surface package (v0.6.0) as an expedient. Health/metrics belong inside the Admin surface as a tab — shared context, no separate nav entry, no install required. Merged in v0.6.4.
Block renderers decoupled from chat	mermaid-renderer, katex-renderer, csv-table, diff-viewer shipped with "requires": ["chat"] because renderer discovery lived inside the chat surface. These are content renderers, not chat features. v0.6.4 removes the constraint; v0.6.5 lifts renderer registration to the kernel SDK so all surfaces share it without reimplementing discovery.