core/ROADMAP.md

# Armature — Roadmap

## Current: v0.6.10 — Viewport Foundation

Self-hosted extensible platform. Auth, identity, packages, Starlark sandbox,
storage, realtime, and ops are kernel primitives. Everything else is an extension.

**Kernel capabilities:** Auth (builtin/mTLS/OIDC) · Users/teams/groups/RBAC ·
Surfaces/extensions/libraries/workflows · Starlark sandbox (capability-gated) ·
Object storage (PVC/S3) + ext_data tables · WebSocket hub + realtime pub/sub ·
Audit log · Notifications · Scheduled tasks

**Completed history:** v0.2.x–v0.5.x fully documented in `CHANGELOG.md`.
Highlights: RBAC + settings cascade, event bus + triggers, SDK stabilization,
workflow engine (multi-stage, team roles, signoff gate, public entry, SLA),
package distribution, Notes surface (CM6, folders, tags, backlinks, graph),
realtime primitive, Chat surface (chat-core library + surface + polish),
upgrade test harness, cluster registry + HA.

---

## v0.6.0 — MVP

Extension, communication, and operations tracks converge. First
externally usable release.

Design docs: `docs/DESIGN-cluster-registry.md` — PG-backed cluster registry and self-assembling mesh.

### v0.6.0 — Cluster Registry + HA

PG is the consensus layer. Zero new infrastructure. `UNLOGGED` table + `LISTEN/NOTIFY` replaces etcd/Consul/Redis for homelab-to-small-team scale.

| Step | Status | Description |
|------|--------|-------------|
| `node_registry` table | ✅ | `UNLOGGED TABLE` — node_id, endpoint, seq, registered_at, heartbeat, stats JSONB. Postgres migration 013. |
| Node registration | ✅ | Self-registration on startup: `INSERT ... ON CONFLICT DO UPDATE`. `node_id` = `hostname-PID` or `CLUSTER_NODE_ID` env override. |
| Heartbeat tick | ✅ | Every 10s: update own heartbeat + collect runtime stats (goroutines, heap, GC, uptime, ws_clients). |
| Stale sweep | ✅ | Every heartbeat tick: `DELETE WHERE heartbeat < now() - 30s`. All nodes run it — idempotent, no ring topology. |
| Self-eviction | ✅ | If heartbeat UPDATE returns 0 rows: node was swept by peer → log error + `os.Exit(1)`. K8s restarts → re-register. |
| LISTEN/NOTIFY routing | ✅ | Durable events (messages, state changes) fan-out via `pg_notify`. All replicas receive, push to local WS subscribers, drop if irrelevant. Phase 1: ephemeral events (typing, presence) also via NOTIFY (8KB limit, ~60 bytes each). |
| Cluster API | ✅ | `GET /api/v1/admin/cluster` — returns `{data: [...]}` envelope with all registered nodes. |
| Admin cluster dashboard | ✅ | `cluster-dashboard` surface package renders one card per node: node_id, endpoint, uptime, ws_clients, heap, GC pause. JSONB stats — future keys render automatically, no schema migration. Auto-refresh every 10s. |
| Health endpoint | ✅ | `GET /health` includes `node_id` and `cluster: {size, peers, heartbeat_age_ms}`. |
| Config | ✅ | `CLUSTER_NODE_ID`, `CLUSTER_HEARTBEAT_INTERVAL` (default 10s), `CLUSTER_STALE_THRESHOLD` (default 30s), `CLUSTER_ENDPOINT` (Phase 2 mesh, auto-detect). |
| Single-node regression | ✅ | One-node behavior identical to pre-cluster: one registry row, NOTIFY delivers back to same instance. No special-casing. SQLite returns nil store — all cluster code guarded. |
| Multi-node integration test | ✅ | Docker Compose: 3 instances, shared PG. `ci/e2e-cluster-test.sh`: registration, stale sweep on stop, re-registration on restart. 3 unit tests + 2 handler tests. |

### v0.6.1 — Backup/Restore + Documentation

| Step | Status | Description |
|------|--------|-------------|
| Backup handler | ✅ | `POST /api/v1/admin/backup` streams `.swb` ZIP (JSONL core + ext_data tables + package assets). `POST /api/v1/admin/restore` wipes DB and restores from archive. Dialect-neutral (SQLite + Postgres). |
| Server-side backups | ✅ | `GET /api/v1/admin/backups` list, `GET /download`, `DELETE`. Store backups in `{STORAGE_PATH}/backups/`. |
| Admin backup section | ✅ | New "Backup" section under `/admin/backup`. Create (download or server-side), list, download, delete, restore with destructive confirmation. |
| Documentation API | ✅ | `GET /api/v1/docs` lists, `GET /api/v1/docs/:name` returns raw markdown. Authenticated (not admin-only). |
| Docs surface | ✅ | Builtin surface at `/docs/:section`. Sidebar navigation, client-side markdown renderer. 5 new docs: Getting Started, Extension Guide, API Reference, Deployment, Package Format. |
| Tests | ✅ | 6 handler tests (basic backup, ext_data backup, round-trip restore, schema mismatch, dump/restore table, list empty). E2E script `ci/e2e-backup-test.sh`. |

### v0.6.2 — Docs Polish + Dynamic OpenAPI

| Step | Status | Description |
|------|--------|-------------|
| Dark mode fix | ✅ | Added `--bg-code` to CSS variables (dark + light). Replaced all hardcoded light-mode fallbacks in docs CSS with theme-aware variables. Table styling, code blocks, nav items all readable in dark mode. |
| Loading & error handling | ✅ | Replaced borrowed `settings-placeholder` with docs-specific pulse animation. Added error state with retry button for failed doc list fetch. |
| Topbar navigation | ✅ | Imported `Topbar` + `UserMenu` into docs surface. Users can now navigate to other surfaces via the avatar menu. |
| Docs icon + menu entry | ✅ | Added `📖 Docs` entry to UserMenu standard items. Docs accessible from any surface's user menu. |
| `api_schema` manifest field | ✅ | Optional `api_schema` array in `manifest.json`. Parsed lazily by spec builder. Malformed entries logged and skipped — never blocks extension loading. |
| OpenAPI spec builder | ✅ | `BuildOpenAPISpec()` merges static kernel spec with extension routes. Tier 1: auto-generated stubs for all `api_routes`. Tier 2: `api_schema` replaces stubs with rich path items (params, body, response). |
| Dynamic spec endpoint | ✅ | `GET /api/docs/openapi.json` serves merged spec. Swagger UI updated to use JSON endpoint. Static YAML preserved for backward compat. |
| Tests | ✅ | 7 new handler tests: zero extensions, stubs, rich schema, multi-extension, malformed schema, disabled exclusion, required fields. |

---

## v0.6.x — Hardening

Closes every audit finding before the public release. No new features — only correctness, dead code elimination, and architectural cleanup. Sequence is fixed: each version is a gate for the next.

### v0.6.3 — Dead Code Sweep + Registry Fix

Pure cleanup. No behavior changes except fixing the broken registry install flow.

| Step | Status | Description |
|------|--------|-------------|
| Fix registry install | ✅ | SDK sends `{ url }`, handler expects `{ download_url }`. Fix `api-domains.js` to send `{ download_url: url }`. Every Install click currently returns 400. |
| Registry settings UI | ✅ | Add "Package Registry" section to admin settings with URL input field. Only way to configure registry today is a raw `PUT /api/v1/admin/settings/package_registry` — no user will find it. |
| Registry tooling + docs | ✅ | `scripts/generate-registry.sh` scans a directory of `.pkg` files and emits registry JSON. `docs/PACKAGE-REGISTRY.md` documents the format. |
| Delete dead kernel Go | ✅ | `store/interfaces.go:178–183` — orphaned ChannelListFilter comments. `pages/pages.go:922–930` — `roleFilterType()` + template registration (chat vestige, maps nonexistent roles). `main.go:67` — orphaned provider-type comment. |
| Delete dead vendor JS | ✅ | `vendor/marked.min.js` and `vendor/purify.min.js` — 62KB, zero production imports. Only referenced in test helpers. |
| Delete `dev.html` | ✅ | 676 lines, not imported by anything. Dead. |
| Remove `dashboard` from default bundle | ✅ | Requires `legacy-sdk` (doesn't exist), auto-installs as dormant on fresh installs. Confusing. Remove from `defaultBundledPackages`. |
| Remove `hello-dashboard` | ✅ | Proof-of-concept from early development. Move to `examples/` or delete. |
| Strip stale version comments | ✅ | 60+ files had legacy version annotations (`// v0.29.x:`, `// v0.33.x:`). Single sed pass. |
| Narrow default bundle | ✅ | Default: `notes`, `chat`, `chat-core`, `mermaid-renderer`, `schedules`. Everything else available via registry or `BUNDLED_PACKAGES=*`. |

### v0.6.4 — Admin Health/Metrics Tab + Cluster Merge

Structural move: cluster dashboard becomes an Admin tab. Better home for health/metrics — shared context with other admin panels, no separate nav entry.

| Step | Status | Description |
|------|--------|-------------|
| "Health / Metrics" admin tab | ✅ | New tab in Admin surface. DB-agnostic metrics for all deployments. Cluster cards conditional on PG + multi-node detection. |
| Runtime metrics | ✅ | Per-node: goroutines, heap alloc/sys, stack in use, GC cycles, last GC pause, GC CPU %, uptime, WebSocket clients, extensions loaded, open FDs. |
| DB pool metrics | ✅ | All deployments: DB latency (`SELECT 1` round-trip), pool active/idle/max, wait count, wait duration. PG-only: table bloat (`n_dead_tup`), active backends (`pg_stat_activity`). |
| Cluster metrics | ✅ | PG multi-node only: cluster size, peer list with endpoint + uptime, heartbeat age per node, event bus publish/deliver rates. |
| Extension runtime metrics | ✅ | Starlark exec/min, errors/min, avg duration, HTTP outbound requests/min, trigger fires/min, schedule overruns. |
| Fatten heartbeat payload | ✅ | Heartbeat JSONB carries full metric set. `GET /api/v1/admin/metrics` for single-node SQLite fallback (same shape). |
| Retire `cluster-dashboard` | ✅ | Remove package once Admin Health tab ships. Update `defaultBundledPackages`. |
| Fix block renderer `requires` | ✅ | `mermaid-renderer`, `katex-renderer`, `csv-table`, `diff-viewer` all have `"requires": ["chat"]`. These are content renderers, not chat features. Remove constraint — they should activate without chat. |
| Health endpoint consolidation | ✅ | `/health` and `/api/v1/health` return near-identical JSON. Merge or clearly differentiate with docs. |

### v0.6.5 — Renderer Pipeline + Docs Rewrite

Most complex sub-version. Lifts block rendering to a kernel SDK primitive so all surfaces share it, then rewrites the docs for an external audience.

| Step | Status | Description |
|------|--------|-------------|
| SDK renderer primitive | ✅ | `sw.renderers.register(pattern, handler)` — kernel-level registration. Extensions call once; all surfaces consume. Decouples renderer discovery from chat surface. |
| Notes hooks SDK renderer pipeline | ✅ | Notes delegates to `sw.markdown.renderSync()` + `sw.renderers.runPostRenderers()`. Hand-rolled renderer deleted. |
| Docs hooks SDK renderer pipeline | ✅ | Docs delegates to `sw.markdown.renderSync()` + `sw.renderers.runPostRenderers()`. Hand-rolled renderer deleted (~160 lines). |
| Unify markdown renderer | ✅ | `sw.markdown` module lazy-loads `marked` v16 (vendored). Notes, Docs, Chat all consume it. Two hand-rolled parsers deleted. |
| Sanitize HTML output | ✅ | DOMPurify wired as default post-render step in `sw.markdown`. SVG-safe config allows mermaid output. Notes sanitizes; Docs opts out (system-authored). |
| Mermaid/KaTeX/CSV/Diff work everywhere | ✅ | Browser extension loader injects renderer scripts into all pages. Extensions register via `sw:ready` event. All four render in Notes, Docs, and Chat. |
| Docs rewrite for external audience | ✅ | All fork references removed from docs, CHANGELOG, ROADMAP. DESIGN-WORKFLOW-REDESIGN-0.2.6.md replaced with DESIGN-WORKFLOWS.md. |
| Add Mermaid architecture diagrams | ✅ | Six diagrams in ARCHITECTURE.md: system overview, request flow, extension lifecycle, realtime events, settings cascade, cluster topology. |
| Surface alias decision | ✅ | Migrated SDK + ICD runner to `/admin/packages/`; aliases removed from main.go. |
| `CONTRIBUTING.md` + tutorial | ✅ | CONTRIBUTING.md at repo root + docs/TUTORIAL-FIRST-EXTENSION.md walkthrough. |

### v0.6.7 — Native mTLS

End-to-end mutual TLS without a reverse proxy. Targets systemd+podman deployments where every connection (client→server, node→node) is mTLS. Design: `docs/DESIGN-native-mtls.md`.

| Step | Status | Description |
|------|--------|-------------|
| `TLS_MODE` config | ✅ | Three values: `none` (default, plain HTTP) · `server` (TLS, no client cert) · `mtls` (mutual TLS, client cert required). Independent of `AUTH_MODE`. |
| TLS server mode | ✅ | Go binary calls `ListenAndServeTLS` directly. `TLS_CERT`, `TLS_KEY`, `TLS_CA` path config. TLS 1.3 minimum, no fallback. `server/config/tls.go` loader + validation. |
| `MTLSNativeProvider` | ✅ | Reads `r.TLS.PeerCertificates[0]` — no header trust. `Subject.CommonName` → username. `sha256(Raw)` → `external_id`. Auto-provisions `auth_source=mtls`. Renamed existing `mtls.go` → `mtls_proxy.go`. Shared helpers in `mtls_helpers.go`. |
| Node-to-node mTLS | ✅ | `BuildPeerTLSConfig()` constructs outbound TLS config with node cert + CA pool. Forward-looking — cluster registry is DB-backed (PG LISTEN/NOTIFY), no HTTP peer calls yet. |
| `armature-ca.sh` | ✅ | Shell wrapper around openssl. Three commands: `init` (CA keypair) · `issue-node --name <n> --san <addrs>` (365d) · `issue-user --cn <name>` (90d). All output PEM. |
| Unit tests | ✅ | Ephemeral CA via `crypto/x509`. Fabricated `PeerCertificates`. Valid cert → user provisioned · no TLS → `ErrNoCert` · empty certs → `ErrNoCert` · no CN → `ErrInvalidCreds`. |
| Integration tests | ✅ | Real TLS listener on localhost. No cert / wrong CA / expired → TLS handshake rejected. Valid cert → 200. Peer certificate CN + email visibility verified. |

### v0.6.6 — Final Hardening

Final pass before public release. Security, correctness, and developer experience.

| Step | Status | Description |
|------|--------|-------------|
| Extension dependency auto-activation | ✅ | Installing a package with unmet `depends`/`requires` auto-installs dependencies from the bundled set. If not bundled: clear error listing what's missing. |
| `ValidateManifest()` gate | ✅ | Single `ValidateManifest()` function in `package_validate.go`. Called at install time (both upload and bundled). 12 unit tests. |
| Package signing hook | ✅ | Optional `signature` field in manifest (reserved). `PACKAGE_VERIFY_SIGNATURES` env var (default false, log-only). No cryptographic code yet — schema slot reserved. |
| OIDC state nonce validation | ✅ | `oidcClaims.Nonce` field added. `ValidateIDTokenNonce()` compares ID token nonce against stored state. Callback rejects mismatched nonces. |
| Schema migration stub decision | ✅ | Stub replaced with log-only function documenting additive-only policy. Downgrade rejection preserved. |
| ICD/SDK runner update pass | ✅ | ICD smoke tier: added metrics, cluster, backups, docs, OpenAPI JSON endpoints. SDK admin domain: added metrics, cluster, backups tests. |
| Stale TODO resolution | ✅ | `main.go` session middleware: replaced with `OptionalAuth()` (auth if token present, anonymous pass-through). `auth.go:221` OIDC nonce: resolved. `starlark_helpers.go:96` migration stub: resolved. |

Then ship.

---

## Post-MVP

- LLM participation (`llm-bridge` extension: subscribes to `chat.message.created`, calls `provider.complete()`, streams response via `realtime.publish`, posts via `chat.send()`. Bot participants with persona config. Multi-model conversations.)
- Rich media extensions: image generation, code sandbox, STT/TTS
- Desktop app (Tauri or Electron)
- Sidecar tier: container-based extensions
- Federation: cross-instance package sharing
- Plugin marketplace with signing and review

---

## Design Decisions Log

| Decision | Rationale |
|----------|-----------|
| Tasks → extension | Scheduler was the most entangled kernel component (~3,400 lines). Rebuilding as extension validates the trigger system and removes the worst compilation debt. Three trigger primitives (time, webhook, event) replace the monolithic scheduler. |
| Sessions removed | Kernel-managed sessions replaced by workflow instances with dedicated storage (ext_data tables or kernel table). |
| `custom` stage mode | Stage mode `custom` delegates to a surface package, proving extension composability. Chat-in-workflow is handled by the chat extension, not the kernel. |
| Providers removed from kernel | Provider configs, model catalog, routing policies — all moved to extension track. Kernel provides credential storage (connections) and the Starlark `provider.complete` module as the interface. |
| Kernel permissions simplified | 6 platform permissions. Extensions define their own capability requirements in manifests. |
| Preact+htm retained | 3KB runtime, no build step, works for extension authors without bundler config. KISS. |
| Single Docker image | Drop the frontend/backend split. Go binary + assets + migrations in one image. Simpler deployment, fewer moving parts. |
| Admin → RBAC group | The `role` column is pre-RBAC. v0.2.0 replaces it with a seeded "Admins" group + `surface.admin.access` grant. All users auto-join "Everyone" group. Admin middleware becomes a grant check, not a role check. |
| Settings cascade | RBAC controls scope auth (who can set at what level). `user_overridable` flag controls whether lower scopes can override higher. Two orthogonal axes, composes cleanly with extension manifests. |
| No new migrations pre-MVP | Edit existing migration SQL files in place. No migration chains until schema is in production. |
| Notes over Editor | First surface is Obsidian-style notes (rich text, folders, backlinks) instead of a code editor. Notes is a stronger E2E proof — it exercises ext_data, storage, and the SDK more fully than a pure-browser CM6 editor. |
| No built-in auto-install | Extensions ship in the repo but are not auto-installed. Distribution model TBD — explicit install only. Keeps the kernel clean and avoids opinionated defaults. |
| Chat as extension, not kernel | Human-to-human chat built entirely as library + surface packages. Zero kernel awareness of conversations, messages, or participants. Kernel gains one generic `realtime` module. Proves near-infinite extensibility. LLM participation layers on top via a separate bridge extension — the chat system doesn't know or care whether a participant is human or AI. |
| PG as consensus layer | Horizontal scaling uses PG as the sole coordinator (UNLOGGED node_registry + LISTEN/NOTIFY). No etcd, Consul, Redis, or Raft. Rationale: system is already tightly coupled to PG; adding a second consensus layer doubles operational complexity for zero benefit at homelab-to-small-team scale. UNLOGGED table is visible to all connections, survives session disconnect, and cleans up automatically on PG crash — correct behavior since all nodes are dead anyway. Sweep-all for health (every node deletes stale rows) is simpler than ring topology and has no edge cases. |
| Two trigger tiers | Event + webhook triggers are extension-declared (manifest contract, full sandbox). Scheduled tasks are user-created ad-hoc (restricted sandbox — no raw HTTP, no DB table creation, connections-only outbound). Separation keeps extension contracts static and user automation safe. |
| Scheduled task identity | Tasks run as their creator (RBAC-scoped). Admin-created tasks can opt into system context. Creator deactivation pauses the schedule. Ensures audit trail and permission boundaries. |
| Builtin package rationale | A builtin must enhance kernel surfaces or be required to demonstrate the platform's own capabilities. **notes** — primary content surface, exercises ext_data/storage/SDK/settings/realtime fully. **chat + chat-core** — primary communication surface, proves the extensibility thesis (100% extension, zero kernel awareness). **mermaid-renderer** — docs surface uses Mermaid diagrams to explain the architecture; without it the platform's own documentation doesn't render (self-bootstrapping). **schedules** — UI for the kernel's scheduled task system; without it users can't manage cron jobs (kernel primitive UI). All others are domain features, examples, dormant, or LLM-only tools — available via registry or `BUNDLED_PACKAGES=*`. |
| Cluster dashboard retired | `cluster-dashboard` shipped as a standalone surface package (v0.6.0) as an expedient. Health/metrics belong inside the Admin surface as a tab — shared context, no separate nav entry, no install required. Merged in v0.6.4. |
| Block renderers decoupled from chat | `mermaid-renderer`, `katex-renderer`, `csv-table`, `diff-viewer` shipped with `"requires": ["chat"]` because renderer discovery lived inside the chat surface. These are content renderers, not chat features. v0.6.4 removes the constraint; v0.6.5 lifts renderer registration to the kernel SDK so all surfaces share it without reimplementing discovery. |