switchboard/core
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Releases 9 Wiki Activity

step 5 (complete): docs purge, fresh ARCHITECTURE + ROADMAP + CHANGELOG

Browse Source 
Purged 29,966 lines of stale chat-switchboard documentation:
  - 41 docs/ files (ICD specs, design docs, archive, workflow docs)
  - 5 root MD files (CHANGESET, TURNOVER, ICD-DRIFT-AUDIT, etc.)

New documentation:
  - docs/ARCHITECTURE.md — kernel components, design principles, data layer
  - ROADMAP.md — v0.1.0 through v0.5.0 MVP with decision log
  - CHANGELOG.md — fresh, starting from v0.1.0 fork
  - README.md — rewritten for switchboard-core

Also in this commit:
  - config.go: stripped 7 dropped fields, DB default → switchboard_core
  - pages/loaders.go: stripped provider/model/notes/projects loaders
  - pages/pages.go: stripped persona store lookup
  - handlers/workflows.go: stripped persona tool grants
  - main.go: stripped team provider routes, avatar routes

Production code: zero references to deleted packages, stores, or models.
-29,966/+352 lines across 73 files.
This commit is contained in:
Jeffrey Smith
2026-03-26 05:10:40 -04:00
parent e4b7ee98a5
commit 7b6e54d5b7
73 changed files with 349 additions and 29963 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5360
CHANGELOG.md
View File

File diff suppressed because it is too large Load Diff

395
CHANGESET.md
View File

@@ -1,395 +0,0 @@
# CS-0.37.1 — Permission Audit & Enforcement
**Date:** 2026-03-20
**Scope:** Backend permission enforcement + new self-service permissions endpoint
**CI Impact:** Adds new test file with 12 test functions. Existing tests unaffected.
---
## Summary
### Problem
- 8 protected routes enforced only `Auth()` (logged-in check) but not
`RequirePermission(perm)`. Any authenticated user could call them
regardless of group permissions.
- No user-facing permissions endpoint — the frontend has zero ability to
know what the user is allowed to do.
- The Everyone group seeds with `model.use`, `kb.read`, `channel.create`,
masking the enforcement gaps in most deployments.
- All existing task tests run as admin, which bypasses RequirePermission
entirely — permission denial was never tested.
### Fix
1. Add `RequirePermission` middleware to 8 routes in `main.go`
2. Mirror the same 8 fixes in the integration test harness
3. Add `GET /profile/permissions` endpoint (new handler file)
4. Add `GET /profile/permissions` to route registration + test harness
5. Add 12 permission enforcement integration tests (deny → grant → allow)
### New Files (drop-in)
- `server/handlers/profile_permissions.go`
- `server/handlers/perm_enforcement_test.go`
### Edited Files (str_replace patches below)
- `server/main.go` — 9 route edits
- `server/handlers/integration_test.go` — 9 route edits (mirror)
- `docs/ICD/profile.md` — add new endpoint documentation
---
## Patch Instructions
Apply each str_replace in order. Each section shows the exact old string
and the exact replacement.
### 1. main.go — Completions (model.use)
**Old:**
```
protected.POST("/chat/completions", comp.Complete)
```
**New:**
```
protected.POST("/chat/completions", middleware.RequirePermission(auth.PermModelUse, stores), comp.Complete)
```
### 2. main.go — Summarize (model.use)
**Old:**
```
protected.POST("/channels/:id/summarize", summarize.Summarize)
```
**New:**
```
protected.POST("/channels/:id/summarize", middleware.RequirePermission(auth.PermModelUse, stores), summarize.Summarize)
```
### 3. main.go — Generate Title (model.use)
**Old:**
```
protected.POST("/channels/:id/generate-title", titleH.GenerateTitle)
```
**New:**
```
protected.POST("/channels/:id/generate-title", middleware.RequirePermission(auth.PermModelUse, stores), titleH.GenerateTitle)
```
### 4. main.go — KB Search (kb.read)
**Old:**
```
protected.POST("/knowledge-bases/:id/search", kbH.SearchKB)
```
**New:**
```
protected.POST("/knowledge-bases/:id/search", middleware.RequirePermission(auth.PermKBRead, stores), kbH.SearchKB)
```
### 5. main.go — KB Update (kb.write)
**Old:**
```
protected.PUT("/knowledge-bases/:id", kbH.UpdateKB)
```
**New:**
```
protected.PUT("/knowledge-bases/:id", middleware.RequirePermission(auth.PermKBWrite, stores), kbH.UpdateKB)
```
### 6. main.go — KB Delete (kb.write)
**Old:**
```
protected.DELETE("/knowledge-bases/:id", kbH.DeleteKB)
```
**New:**
```
protected.DELETE("/knowledge-bases/:id", middleware.RequirePermission(auth.PermKBWrite, stores), kbH.DeleteKB)
```
### 7. main.go — KB Document Delete (kb.write)
**Old:**
```
protected.DELETE("/knowledge-bases/:id/documents/:docId", kbH.DeleteDocument)
```
**New:**
```
protected.DELETE("/knowledge-bases/:id/documents/:docId", middleware.RequirePermission(auth.PermKBWrite, stores), kbH.DeleteDocument)
```
### 8. main.go — KB Rebuild (kb.write)
**Old:**
```
protected.POST("/knowledge-bases/:id/rebuild", kbH.RebuildKB)
```
**New:**
```
protected.POST("/knowledge-bases/:id/rebuild", middleware.RequirePermission(auth.PermKBWrite, stores), kbH.RebuildKB)
```
### 9. main.go — New route: GET /profile/permissions
Insert AFTER the existing profile routes block. Find:
```
protected.GET("/settings", settings.GetSettings)
protected.PUT("/settings", settings.UpdateSettings)
```
Insert immediately after:
```
// Permission bootstrap (v0.37.1) — self-service resolved permissions
permH := handlers.NewProfilePermissionsHandler(stores)
protected.GET("/profile/permissions", permH.GetMyPermissions)
```
---
### 10. integration_test.go — Mirror: Completions (model.use)
**Old:**
```
protected.POST("/chat/completions", completions.Complete)
```
**New:**
```
protected.POST("/chat/completions", middleware.RequirePermission(authpkg.PermModelUse, stores), completions.Complete)
```
### 11. integration_test.go — Mirror: KB Search (kb.read)
**Old:**
```
protected.POST("/knowledge-bases/:id/search", kbH.SearchKB)
```
**New:**
```
protected.POST("/knowledge-bases/:id/search", middleware.RequirePermission(authpkg.PermKBRead, stores), kbH.SearchKB)
```
### 12. integration_test.go — Mirror: KB Update (kb.write)
**Old:**
```
protected.PUT("/knowledge-bases/:id", kbH.UpdateKB)
```
**New:**
```
protected.PUT("/knowledge-bases/:id", middleware.RequirePermission(authpkg.PermKBWrite, stores), kbH.UpdateKB)
```
### 13. integration_test.go — Mirror: KB Delete (kb.write)
**Old:**
```
protected.DELETE("/knowledge-bases/:id", kbH.DeleteKB)
```
**New:**
```
protected.DELETE("/knowledge-bases/:id", middleware.RequirePermission(authpkg.PermKBWrite, stores), kbH.DeleteKB)
```
### 14. integration_test.go — Mirror: KB Doc Delete (kb.write)
**Old:**
```
protected.DELETE("/knowledge-bases/:id/documents/:docId", kbH.DeleteDocument)
```
**New:**
```
protected.DELETE("/knowledge-bases/:id/documents/:docId", middleware.RequirePermission(authpkg.PermKBWrite, stores), kbH.DeleteDocument)
```
### 15. integration_test.go — Mirror: KB Rebuild (kb.write)
**Old:**
```
protected.POST("/knowledge-bases/:id/rebuild", kbH.RebuildKB)
```
**New:**
```
protected.POST("/knowledge-bases/:id/rebuild", middleware.RequirePermission(authpkg.PermKBWrite, stores), kbH.RebuildKB)
```
### 16. integration_test.go — New route: GET /profile/permissions
Find this block in setupHarness:
```
// Settings / Profile
settings := NewSettingsHandler(stores, nil)
```
Insert BEFORE it:
```
// Profile permissions (v0.37.1)
permH := NewProfilePermissionsHandler(stores)
protected.GET("/profile/permissions", permH.GetMyPermissions)
```
### 17. Existing KB Permission Test Update
The existing `TestIntegration_KB_PermissionEnforcement` test (line ~2808)
tests kb.create denial. It still passes because it relies on TruncateAll
wiping the Everyone group. No changes needed.
However, note that this test's comment says:
```
// (TruncateAll wipes the Everyone group, so no default permissions)
```
This is the only reason it works. On a live system with an intact Everyone
group seeded with `kb.read`, the denial would not trigger for kb.read
routes. The new `perm_enforcement_test.go` tests do NOT rely on Everyone
group presence/absence — they work from a clean slate.
---
## ICD Update
Add to `docs/ICD/profile.md` before the closing `---`:
```markdown
### Get My Permissions
\```
GET /profile/permissions
\```
**Auth:** Authenticated user
Returns the current user's resolved permission set, contributing group
IDs, team memberships, and UI-relevant policies. Admin users receive all
permissions.
**Response:**
\```json
{
"permissions": ["model.use", "kb.read", "channel.create"],
"groups": ["group-id-1", "00000000-0000-0000-0000-000000000001"],
"teams": [
{ "id": "uuid", "name": "Engineering", "my_role": "member" }
],
"policies": {
"allow_user_byok": false,
"allow_user_personas": false,
"allow_raw_model_access": true,
"kb_direct_access": true
}
}
\```
| Field | Type | Description |
|-------|------|-------------|
| `permissions` | string[] | Resolved permission set (union of all groups) |
| `groups` | string[] | Contributing group IDs (always includes Everyone) |
| `teams` | object[] | User's active team memberships with role |
| `policies` | object | Boolean policy flags relevant to UI feature gating |
```
---
## Audit Summary
### Routes Now Enforcing RequirePermission
| Route | Permission | Was | Now |
|-------|-----------|-----|-----|
| `POST /chat/completions` | `model.use` | Auth() only | RequirePermission |
| `POST /channels/:id/summarize` | `model.use` | Auth() only | RequirePermission |
| `POST /channels/:id/generate-title` | `model.use` | Auth() only | RequirePermission |
| `POST /knowledge-bases/:id/search` | `kb.read` | Auth() only | RequirePermission |
| `PUT /knowledge-bases/:id` | `kb.write` | Auth() only | RequirePermission |
| `DELETE /knowledge-bases/:id` | `kb.write` | Auth() only | RequirePermission |
| `DELETE /knowledge-bases/:id/documents/:docId` | `kb.write` | Auth() only | RequirePermission |
| `POST /knowledge-bases/:id/rebuild` | `kb.write` | Auth() only | RequirePermission |
### Routes Already Correctly Enforced (no changes)
| Route | Permission |
|-------|-----------|
| `POST /channels` | `channel.create` |
| `POST /channels/:id/participants` | `channel.invite` |
| `POST /personas` | `persona.create` |
| `PUT /personas/:id` | `persona.manage` |
| `DELETE /personas/:id` | `persona.manage` |
| `POST /knowledge-bases` | `kb.create` |
| `POST /knowledge-bases/:id/documents` | `kb.write` |
| `POST /workflows` | `workflow.create` |
| `PATCH /workflows/:id` | `workflow.create` |
| `DELETE /workflows/:id` | `workflow.create` |
| All workflow stage routes | `workflow.create` |
| `POST /tasks` | `task.create` |
| `PUT /tasks/:id` | `task.create` |
| `DELETE /tasks/:id` | `task.create` |
| `POST /tasks/:id/run` | `task.create` |
| `POST /tasks/:id/kill` | `task.create` |
| Team tasks (scoped) | `task.create` |
| All admin routes | `RequireAdmin()` |
| Team-scoped routes | `RequireTeamAdmin()` |
### Routes Correctly Using Auth() Only (no permission needed)
Read-only data for the authenticated user: channels list, messages, notes,
projects, profile, settings, models/enabled, notifications, extensions,
groups/mine, teams/mine, presence, folders, workspaces, files, memories,
packages, surfaces, usage, search, export/me.
### Everyone Group Default Permissions
The migration seeds Everyone with: `model.use`, `kb.read`, `channel.create`.
These three permissions mean that on a default deployment, the new
enforcement on completions (`model.use`) and KB search (`kb.read`) will
not change behavior — users already have these via Everyone.
The enforcement matters when an admin removes permissions from Everyone
or creates restricted groups. Without these route-level gates, removing
`model.use` from Everyone did nothing — users could still call completions.
### New Tests
12 test functions in `perm_enforcement_test.go`:
| Test | Permission | Pattern |
|------|-----------|---------|
| `TestPermEnforce_ModelUse_Completions` | `model.use` | deny → grant → allow |
| `TestPermEnforce_ChannelCreate` | `channel.create` | deny → grant → allow |
| `TestPermEnforce_ChannelInvite` | `channel.invite` | deny → grant → allow |
| `TestPermEnforce_KBCreate` | `kb.create` | deny → grant → allow |
| `TestPermEnforce_KBSearch` | `kb.read` | deny → grant → allow |
| `TestPermEnforce_KBWrite` | `kb.write` | deny → grant → allow |
| `TestPermEnforce_PersonaCreate` | `persona.create` | deny → grant → allow |
| `TestPermEnforce_PersonaManage` | `persona.manage` | deny → grant → allow |
| `TestPermEnforce_WorkflowCreate` | `workflow.create` | deny → grant → allow |
| `TestPermEnforce_TaskCreate` | `task.create` | deny → grant → allow |
| `TestPermEnforce_ProfilePermissions_NoGroups` | — | empty perms |
| `TestPermEnforce_ProfilePermissions_WithGrant` | — | granted perms appear |
| `TestPermEnforce_ProfilePermissions_Admin` | — | admin gets all |
### Not Covered (test harness limitation)
Summarize and GenerateTitle routes are not in the integration test harness
(`setupHarness`). They share `model.use` with completions, which IS tested.
Route-level wiring is applied in main.go. Full coverage comes when the
test harness is extended to include these routes.

237
FE-REWRITE-REVIEW-0.37.14.md
View File

@@ -1,237 +0,0 @@
# FE Rewrite Review — CS-0.37.14
**Reviewer:** Claude (Opus 4.6)
**Date:** 2026-03-23
**Scope:** Full critical review of the Preact+htm frontend rewrite (src/js/sw/) and supporting backend changes (CS-0.37.1 permission enforcement).
**Stats:** ~16,200 lines of new JS across 115 files, ~6,800 lines of CSS across 17 files. Backend: 2 new handler files, 9 route edits, 12 new integration tests.
---
## Severity Legend
- **P0** — Breaks functionality or security in a live deployment. Fix before merging.
- **P1** — Significant regression or correctness issue. Fix before tagging.
- **P2** — Design debt or missing wiring. Track for next changeset.
- **P3** — Cleanup, dead code, cosmetic. Fix opportunistically.
---
## P0 — Fix Before Merge
### 1. Pipe/filter pipeline is wired but never invoked
`sw.pipe` is fully built (pre/stream/render stages, scoping, stats, dedup detection) but no component calls `_runPre`, `_runStream`, or `_runRender`. The chat completion flow in `use-chat.js` calls `sw.api.channels.complete()` directly. `use-stream.js` parses SSE without running stream filters. Markdown rendering skips render filters entirely.
**Impact:** Extensions that register pipe filters are silently ignored. The v0.30.0 compaction-as-filter roadmap item cannot work. This is a hard regression from the old FE where the pipe was invoked during completion.
**Fix:** Wire `sw.pipe._runPre(ctx)` before the completion call in `use-chat.sendMessage()`, `sw.pipe._runStream(ctx)` in the SSE loop in `use-stream.js`, and `sw.pipe._runRender(ctx)` in the markdown rendering path. The calling conventions match the old SDK — the hooks just need to invoke them at the right points.
**Files:** `src/js/sw/components/chat-pane/use-chat.js`, `src/js/sw/components/chat-pane/use-stream.js`, `src/js/sw/components/chat-pane/markdown.js`
### 2. User menu RBAC gating is inverted
```js
if ((!authenticated || sw?.isAdmin) && current !== 'admin') {
list.push({ label: 'Admin', ... });
}
```
The `!authenticated` branch means unauthenticated users see Admin, Team Admin, and Debug menu items. The same inverted pattern appears on all three privileged items.
**Impact:** Security-facing. Non-admin users see (and can navigate to) admin pages. The backend still enforces `RequireAdmin()` on the route, so it's defense-in-depth, but the menu exposes attack surface.
**Fix:** Change `!authenticated || sw?.isAdmin` to `sw?.isAdmin` (and similarly for Team Admin/Debug). Remove the `!authenticated` fallback — it was clearly a dev-time bypass.
**File:** `src/js/sw/shell/user-menu.js`, lines 86–100
### 3. `_unwrap` misses bare `{data:[...]}` envelopes
The REST client strips the `data` array only when pagination fields (`page`, `total`, `per_page`) are present:
```js
if (json && Array.isArray(json.data) && ('page' in json || 'total' in json || 'per_page' in json)) {
return json.data;
}
```
Multiple backend endpoints return `{"data": [...]}` without pagination fields: notes backlinks, notes titles, notes graph, notes search results, channel KBs, and others. These pass through as `{data: [...]}` objects. **24 call sites** defensively work around this with `resp.data || resp || []`.
**Impact:** The SDK's stated contract — "List methods always return the unwrapped array" — is silently broken. Every consumer must know which endpoints include pagination and which don't. New code will get this wrong.
**Fix:** Change the condition to `json && Array.isArray(json.data)`. This matches the response envelope convention (`{"data": []}` for all lists). Single-object responses don't have a `data` array, so they pass through correctly.
**File:** `src/js/sw/sdk/rest-client.js`, `_unwrap` function
---
## P1 — Fix Before Tag
### 4. `deleteMessage` calls a phantom API
`use-chat.js` calls `sw.api.channels.deleteMessage(...)` which doesn't exist in `api-domains.js` and has no corresponding backend endpoint in the ICD. The `?.` guard makes it a no-op, but the optimistic UI remove (`setMessages(prev => prev.filter(...))`) means messages disappear from the UI until page refresh, then reappear.
**Impact:** Users think they deleted a message; they didn't. Data inconsistency.
**Fix:** Either add the backend endpoint + SDK method, or remove the optimistic UI remove and surface a "not supported" toast.
**Files:** `src/js/sw/components/chat-pane/use-chat.js` (deleteMessage), `src/js/sw/sdk/api-domains.js` (missing method)
### 5. Menu submenu positioning is wrong
The recursive `<Menu>` for submenus uses `anchor=${menuRef.current}` — the parent menu's root div. The submenu positions relative to the entire parent menu element, not the individual item that triggered it.
**Impact:** Submenus appear at the parent menu's top-left corner instead of adjacent to the hovered row.
**Fix:** Track a ref per menu item (or pass the item's DOM element as anchor) so the submenu positions relative to its trigger row.
**File:** `src/js/sw/primitives/menu.js`, submenu rendering block
### 6. ChatDashboard recent items are not clickable
The recent channel list in the empty-state dashboard renders items without an `onClick` handler. Users can see their recent chats but cannot click to navigate to them.
**Impact:** UX dead end. The dashboard shows data it doesn't act on.
**Fix:** Add `onClick=${() => onSelect(ch.id, ch.type)}` to the recent item divs.
**File:** `src/js/sw/surfaces/chat/chat-workspace.js`, `ChatDashboard` component
### 7. Version string mismatch
Three different version strings in the same changeset:
- `sw._sdk = '0.37.14'` (sdk/index.js)
- `console.log('[sw] SDK v0.37.9 ready')` (sdk/index.js)
- `VERSION` file: `0.37.14.14` (four segments, breaks `0.<major>.<minor>` convention)
**Fix:** Unify to one version string, probably `0.37.14`. Fix the console log. Decide whether the four-segment VERSION is intentional.
**Files:** `src/js/sw/sdk/index.js`, `VERSION`
### 8. New FE code has zero CI coverage
The CI syntax check globs `src/js/*.js` (top-level only), missing all 115 files under `src/js/sw/`. Additionally, `node --check` doesn't work on ES modules with `import` statements. The existing `__tests__` test against old globals, not the new Preact stack.
**Impact:** Syntax errors, import typos, or broken modules in the new code won't be caught until runtime.
**Fix:** Add a CI step that either validates ES module syntax (e.g., a lightweight bundler dry-run or a glob that covers `src/js/sw/**/*.js`) or adds basic smoke tests for the new SDK boot path.
**File:** `.gitea/workflows/ci.yaml`, `test-frontend` job
---
## P2 — Track for Next Changeset
### 9. `sw.can()` is essentially unused — **RESOLVED v0.37.19**
~~Only 1 reference in all surface code (`team-admin/index.js` uses `sw.isAdmin`, not `sw.can()`). No surface gates create/edit/delete buttons with `sw.can('kb.write')`, `sw.can('channel.create')`, etc. The backend permission enforcement from CS-0.37.1 has no client-side counterpart.~~
The settings surface does its own policy resolution from `__PAGE_DATA__` + a separate API fallback, rather than reading `sw.auth.policies` which is already cached from `/profile/permissions`.
**Impact:** The entire client-side RBAC story — the motivating feature for this rewrite — is infrastructure without consumers. Users will see buttons they can't use; the backend returns 403s but the UI doesn't prevent the click.
**Recommendation:** A follow-up pass through all surfaces to add `sw.can()` gates on action buttons, and migrate settings to use `sw.auth.policies`.
### 10. Pipe `_runChain` has no Promise guard
The comment says "Sync-only — filters must not return Promises" but there's no guard. If a filter returns a Promise (which is truthy), it silently replaces `ctx` with the Promise object, corrupting the chain for all downstream filters.
**Fix:** Add `if (result instanceof Promise) { console.error(...); continue; }` after the filter call.
**File:** `src/js/sw/sdk/pipe.js`, `_runChain`
### 11. ChatPane handleRef `useEffect` runs every render
The `useEffect` that sets `handleRef.current = {...}` has no dependency array, rebuilding the imperative handle object on every render cycle.
**Fix:** Add `[]` or `[handleRef]` as dependency array.
**File:** `src/js/sw/components/chat-pane/index.js`, line 66
### 12. `__USER__` / `__PAGE_DATA__` injection is mostly dead — **RESOLVED v0.37.19**
~~The Go template still injects `window.__USER__` and `window.__PAGE_DATA__` on every page load. The new SDK boots from API calls. Only `settings/index.js` reads `__PAGE_DATA__` (for BYOK/persona policies) and it already has an API fallback. `window.__USER__` is never read by new code.~~
~~**Recommendation:** Remove `__USER__` injection. Migrate settings to use `sw.auth.policies` (see #9), then remove `__PAGE_DATA__` injection.~~
### 13. Debug modal is old-world code
The debug modal in `base.html` (~50 lines of raw HTML with inline `onclick` handlers, `getElementById`, global functions like `switchDebugTab()`) is the largest remaining "no raw DOM" violation and is shared across all surfaces.
**Recommendation:** Rebuild as a Preact component in a follow-up, or at minimum flag it as intentional legacy.
---
## P3 — Cleanup
### 14. Dual toast container
`base.html` still has `<div id="toastContainer" class="toast-container"></div>` from the old world. Every surface also renders `<ToastContainer />` (Preact). The old div is orphaned dead DOM.
**Fix:** Remove the `#toastContainer` div from `base.html`.
### 15. Dead code: `esc()` in admin/users.js
`function esc(s) {...}` is defined but never called. Leftover from a template that got Preact-ified.
**Fix:** Delete it.
**File:** `src/js/sw/surfaces/admin/users.js`, line 7
### 16. `sidebar-chats.js` reaches into `#surfaceInner` — **RESOLVED v0.37.19**
~~`_getScale()` reads `document.getElementById('surfaceInner')` to compensate for CSS zoom on `position: fixed` menus. Technically correct but creates an implicit coupling between a surface component and the shell's DOM structure.~~
~~**Recommendation:** Move scale detection into the SDK (e.g., `sw.shell.getScale()`) so the coupling is explicit.~~
### 17. Notes markdown rendering uses raw DOM
`markdown.js` uses `innerHTML`, `querySelector`, `querySelectorAll` — inherent to the marked+DOMPurify pipeline. `attachWikilinkHandlers` adds click listeners imperatively on every render without explicit cleanup (Preact's DOM replacement handles GC, but the pattern is fragile).
**Recommendation:** Document this as an intentional exception to the "no raw DOM" principle. Consider a wrapper component that manages the imperative lifecycle.
### 18. Double-unwrap defensive pattern ~~(24 instances)~~ — **RESOLVED v0.37.14.21**
~~Once P0 #3 is fixed (`_unwrap` condition change), all 24 instances of `resp.data || resp || []` become dead fallback code.~~
**Resolved:** Actually 93 instances (not 24). Fixed `_unwrap` to preserve sibling fields (`Object.keys` check), normalized all 18 non-standard Go handler envelopes to `{data: [...]}`, and simplified all frontend call sites. Also fixed 3 latent bugs (roles data lost, provider types empty, audit entries empty). See CHANGELOG v0.37.14.21.
### 19. Keyboard shortcuts in use-notes.js use DOM presence check
`if (document.querySelector('.sw-notes-pane'))` gates Ctrl+N and Ctrl+D shortcuts. This means the shortcuts fire whenever the notes pane exists anywhere in the DOM, not when it's focused. If notes and chat are ever composed on the same page, shortcuts will conflict.
**Recommendation:** Use a focus-based check or a "surface active" flag from the SDK.
---
## Positives
### Architecture
The layer separation is clean and consistently applied. Primitives have zero business logic. The SDK has zero DOM. Shell has zero API calls. Surfaces compose from both. This matches the design doc precisely. The factory pattern for all SDK modules avoids singleton coupling — every module is testable in isolation.
### SDK internals
The auth module handles token persistence with cookie sync for Go SSR, proactive refresh at 80% expiry, coalesced refresh promises (preventing thundering herd on concurrent 401s), and a clean boot sequence with offline tolerance. The event bus correctly classifies local-only events to prevent UI concerns from crossing the WebSocket. The CRUD factory in `api-domains.js` eliminates boilerplate across 18 domain namespaces.
### Performance
The `use-stream.js` hook uses rAF-batched state updates during SSE parsing. Content accumulates in refs and flushes at display frame rate, preventing re-render storms during fast token output. This is the correct pattern for streaming LLM output into a reactive UI.
### Developer experience
The crash catcher in surface templates (`window.addEventListener('error', ...)` with a visible banner) is a smart production debugging aid for a scorched-earth rewrite. The admin surface's lazy section loading with version-busted dynamic imports means sections load on demand and cache-bust on upgrade. The `sw.chatPane()` and `sw.notesPane()` imperative render helpers let extension surfaces mount chat/notes without knowing Preact internals.
### Backend (CS-0.37.1)
The permission handler correctly gives admins `auth.AllPermissions` and resolves the full permission set for non-admins. The 12 new integration tests use a deny→grant→allow pattern that doesn't depend on the Everyone group's seed data. The route audit is thorough — 8 routes fixed, 18 verified correct, and the ICD is updated.
---
## Recommended Merge Order
1. Fix P0 #1–3 (pipe wiring, user menu RBAC, `_unwrap` condition)
2. Fix P1 #4–8 (phantom API, submenu anchor, dashboard clicks, version, CI)
3. Merge and deploy to dev
4. Track P2 items for v0.37.15 or v0.38.0
5. Clean up P3 items opportunistically

95
ICD-DRIFT-AUDIT.md
View File

@@ -1,95 +0,0 @@
# ICD Drift Audit — SDK vs Backend
**Date:** 2026-03-23
**Method:** Compared `src/js/sw/sdk/api-domains.js` against `docs/ICD/*.md` and `server/main.go`.
**Principle:** ICD is the zone of truth. SDK must conform to it.
---
## A. SDK Bugs — Wrong method or path (fix in SDK)
| SDK Method | SDK Calls | Backend Route | Issue |
|---|---|---|---|
| `workflows.update(id, data)` | `PUT /workflows/:id` | `PATCH /workflows/:id` | `crud()` generates PUT; backend is PATCH → 404/405 |
| `workspaces.update(id, data)` | `PUT /workspaces/:id` | `PATCH /workspaces/:id` | Same — `crud()` generates PUT; backend is PATCH |
| `tasks.start(id, data)` | `POST /tasks/:id/start` | `POST /tasks/:id/run` | Path mismatch → 404 |
| `tasks.stop(id)` | `POST /tasks/:id/stop` | `POST /tasks/:id/kill` | Path mismatch → 404 |
| `dataPortability.deleteAccount(data)` | `POST /profile/delete` | `DELETE /me` | Both method and path wrong |
## B. SDK Bug — Duplicate key overwrites (fix in SDK)
`notifications` is defined **three times** as a top-level key in the `createDomains` return object:
1. **Line 186** — Domain #11 (has `prefs`, `setPref`, `delPref`, uses `POST` for markRead)
2. **Line 431** — Inside `admin` namespace (correct, no conflict)
3. **Line 501** — "Misc" section (uses `PATCH` for markRead, no preference methods)
JavaScript last-write-wins: #3 overwrites #1. The notification preference methods (`prefs`, `setPref`, `delPref`) are silently lost. The `markRead` method changes from POST to PATCH (PATCH matches the ICD, so #3 has the correct HTTP method but is missing methods).
**Fix:** Delete the duplicate at line 501. Merge its `del` method and `PATCH` fix into the domain #11 block at line 186.
## C. ICD Gaps — Endpoints exist in main.go, not documented in ICD
These endpoints are real (registered in main.go, used by the SDK) but have no ICD documentation.
| Endpoint | Belongs in ICD File | Notes |
|---|---|---|
| `POST /channels/:id/mark-read` | channels.md | SDK: `channels.markRead()` |
| `POST /channels/:id/generate-title` | channels.md | SDK: `channels.generateTitle()` |
| `POST /channels/:id/summarize` | channels.md | SDK: `channels.summarize()` |
| `GET /channels/:id/messages/:msgId/siblings` | channels.md | SDK: `channels.siblings()` |
| `GET /knowledge-bases-discoverable` | knowledge.md | SDK: `knowledge.discoverable()` |
| `GET /folders`, `POST`, `PUT /:id`, `DELETE /:id` | (new section) | Full CRUD, needs own ICD section or add to channels.md |
| `GET /users/search?q=...` | (new section) | SDK: `users.search()` |
| `POST /presence/heartbeat`, `GET /presence` | (new section) | SDK: `presence.heartbeat()` |
| `GET /usage` | (new section) | SDK: `usage.mine()` |
| `GET /groups/mine` | (new section) | SDK: `groups.mine()` |
| `GET /export/me` | utility.md | SDK: `dataPortability.exportMe()` |
| `DELETE /me` | utility.md | GDPR account deletion |
| `GET /git-credentials`, `POST`, `DELETE /:id` | (new section) | SDK: `git.credentials.*` |
| `POST /git-credentials/generate` | (new section) | Not in SDK — SSH key generation |
| `GET /git-credentials/:id/public-key` | (new section) | Not in SDK |
| `GET /tools` | (new section) | SDK: `tools.list()` |
## D. ICD-Documented Endpoints Missing from SDK
In the ICD but the SDK doesn't expose them. Add only as surfaces need them.
| Endpoint | ICD File | Notes |
|---|---|---|
| Workflow stages CRUD (`GET/POST/PUT/DELETE /workflows/:id/stages/*`) | workflows.md | Workflow editor needs these |
| `PATCH /workflows/:id/stages/reorder` | workflows.md | |
| `POST /workflows/:id/publish` | workflows.md | Team-scoped version exists in SDK |
| `POST /workflows/:id/start` | workflows.md | Instance creation |
| `GET /channels/:id/workflow/status` | workflows.md | |
| `POST /channels/:id/workflow/advance` | workflows.md | |
| Workspace archive (`download`, `upload`) | workspaces.md | |
| `POST /workspaces/:id/reconcile` | workspaces.md | |
| `GET /workspaces/:id/stats` | workspaces.md | |
| `GET /workspaces/:id/index-status` | workspaces.md | |
| `POST /workspaces/:id/git/clone`, `git/pull` | workspaces.md | |
| `GET /extensions/:id/manifest` | extensions.md | |
| `GET /extensions/tools` | extensions.md | |
| `POST /personas/:id/avatar`, `DELETE` | personas.md | User-scoped (admin-scoped exists) |
| `POST /knowledge-bases/:id/rebuild` | knowledge.md | |
## E. Response Shape Mismatches
| Endpoint | Response Shape | Convention | Notes |
|---|---|---|---|
| `GET /api-configs` | `{"configs": [...]}` | Should be `{"data": [...]}` | SDK `_unwrap` won't strip |
| `GET /admin/configs` | `{"configs": [...]}` | Should be `{"data": [...]}` | Same |
| `GET /admin/models` | `{"models": [...]}` | Should be `{"data": [...]}` | Same |
| `GET /admin/users` | `{"users": [...]}` | Should be `{"data": [...]}` | Same |
These use named keys instead of the `{"data": [...]}` envelope convention. The SDK's `_unwrap` doesn't strip these, so consumers get the envelope object.
---
## Recommended Priority
1. Fix §A (SDK 404s) — these are broken in production
2. Fix §B (duplicate notifications key)
3. Document §C (ICD gaps) — endpoints work, just undocumented
4. Add §D methods to SDK — only as surfaces need them
5. Migrate §E response shapes to `{"data": [...]}` — breaking change, defer

322
README.md
View File

@@ -1,295 +1,57 @@
# Chat Switchboard
# Switchboard Core
A self-hosted AI chat application for enterprise and government environments. Unified interface for multiple AI providers with admin controls, team management, and security features.
A self-hosted extension platform. Identity, teams, permissions, workflows,
and a package system. Everything else ships as installable extensions.
## What This Is
Switchboard Core is the kernel. It provides the primitives that extensions
build on: users, teams, groups, scoped credentials, a Starlark sandbox,
workflow orchestration, notifications, and a package installer. It does not
include AI chat, providers, personas, or any domain-specific features —
those are all extension packages.
## Stack
- **Backend**: Go / Gin
- **Frontend**: Preact + htm (no build step)
- **Database**: PostgreSQL (production) + SQLite (dev/test/edge)
- **Sandbox**: Starlark with capability-gated modules
- **Deployment**: Single Docker image, Kubernetes
## Quick Start
```bash
# Clone and start
git clone <repo-url> && cd chat-switchboard
docker compose up -d
# Access at http://localhost:3000
# Default admin: admin / admin
git clone <repo-url> && cd switchboard-core
cp server/.env.example server/.env # edit DB credentials
cd server && go run .
# → http://localhost:8080
```
## Features
## Kernel Features
- **Multi-Provider**: Anthropic, OpenAI, OpenRouter, Venice AI — with BYOK support
- **Projects**: Group related conversations, KBs, and notes into workspaces with scope-aware visibility
- **Team Management**: Roles (admin/user) for vertical permissions, Teams for horizontal visibility
- **Personas**: Preset configurations (model + system prompt + parameters) at global, team, or personal scope
- **Model Catalog**: Three-state visibility (enabled/disabled/team-only) with admin controls
- **Message Trees**: Edit and regenerate with full conversation forking — navigate sibling branches
- **Notes**: Markdown notes with full-text search, folders, and tags
- **Extensions**: Browser extension system with custom renderers, tool bridge, and self-contained styling
- Built-in: Mermaid diagrams, KaTeX math, CSV tables, diff viewer, JS sandbox, regex tester
- **Server Tools**: Calculator and datetime tools (auto-registered, zero config)
- **Audit Log**: All admin operations logged with actor, action, and resource details
- **Security**: JWT + refresh tokens, AES-256-GCM API key encryption, optional mTLS, optional OIDC/Keycloak, environment classification banners
- **Airgapped**: Local vendor files (marked.js, DOMPurify, KaTeX, mermaid.js), no CDN required
- **Mobile**: Responsive design, PWA manifest
- **Real-time**: WebSocket event bus for tool bridge, live updates
- **Notifications**: In-app notification bell with unread badge, per-type email delivery (SMTP), user preference controls, admin SMTP configuration
- **Multi-model**: @mention routing for multiple AI models per channel, fan-out completions, model attribution
- **Auth**: Builtin password, mTLS (client cert), OIDC (Keycloak et al.)
- **Teams & Groups**: Horizontal isolation + vertical permissions
- **Packages**: Unified registry for surfaces, extensions, libraries, workflows
- **Starlark Sandbox**: Capability-gated server-side scripting for extensions
- **Workflows**: Staged processes with forms, review, and webhooks
- **Connections**: Scoped credential storage (global/team/personal), AES-256-GCM encrypted
- **Notifications**: In-app with per-type preferences
- **Audit Log**: All admin operations logged
- **Object Storage**: PVC or S3-compatible
- **Multi-Replica HA**: PG-backed WS tickets and rate limit counters
## Architecture
## Documentation
```
┌─────────────┐ ┌──────────────────────────────┐
│ Browser │────▶│ nginx (port 80) │
│ (vanilla JS)│◀────│ ├─ /api/* → Go backend:8080 │
└─────────────┘ │ ├─ /ws → WebSocket proxy │
│ └─ /* → static files │
└──────────────────────────────┘
│
┌─────────▼─────────┐
│ Go Backend │
│ ├─ handlers/ │
│ ├─ store/postgres/ │
│ ├─ store/sqlite/ │
│ ├─ providers/ │
│ └─ capabilities/ │
└─────────┬─────────┘
│
┌─────────▼─────────┐
│ PostgreSQL 16 or │
│ SQLite (embedded) │
└───────────────────┘
```
- [Architecture](docs/ARCHITECTURE.md) — kernel components and design reasoning
- [Roadmap](ROADMAP.md) — current status and planned milestones
- [Changelog](CHANGELOG.md) — version history
**Go backend** (vanilla, no framework beyond Gin) with a store layer abstracting all database access. Dual-driver architecture: `store/postgres` for production deployments, `store/sqlite` for single-user, edge, and development scenarios — selected at startup via `DB_DRIVER`. Providers package handles LLM API calls. Capabilities package resolves model features from catalog data, known model tables, and heuristic inference. Server tools (calculator, datetime) auto-register via `init()`. EventBus + WebSocket hub routes tool calls between backend and browser extensions.
## Project Status
**Frontend** is vanilla JavaScript — no build step, no bundler. 15 files organized by domain: `api.js` (HTTP client), `app.js` (state + init), `chat.js` (send, regen, edit, branch), `events.js` (event bus + WebSocket), `extensions.js` (loader, registry, renderer pipeline, tool bridge), `ui-core.js` (DOM rendering + streaming), `ui-format.js` (markdown, code blocks), `ui-primitives.js` (shared components), `ui-settings.js` / `ui-admin.js` (settings and admin panels), `notes.js`, `tokens.js`, `debug.js`, `settings-handlers.js`, `admin-handlers.js`.
## Configuration
All configuration via environment variables. See `server/.env.example` for the full list.
| Variable | Default | Description |
|----------|---------|-------------|
| `PORT` | `8080` | Backend listen port |
| `BASE_PATH` | ` ` | URL prefix (e.g. `/chat`) |
| `DB_DRIVER` | `postgres` | Database backend: `postgres` or `sqlite` |
| `DATABASE_URL` | ` ` | SQLite: path to database file (e.g. `/data/switchboard.db`) |
| `DB_HOST` | `localhost` | PostgreSQL host (ignored when `DB_DRIVER=sqlite`) |
| `DB_NAME` | `chat_switchboard` | Database name (ignored when `DB_DRIVER=sqlite`) |
| `JWT_SECRET` | (required) | Token signing key |
| `SWITCHBOARD_ADMIN_USERNAME` | ` ` | Bootstrap admin username |
| `SWITCHBOARD_ADMIN_PASSWORD` | ` ` | Bootstrap admin password |
| `ENCRYPTION_KEY` | ` ` | AES key for API key encryption (required if encrypted keys exist) |
| `SEED_USERS` | ` ` | Dev/test only: `user:pass:role,user2:pass2:role2` |
| `ENVIRONMENT` | `development` | Environment name: `development`, `test`, or `production`. Auto-sets Gin release mode in production. |
| `GIN_MODE` | (auto) | Gin framework mode: `debug`, `test`, or `release`. Auto-derived from `ENVIRONMENT` if not set. |
| `STORAGE_BACKEND` | (auto) | `pvc` or `s3`. Auto-detects PVC if not set. |
| `STORAGE_PATH` | `/data/storage` | PVC mount point (also scratch dir for S3 extraction) |
| `S3_ENDPOINT` | ` ` | S3 endpoint (e.g. `http://minio:9000`, required for S3) |
| `S3_BUCKET` | ` ` | S3 bucket name (must exist, required for S3) |
| `S3_ACCESS_KEY` | ` ` | S3 access key ID |
| `S3_SECRET_KEY` | ` ` | S3 secret access key |
| `S3_REGION` | `us-east-1` | S3 region |
| `S3_PREFIX` | ` ` | Optional key prefix within bucket |
| `S3_FORCE_PATH_STYLE` | `true` | Path-style URLs (required for MinIO, Ceph) |
## Deployment
Three Docker images support different scenarios:
| Image | Dockerfile | Use Case |
|-------|-----------|----------|
| **Unified** | `Dockerfile` | Dev, docker-compose, single-node |
| **Backend** | `server/Dockerfile` | K8s — scale API pods independently |
| **Frontend** | `Dockerfile.frontend` | K8s — scale FE pods independently |
### Docker Compose (development — unified image)
```bash
docker compose up -d # start all services
docker compose up -d --build # rebuild after code changes
docker compose --profile dev up # include Adminer DB UI
```
### Kubernetes (split images)
Build and push both images:
```bash
docker build -f server/Dockerfile -t your-registry/switchboard-api:latest server/
docker build -f Dockerfile.frontend -t your-registry/switchboard-fe:latest .
```
**Backend deployment:**
```yaml
containers:
- name: api
image: your-registry/switchboard-api:latest
ports:
- containerPort: 8080
env:
- name: DB_HOST
value: "postgres-service"
- name: BASE_PATH
value: "/chat" # must match ingress path
- name: JWT_SECRET
valueFrom:
secretKeyRef:
name: switchboard-secrets
key: jwt-secret
```
**Frontend deployment:**
```yaml
containers:
- name: frontend
image: your-registry/switchboard-fe:latest
ports:
- containerPort: 80
env:
- name: BASE_PATH
value: "/chat" # injected into index.html at startup
volumeMounts:
- name: branding
mountPath: /branding
readOnly: true # optional: custom logo, colors
```
**Ingress** routes `/api/*` and `/ws` to the backend Service, everything else to the frontend Service. The frontend entrypoint generates the nginx config dynamically based on `BASE_PATH`.
### Path-Based Routing
Set `BASE_PATH=/chat` to serve the application under a subpath. The frontend reads `window.__BASE__` injected at container startup, and all API calls are prefixed automatically.
### Airgapped / Disconnected
The Docker build bakes in `marked.js`, `DOMPurify`, and `KaTeX` (JS + CSS + fonts) from npm during the vendor stage. Mermaid.js loads dynamically from local vendor with CDN fallback. No CDN calls at runtime in airgapped deployments. The `src/vendor/` directory contains local copies as fallback for development without Docker.
## Database
### PostgreSQL (default)
PostgreSQL 16+ recommended. The `pgcrypto` and `vector` (pgvector) extensions are used for UUID generation and vector similarity search respectively.
### SQLite (single-user / edge / dev)
Set `DB_DRIVER=sqlite` and `DATABASE_URL=/path/to/switchboard.db` to run with an embedded SQLite database. No external dependencies — the binary is self-contained (pure Go, no CGO). The SQLite backend has full feature parity with Postgres including knowledge base vector search, which uses app-level cosine similarity computed in Go rather than pgvector.
```bash
# Minimal single-binary startup
DB_DRIVER=sqlite DATABASE_URL=./data/switchboard.db JWT_SECRET=changeme ./switchboard
```
SQLite is best suited for single-user workstations, edge deployments, air-gapped laptops, and local development. For multi-user production with concurrent writes, use PostgreSQL.
### Schema Management
The Go backend auto-migrates on startup using files in `server/database/migrations/`. For manual operations:
```bash
# Bootstrap database (superuser, creates role + DB)
scripts/db-bootstrap.sh
# Manual migration (usually not needed)
scripts/db-migrate.sh
# Validate schema
scripts/db-validate.sh
```
### Key Tables (v0.19)
| Table | Purpose |
|-------|---------|
| `users` | Accounts with role, avatar, settings, encrypted vault (UEK) |
| `provider_configs` | API provider configurations (scope: global/team/personal) |
| `model_catalog` | Synced model list with capabilities, visibility, and type |
| `personas` | Model presets (scope: global/team/personal) |
| `projects` | Workspaces grouping channels, KBs, and notes (scope: personal/team/global) |
| `project_channels` | Ordered channel-to-project membership (UNIQUE channel_id) |
| `project_knowledge_bases` | KB-to-project binding with auto_search flag |
| `project_notes` | Note-to-project association |
| `channels` | Conversations with type (direct/group/channel), optional `project_id` |
| `messages` | Message tree with parent_id for forking, tool_calls JSONB |
| `teams` / `team_members` | Organizational units |
| `notes` | Markdown notes with full-text search |
| `knowledge_bases` / `kb_documents` / `kb_chunks` | RAG pipeline with pgvector embeddings |
| `extensions` / `extension_user_settings` | Browser extension registry and per-user config |
| `usage_log` / `model_pricing` | Token usage tracking and cost calculation |
| `audit_log` | Admin action audit trail |
| `user_model_settings` | Per-user model visibility and sort preferences |
## API
All endpoints under `/api/v1/`. Authentication via `Authorization: Bearer <token>` header.
### Auth
- `POST /auth/register` — Register (if `allow_registration` policy is true)
- `POST /auth/login` — Login, returns access + refresh tokens
- `POST /auth/refresh` — Refresh access token
- `POST /auth/logout` — Revoke refresh token
### Channels & Messages
- `GET/POST /channels` — List/create conversations
- `GET/PUT/DELETE /channels/:id` — Channel CRUD
- `GET/POST /channels/:id/messages` — Message list/create
- `POST /chat/completions` — Stream AI completions (SSE)
- `POST /channels/:id/messages/:msgId/edit` — Edit and fork
- `POST /channels/:id/messages/:msgId/regenerate` — Regenerate response
### Models
- `GET /models/enabled` — Models available to the user (global + team + personal)
- `GET/PUT /models/preferences` — User model visibility settings
### Personal Providers (BYOK)
- `GET/POST /api-configs` — User's personal provider configs
- `GET/PUT/DELETE /api-configs/:id` — Provider CRUD
- `POST /api-configs/:id/models/fetch` — Refresh models from provider API
- `GET /api-configs/:id/models` — List models for a personal provider
### Teams
- `GET /teams` — Teams the user belongs to
- `GET/POST /teams/:id/providers` — Team provider management (team admins)
- `GET/POST /teams/:id/presets` — Team preset management (team admins)
### Projects
- `GET/POST /projects` — List and create projects
- `GET/PUT/DELETE /projects/:id` — Project CRUD (owner-only delete)
- `POST/DELETE /projects/:id/channels/:channelId` — Add/remove channel
- `GET /projects/:id/channels` — List project channels
- `PUT /projects/:id/channels/reorder` — Reorder channels
- `POST/DELETE /projects/:id/knowledge-bases/:kbId` — Bind/unbind KB
- `POST/DELETE /projects/:id/notes/:noteId` — Bind/unbind note
### Admin
- `GET/POST /admin/users` — User management
- `GET/PUT /admin/settings/:key` — Global settings and policies
- `GET/POST /admin/configs` — Global provider configs
- `GET/POST /admin/models/fetch` — Model catalog sync
- `GET/PUT /admin/roles/:role` — Model role configuration
- `GET /admin/usage` — Usage dashboard
- `GET /admin/audit` — Audit log
- `GET/POST/PUT/DELETE /admin/extensions` — Extension management
### Extensions
- `GET /extensions?tier=browser` — List enabled browser extensions
- `GET /extensions/:id/assets/*path` — Serve extension assets (public, no auth)
### WebSocket
- `GET /ws?token=<jwt>` — EventBus WebSocket for real-time events and tool bridge
## Development
```bash
# Backend (requires Go 1.22+)
cd server
cp .env.example .env # edit with your DB credentials
go run .
# Frontend (just serve static files)
# Use any HTTP server pointed at src/
python3 -m http.server 3000 --directory src
```
**v0.1.0** (in progress) — kernel extracted from chat-switchboard v0.38.5.
~44K lines of chat/AI code removed, 27 kernel tables, 20 store interfaces.
See [ROADMAP.md](ROADMAP.md) for details.
## License

84
ROADMAP.md Normal file
View File

@@ -0,0 +1,84 @@
# Switchboard Core — Roadmap
## Current: v0.1.0 — The Kernel
Fork of chat-switchboard, gutted to a pure extension platform. All AI/chat
features removed from the kernel. What remains is the minimum viable
platform that extensions build on.
### Phase 0 (in progress)
| Step | Status | Description |
|------|--------|-------------|
| 1. Module rename | ✅ | `chat-switchboard` → `switchboard-core` |
| 2. Delete packages | ✅ | 15 Go packages, 29 handler files removed |
| 3. Gut stores/models | ✅ | 40 → 20 store interfaces, kernel-only models |
| 4. Fresh migrations | ✅ | 9 files × 2 dialects, 27 tables |
| 5. Fix compilation | 🔧 | ~95% done, ~5 compile errors remaining |
| 6. Fix tests | ⬜ | Prune 13K test lines to kernel-only |
| 7. Frontend gut | ⬜ | Shell + SDK only, vendor libs baked |
| 8. New ICD | ⬜ | Kernel-only API spec |
| 9. CI/CD + Dockerfile | ⬜ | Single image, new DB names |
| 10. Smoke test + tag | ⬜ | Deploy, verify, tag v0.1.0 |
## v0.2.0 — SDK & Triggers
The contract that extensions build against. Three trigger primitives,
SDK stabilization, and the first rebuilt extension (tasks).
- **Trigger system**: time (cron), webhook (inbound HTTP), event (bus subscription)
- **Event bus subscriptions**: extensions register match expressions at install time
- **SDK contract**: `sb.slots()`, `sb.actions`, `sb.api.ext()`, `sb.storage`
- **Primitive UI components**: toast, confirm, prompt, dialog (stabilize existing)
- **Theme tokens** exposed to extensions
- **Task extension**: first proof-of-concept — full task system rebuilt as a
Starlark extension using triggers + ext_data + notifications
## v0.3.0 — Editor Surface
The code/markdown editor rebuilt as an installable surface package.
Zero platform special-casing. Proves the full extension stack E2E.
- Editor as `.pkg` archive
- CM6 integration via surface viewport
- File tree, tabs, preview pane — all extension-provided
## v0.4.0 — Chat Extension
The AI chat system rebuilt as an installable extension package.
- Provider registry as extension (BYOK chain, model catalog)
- Completion streaming via Starlark `provider.complete`
- Personas as extension data
- Channel/message storage via ext_data tables
- Tool system as extension hooks
## v0.5.0 — MVP
Extension and operations tracks converge. First externally usable release.
- Package registry (browse, install, update, uninstall)
- Extension marketplace foundations
- Health monitoring dashboard
- Backup/restore tooling
- Documentation site
## Post-MVP
- Rich media extensions: image generation, code sandbox, STT/TTS
- Desktop app (Tauri or Electron)
- Sidecar tier: container-based extensions
- Federation: cross-instance package sharing
- Plugin marketplace with signing and review
## Design Decisions Log
| Decision | Rationale |
|----------|-----------|
| Tasks → extension | Scheduler was the most entangled kernel component (~3,400 lines). Rebuilding as extension validates the trigger system and removes the worst compilation debt. Three trigger primitives (time, webhook, event) replace the monolithic scheduler. |
| Sessions removed | Channel-based sessions coupled to deleted chat system. Workflow instances need new storage model — either ext_data tables or a dedicated kernel table. |
| `chat_only` → `custom` | Stage mode `chat_only` implied chat as a kernel concept. Renamed to `custom` which delegates to a surface package, proving extension composability. |
| Providers removed from kernel | Provider configs, model catalog, routing policies — all moved to extension track. Kernel provides credential storage (connections) and the Starlark `provider.complete` module as the interface. |
| Kernel permissions simplified | From 16 chat-centric permissions to 6 platform permissions. Extensions define their own capability requirements in manifests. |
| Preact+htm retained | 3KB runtime, no build step, works for extension authors without bundler config. KISS. |
| Single Docker image | Drop the frontend/backend split. Go binary + assets + migrations in one image. Simpler deployment, fewer moving parts. |

129
TURNOVER-0.37.14-S8.md
View File

@@ -1,129 +0,0 @@
---
**Turnover — changeset-0.37.14 Session 8**
**Version:** 0.37.14.20 | 3 commits (2 yours + 1 triage) | PR #226
**Branch:** `changeset-0.37.14` | Commit: `e2481a2`
---
## Commits this session
- `b8e03c4` — Upload 3 new, 3 modified files (profile bootstrap, ICD drift audit)
- `0ceb20a` — Delete Critical Review.md (renamed to FE-REWRITE-REVIEW-0.37.14.md)
- `e2481a2` — Critical Review triage: RBAC fix, _unwrap, pipe wiring, menu/dashboard, CI
## What was done (triage commit)
### P0 Fixes (3) — Fix Before Merge
1. **RBAC inversion** (1 file, 3 lines) — `user-menu.js` lines 85/91/96: `!authenticated || sw?.isAdmin` was inverted, showing Admin/Team Admin/Debug to unauthenticated users. Fixed to `authenticated && sw?.isAdmin`. Backend already enforced 403, but UI now correctly gates visibility.
2. **`_unwrap` SDK contract** (1 file, 1 line) — `rest-client.js` line 101: condition required pagination fields (`page`, `total`, `per_page`) to unwrap `{data: [...]}` envelopes. 24 call sites defensively did `resp.data || resp || []`. Relaxed to just check `Array.isArray(json.data)`.
3. **Pipe/filter pipeline wiring** (3 files) — `sw.pipe._runPre/Stream/Render` was fully built but never invoked. Wired into:
- `use-chat.js`: pre-send filter before completion call (can block/transform)
- `use-stream.js`: per-chunk stream filter in SSE loop
- `markdown.js`: render filter after sanitization + @mention highlighting
### P1 Fixes (5) — Fix Before Tag
4. **deleteMessage phantom API** (1 file) — No server DELETE `/channels/:id/messages/:msgId` route exists. Removed `onDelete` prop from ChatPane so delete button is hidden until backend is wired.
5. **Menu submenu positioning** (1 file) — `menu.js`: submenu anchored to parent menu root div instead of triggering item. Added `itemRefs` map, `setSubmenu` now stores `{ item, anchorEl }`, submenu renders at correct position.
6. **Dashboard recent items** (1 file) — `chat-workspace.js`: added `onNavigate` prop to ChatDashboard, recent items now have `onClick` handler + cursor pointer.
7. **Version string mismatch** (1 file) — `sdk/index.js` line 164: hardcoded `'v0.37.9'` replaced with template literal `` `v${sw._sdk}` ``. Console now shows correct version.
8. **CI frontend coverage** (1 file) — `ci.yaml`: glob changed from `src/js/*.js` (top-level only) to `find src/js -name '*.js'` (recursive). Uses `--input-type=module` for ES module syntax checking.
### P2/P3 Quick Fixes (4) + Additional (2)
9. **Pipe Promise guard** — `pipe.js` `_runChain`: detects async filters returning Promises, logs error + increments error counter, continues chain.
10. **useEffect no-deps documented** — `chat-pane/index.js` line 66: added comment explaining intentional missing dependency array (handle must reference latest closures).
11. **Dual toast container** — `base.html`: removed orphaned `#toastContainer` div (Preact toast primitive creates its own).
12. **Dead `esc()` function** — `admin/users.js` line 7: deleted unused function.
13. **WS message deduplication** — `use-chat.js` line 288: added `prev.some(m => m.id && m.id === msg.id)` check before appending from WebSocket. Prevents duplicates on reconnect.
14. **Optimistic message key stability** — `use-chat.js` line 103: optimistic user messages now get `id: 'tmp-' + Date.now()` so Preact key is always stable.
### Roadmap Updates
- `docs/ROADMAP.md` v0.37.17 row: added "debug modal Preact rebuild (CR P2-5)"
- `docs/ROADMAP.md` v0.37.18 row: added "sw.can() gates (P2-1), __USER__/__PAGE_DATA__ removal (P2-4), _getScale SDK (P3-3)"
- `FE-REWRITE-REVIEW-0.37.14.md`: P0/P1 sections marked resolved, P2/P3 sections annotated with version assignments.
## Files changed (+353/−35, 21 files)
| File | Change |
|---|---|
| src/js/sw/shell/user-menu.js | ±6 RBAC condition fix |
| src/js/sw/sdk/rest-client.js | ±1 _unwrap condition |
| src/js/sw/components/chat-pane/use-chat.js | +11/−2 pipe pre, dedup, temp ID |
| src/js/sw/components/chat-pane/use-stream.js | +9/−1 pipe stream filter |
| src/js/sw/components/chat-pane/markdown.js | +12 pipe render, doc comment |
| src/js/sw/components/chat-pane/index.js | ±4 remove onDelete, useEffect doc |
| src/js/sw/primitives/menu.js | +16/−7 submenu anchor refactor |
| src/js/sw/surfaces/chat/chat-workspace.js | +8/−3 dashboard onClick, onNavigate |
| src/js/sw/sdk/index.js | ±1 version template literal |
| src/js/sw/sdk/pipe.js | +6 Promise guard |
| src/js/sw/surfaces/admin/users.js | −2 dead code |
| server/pages/templates/base.html | −1 orphaned toast div |
| .gitea/workflows/ci.yaml | +14/−4 recursive glob + ES module |
| docs/ROADMAP.md | ±4 v0.37.17/18 deferred items |
| FE-REWRITE-REVIEW-0.37.14.md | ±10 resolution annotations |
| VERSION | 0.37.14.20 |
| server/handlers/profile_bootstrap.go | +117 (your commit) |
| server/main.go | +4 (your commit) |
| server/handlers/integration_test.go | +4 (your commit) |
| ICD-DRIFT-AUDIT.md | +95 (your commit) |
| docs/ICD/profile.md | +59 (your commit) |
## Verification results
| Check | Result |
|---|---|
| Admin menu items visible for admin | PASS |
| Channel messages load (unwrap working) | PASS |
| Dashboard recent items clickable | PASS |
| Delete button hidden | PASS |
| Console version: `SDK v0.37.14` | PASS |
| Zero console errors | PASS |
| @mentions rendered correctly | PASS |
| Docker build + server start | PASS |
## Deferred items — version mapping
| Item | Assigned To | Description |
|---|---|---|
| P3-5: Double-unwrap cleanup | **DONE v0.37.14.21 (S9)** | 93 patterns (not 24) cleaned; API envelope normalization; 3 bug fixes |
| P2-5: Debug modal rebuild | **v0.37.17** | Raw HTML + inline onclick in base.html → Preact component |
| P2-1: sw.can() RBAC gates | **v0.37.18** | Gate action buttons with `sw.can()` across all surfaces |
| P2-4: __USER__/__PAGE_DATA__ | **v0.37.18** | Remove dead Go template injection, migrate settings to sw.auth.policies |
| P3-3: _getScale SDK | **v0.37.18** | Move sidebar DOM coupling to `sw.shell.getScale()` API |
| P3-4: Notes markdown raw DOM | **documented** | Intentional for perf, comment added in markdown.js |
## Known issues (carried forward)
1. **Thinking tags lost on refresh** — `<think>` content spills into message body after reload. DB stores raw, load path doesn't extract like streaming path does.
2. **Model settings select/unselect all** — no bulk toggle. Deferred to per-provider model management UI.
3. **Double-unwrap cleanup** — 24 call sites still have defensive `resp.data || resp || []`. Safe now but redundant. Next session.
## Open backlog
1. Double-unwrap cleanup (24 sites, next session)
2. Thinking tags persistence across refresh
3. ICD remaining failures (Venice stream, knowledge 403-vs-404)
4. Extension surface "? User" label
5. Admin panel audit
6. Model qualified display + per-provider management UI
7. SDK gaps (20+ missing endpoints)
8. Team-admin surface audit
9. Settings surface audit
10. Notes surface audit
11. deleteMessage server endpoint (server route + handler needed)
---

685
UI redesign.md
View File

@@ -1,685 +0,0 @@
# Chat Switchboard UI Rewrite — Design Document
**Version:** 0.0.1 (draft)
**Date:** 2026-03-20
**Scope:** Scorched earth rebuild of frontend on Preact+htm
---
## 1. Survey Findings
### What Exists
| Metric | Value |
|--------|-------|
| JS files | 55 |
| Total JS lines | ~29,350 |
| CSS files | 22 (~203KB) |
| Go templates | ~1,920 lines across 14 template files |
| `innerHTML =` assignments | 294+ (91 in ui-admin.js alone) |
| `querySelector` / `getElementById` calls | 600+ (130 in ui-admin.js) |
| Direct `fetch()` outside api.js | ~24 calls across 10 files |
| Vendor deps | marked, DOMPurify, mermaid, KaTeX, CM6 (esbuild bundle) |
### The SDK (switchboard-sdk.js)
- ~860 lines, version-tagged `v0.28.5` in code but `v0.30.2` in docs
- Thin wrapper over globals: `API`, `Events`, `Theme`, `UserMenu`
- Delegates to `API._get()`, `API._post()`, etc. — not its own fetch layer
- Has pipe/filter pipeline (pre-send, stream, render) — this is good and reusable
- Has event bus bridge — good, reusable
- Has theme control — good, reusable
- No RBAC. Only `sw.isAdmin` (boolean derived from `API.user.role`)
- No namespaced domain methods (e.g., `sw.api.channels.list()`)
### The Registry (sb.js)
- Action registry: `sb.register(name, fn)` + `sb.ns(name, obj)`
- Dual-write to `window[name]` for backward compat
- Go templates call `sb.call('action', args)` via onclick handlers
- 55 files all register into this flat global space
### Server Templates
- Go templates generate initial DOM with hardcoded IDs
- JS hydrates by ID lookups (`getElementById`, `querySelector`)
- Templates inject `window.__USER__`, `window.__PAGE_DATA__`, `window.__SURFACE__`
- This server-rendering + client-hydration pattern is deeply coupled
### What's Missing for RBAC
- **No user-facing permissions endpoint.** `GET /admin/users/:id/permissions`
is admin-only. There is no `GET /profile/permissions` or equivalent. The
login/refresh response includes only `role: "user"|"admin"`, not the
resolved permission set.
- The frontend has zero permission-level gating. Everything is gated by
`API.isAdmin` (binary) or by server-side template conditionals.
### Dependency Pipeline
- `Dockerfile.frontend`: Stage 1 `npm pack` pulls marked, DOMPurify,
mermaid, KaTeX. Stage 2 builds CM6 via esbuild. Stage 3 copies into
nginx image.
- Preact + htm would be added to Stage 1 as additional `npm pack` targets.
- No bundler for app code — scripts loaded sequentially via `<script>` tags
in `base.html`. This stays the same (Preact + htm work without a bundler).
---
## 2. Architecture
### Layer Model
```
┌─────────────────────────────────────────────────┐
│ Layer 2: Shell │
│ UserMenu, AdminSettings, UserSettings, │
│ SurfaceViewport, Banners │
├─────────────────────────────────────────────────┤
│ Layer 1: SDK │
│ sw.api.{domain}.* — namespaced REST client │
│ sw.auth.* — tokens, session lifecycle │
│ sw.can(perm) — RBAC gate │
│ sw.on/off/emit — event bus │
│ sw.pipe.* — filter pipeline │
│ sw.theme.* — theme control │
├─────────────────────────────────────────────────┤
│ Layer 0: Primitives │
│ Menu, Dialog, Toast, Drawer, Banner, │
│ Button, FormField, Tabs, Dropdown, Tooltip │
├─────────────────────────────────────────────────┤
│ Preact + htm (~4KB) │
└─────────────────────────────────────────────────┘
```
### Design Principles
1. **No raw DOM.** Every UI element is a Preact component. No `innerHTML`,
no `querySelector` in application code.
2. **No raw fetch.** All API access through `sw.api.{domain}.*`. Direct
`fetch()` only inside the SDK internals.
3. **RBAC at the SDK level.** `sw.can('channel.create')` checks the cached
permission set. Primitives consume it: a menu item that requires
`admin.access` simply doesn't render if the user lacks it.
4. **Surfaces own their content.** The shell provides the viewport and
chrome (menu, banners). Everything inside the viewport is a surface's
responsibility. Shell and surfaces communicate only through the SDK.
5. **CSS custom properties, not component-scoped CSS.** Keep the existing
`variables.css` token system. Primitives use those tokens. No Shadow DOM,
no CSS-in-JS.
---
## 3. Layer 0 — Primitives
Preact functional components with hooks. Each primitive is a self-contained
file. No business logic, no API calls.
### Catalog
| Primitive | Props (key ones) | Notes |
|-----------|-------------------|-------|
| `Menu` | `items[], anchor, direction, onSelect` | Flyout menu. Directions: `up-left`, `up-right`, `down-left`, `down-right`. Handles viewport overflow (reposition if clipped). Closes on outside click/Escape. |
| `Dialog` | `open, title, children, onClose, actions[]` | Modal dialog. Focus trapping, Escape to close. Actions are `{label, onClick, variant}`. |
| `Confirm` | `message, onConfirm, onCancel, destructive?` | Specialized Dialog. Returns Promise<boolean> via `sw.confirm()`. |
| `Prompt` | `message, defaultValue, onSubmit, onCancel` | Specialized Dialog. Returns Promise<string\|null> via `sw.prompt()`. |
| `Toast` | `message, variant, duration` | Auto-dismiss notification. Variants: `success`, `error`, `info`, `warn`. Stacks via a ToastContainer. |
| `Banner` | `text, variant, dismissible?` | Persistent bar (top/bottom). Server can inject initial banner via `__PAGE_DATA__`. |
| `Drawer` | `open, side, width, title, children, onClose` | Slide-in panel (left/right). Used for settings, admin sections. |
| `Tabs` | `tabs[], active, onChange` | Horizontal tab bar with overflow scroll arrows (existing `checkTabsOverflow` logic). |
| `Dropdown` | `options[], value, onChange, placeholder` | Select replacement with search/filter for long lists. |
| `Tooltip` | `content, children, position` | Hover tooltip. Positions: `top`, `bottom`, `left`, `right`. |
| `FormField` | `label, error, children` | Wrapper for form inputs with label + validation display. |
| `Button` | `variant, size, disabled, loading, onClick` | Variants: `primary`, `secondary`, `danger`, `ghost`. |
| `Avatar` | `src, name, size` | Circle avatar with letter fallback. |
| `Spinner` | `size` | Loading indicator. |
### Menu Positioning Rules
Menus are the hardest primitive. Rules:
1. **Anchor-relative.** Menu positions relative to its anchor element.
2. **Direction preference.** Caller specifies preferred direction. Menu
tries that first.
3. **Viewport clamping.** If the menu would overflow the viewport in the
preferred direction, flip to the opposite axis. If still overflowing,
clamp to viewport edge with scroll.
4. **Clipping ancestor escape.** If any ancestor has `overflow: hidden/auto`,
the menu switches to `position: fixed` and uses anchor's
`getBoundingClientRect()`. (This is the existing `_positionFlyout`
logic from switchboard-sdk.js — proven correct.)
5. **Close triggers.** Click outside, Escape key, scroll of a parent.
6. **Keyboard navigation.** Arrow keys move focus, Enter selects, Home/End
jump to first/last.
7. **Nested submenus.** Open on hover/arrow-right, close on arrow-left.
Only one submenu level deep (KISS).
### Dialog Rules
1. **Focus trap.** Tab cycles within the dialog. Focus starts on the first
focusable element (or the primary action button).
2. **Backdrop click** closes the dialog (unless `persistent` prop).
3. **Escape** closes the dialog.
4. **Scroll lock.** Body scroll is locked while dialog is open.
5. **Stacking.** Multiple dialogs stack. Only the topmost receives input.
Z-index increments per dialog.
### Toast Rules
1. **Container** is a fixed-position stack in bottom-right.
2. **Auto-dismiss** after `duration` ms (default: 3000, errors: 5000).
3. **Max visible:** 5. Older toasts are dismissed to make room.
4. **Hover pauses** the auto-dismiss timer.
5. **Animations:** Slide-in from right, fade-out on dismiss.
---
## 4. Layer 1 — SDK
### 4.1 Namespaced API Client
Every ICD domain becomes an SDK namespace. Methods are typed wrappers
around the REST client — no raw path construction in application code.
```js
// Instead of:
sw.api.get('/api/v1/channels')
// You write:
sw.api.channels.list()
sw.api.channels.get(id)
sw.api.channels.create({ title, type })
sw.api.channels.update(id, { title })
sw.api.channels.del(id)
```
#### Namespace Map (derived from ICD)
| Namespace | ICD Source | Key Methods |
|-----------|-----------|-------------|
| `sw.api.auth` | auth.md | `login`, `register`, `refresh`, `logout` |
| `sw.api.channels` | channels.md | `list`, `get`, `create`, `update`, `del`, `messages`, `send`, `complete` |
| `sw.api.personas` | personas.md | `list`, `get`, `create`, `update`, `del`, `groups` |
| `sw.api.knowledge` | knowledge.md | `list`, `get`, `create`, `upload`, `search` |
| `sw.api.notes` | notes.md | `list`, `get`, `create`, `update`, `del`, `search`, `graph` |
| `sw.api.projects` | projects.md | `list`, `get`, `create`, `update`, `del` |
| `sw.api.workspaces` | workspaces.md | `list`, `get`, `create`, `files`, `git` |
| `sw.api.memory` | memory.md | `list`, `review`, `extract` |
| `sw.api.models` | models.md | `list`, `preferences` |
| `sw.api.providers` | providers.md | `list`, `get`, `create`, `update`, `del`, `health`, `models` |
| `sw.api.notifications` | notifications.md | `list`, `markRead`, `preferences` |
| `sw.api.extensions` | extensions.md | `list`, `get`, `settings` |
| `sw.api.profile` | profile.md | `get`, `update`, `avatar`, `password`, `settings` |
| `sw.api.teams` | teams.md | `mine`, `get`, `members`, `providers`, `models`, `roles`, `audit`, `usage` |
| `sw.api.workflows` | workflows.md | `list`, `get`, `create`, `instances`, `advance`, `reject` |
| `sw.api.tasks` | tasks.md | `list`, `get`, `create`, `runs` |
| `sw.api.surfaces` | surfaces.md | `list`, `get`, `install` |
| `sw.api.admin` | admin.md | `users`, `settings`, `stats`, `audit`, `usage`, `vault`, `groups`, `grants`, `permissions` |
Each namespace method returns a Promise. List methods always return
the unwrapped array (SDK strips the `{"data": [...]}` envelope).
Error responses throw with `{ status, message }`.
#### Internal REST Client
The SDK's internal `_fetch` handles:
- Base path prefixing
- JWT injection (`Authorization: Bearer ...`)
- 401 → refresh → retry (once)
- Response envelope unwrapping (`data` arrays, error objects)
- AbortSignal forwarding
- Network error normalization
This replaces the current `API._get`, `API._post`, etc. The old `API`
global is **not exposed**. Nothing should call `fetch()` directly.
### 4.2 Auth Module
```js
sw.auth.isAuthenticated // boolean getter
sw.auth.user // { id, username, display_name, email, role, avatar }
sw.auth.permissions // Set<string> — resolved permission set
sw.auth.teams // [{ id, name, my_role }] — cached from /teams/mine
sw.auth.login(login, password)
sw.auth.logout()
sw.auth.refresh() // manual refresh (auto-refresh is internal)
```
`sw.auth.permissions` is populated at boot from a new backend endpoint
(see §7 Backend Changes). It's refreshed on token refresh.
### 4.3 RBAC Gate
```js
sw.can(permission) // boolean — checks sw.auth.permissions
sw.isAdmin // shortcut: sw.auth.user?.role === 'admin'
sw.isTeamAdmin(teamId) // checks sw.auth.teams for my_role === 'admin'
```
#### Primitives Integration
```js
// Menu item that only renders if user has the permission
html`<${Menu} items=${[
{ label: 'Settings', action: 'settings' },
sw.can('admin.access') && { label: 'Admin', action: 'admin' },
sw.isTeamAdmin(teamId) && { label: 'Team Admin', action: 'team-admin' },
].filter(Boolean)} />`
```
No special component needed. Standard conditional rendering in Preact
combined with `sw.can()` calls. The permission set is synchronous (cached
in memory), so there's no async awkwardness in render paths.
### 4.4 Event Bus
Carry forward the existing pattern. Same API:
```js
sw.on('channel.switched', handler)
sw.once('chat.message.received', handler)
sw.off('channel.switched', handler)
sw.emit('custom.event', payload)
```
Wildcard patterns (`chat.message.*`) are supported.
### 4.5 Pipe/Filter Pipeline
Carry forward as-is from current SDK. Same three stages:
- `sw.pipe.pre(priority, fn, opts)` — pre-send
- `sw.pipe.stream(priority, fn, opts)` — post-receive stream
- `sw.pipe.render(priority, fn, opts)` — post-render
Same scoping, same stats, same chain execution. This is proven code.
### 4.6 Theme
Same as current: `sw.theme.current`, `sw.theme.mode`, `sw.theme.set(mode)`,
`sw.theme.on('change', fn)`. Backed by `localStorage` + `data-theme`
attribute on `<html>`. Early inline script in `base.html` prevents FOUC
(keep this).
---
## 5. Layer 2 — Shell
The shell is the application frame. It renders five things:
1. **User Menu** — avatar + flyout (Settings, Admin, Team Admin, Debug, Sign Out)
2. **User Settings** — drawer or route (`/settings`)
3. **Admin Settings** — drawer or route (`/admin`)
4. **Surface Viewport** — the `<div>` where the active surface renders
5. **Banners** — top/bottom persistent bars (server-injected or SDK-triggered)
### Shell Component Tree
```
<App>
<Banner position="top" />
<div class="shell">
<SurfaceViewport surface={__SURFACE__} />
</div>
<Banner position="bottom" />
<ToastContainer />
<DialogStack />
</App>
```
The `UserMenu` lives inside the surface viewport — each surface decides
where to place it (top-left in chat, top-right in admin, etc.). The
shell provides `sw.userMenu(container, opts)` as a Preact render helper
so surfaces can mount it wherever they want.
### SurfaceViewport
The viewport is a plain `<div>` that the active surface renders into.
Surfaces are loaded by the Go template engine (same as today). The
difference: surfaces receive the Preact+htm runtime and the SDK, and
build their UI with components instead of raw DOM.
Built-in surfaces (chat, admin, settings, notes) are rebuilt as Preact
component trees. Extension surfaces (`.pkg` archives) get `sw.*` and
the primitives library, and render into `#extension-mount`.
### Admin & Settings
These are the biggest DOM-manipulation offenders (ui-admin.js: 95KB,
settings-handlers.js: 45KB, ui-settings.js: 42KB). The rewrite converts
them from imperative DOM manipulation to declarative Preact components.
**Admin sections** become routed sub-components:
- Overview, Users, Teams, Providers, Models, Roles, Routing, Settings,
Audit, Usage, Packages, Extensions, Surfaces
**Settings sections:**
- Profile, Appearance, Providers (BYOK), Models, Notifications, Tasks,
Extensions, Data Portability
Each section is a standalone Preact component that uses `sw.api.admin.*`
or `sw.api.profile.*` for data.
---
## 6. Dependency Story
### Vendoring
Add to `Dockerfile.frontend` Stage 1:
```dockerfile
RUN npm pack preact@10.x.x htm@3.x.x 2>/dev/null && \
tar xzf preact-*.tgz -C /tmp && \
mkdir -p /vendor/preact && \
cp /tmp/package/dist/preact.module.js /vendor/preact/preact.module.js && \
cp /tmp/package/hooks/dist/hooks.module.js /vendor/preact/hooks.module.js && \
rm -rf /tmp/package && \
tar xzf htm-*.tgz -C /tmp && \
cp /tmp/package/dist/htm.module.js /vendor/preact/htm.module.js && \
rm -rf /tmp/package /tmp/*.tgz
```
### Loading
In `base.html`, before all application scripts:
```html
<script type="module">
import { h, render, Component } from '/vendor/preact/preact.module.js';
import { useState, useEffect, useRef, useMemo, useCallback }
from '/vendor/preact/hooks.module.js';
import htm from '/vendor/preact/htm.module.js';
const html = htm.bind(h);
// Expose globally for non-module scripts and surfaces
window.preact = { h, render, Component };
window.hooks = { useState, useEffect, useRef, useMemo, useCallback };
window.html = html;
</script>
```
Application files continue to load as `<script type="module">` tags.
They access `html`, `hooks.*`, and `preact.*` from the global scope.
This avoids needing a bundler while keeping the ergonomics clean.
Surfaces (including extension packages) use the same globals. A surface
JS entry point looks like:
```js
const { useState, useEffect } = hooks;
const { render } = preact;
function MySurface() {
const [channels, setChannels] = useState([]);
useEffect(() => {
sw.api.channels.list().then(setChannels);
}, []);
return html`<div>${channels.map(c => html`<p>${c.title}</p>`)}</div>`;
}
render(html`<${MySurface} />`, document.getElementById('extension-mount'));
```
---
## 7. Backend Changes Required
### New Endpoint: `GET /api/v1/profile/permissions`
**Auth:** Authenticated user
Returns the current user's resolved permission set:
```json
{
"permissions": ["model.use", "kb.read", "channel.create", ...],
"groups": ["group-id-1", "group-id-2"]
}
```
This is the same resolution logic as `GET /admin/users/:id/permissions`
but scoped to self and available to non-admins. The SDK calls this at
boot and on token refresh to populate `sw.auth.permissions`.
### New Endpoint: `GET /api/v1/profile/bootstrap`
**Auth:** Authenticated user
Optional optimization — single call that returns everything the shell
needs at boot, avoiding a waterfall of requests:
```json
{
"user": { "id", "username", "display_name", "email", "role", "avatar" },
"permissions": ["model.use", ...],
"teams": [{ "id", "name", "my_role" }],
"settings": { "theme": "dark", ... },
"policies": { "allow_user_byok": true, ... }
}
```
This replaces the current pattern where `__USER__` and `__PAGE_DATA__`
are injected by Go templates. The shell calls `sw.auth.bootstrap()` once
at startup, then renders.
### Template Simplification
Go templates are drastically simplified. `base.html` becomes:
1. `<head>` with CSS + early theme script (keep)
2. `<body>` with a single `<div id="app">` mount point
3. Script tags for vendor libs + primitives + SDK + shell
4. Surface-specific script tag (same conditional block as today)
All the server-rendered DOM (`surface-admin`, `surface-settings`,
`surface-chat` templates with hundreds of lines of HTML) is replaced
by Preact component trees that render client-side.
The Go template engine still handles routing (which surface to load)
and injects `__SURFACE__`, `__BASE__`, `__VERSION__` globals. It no
longer generates any UI DOM.
---
## 8. Version Roadmap
```
0.37.1 Permission audit + enforcement fixes + GET /profile/permissions ✅
0.37.2 Layer 0 — UI primitives + Layer 1 Shell layout ✅
0.37.3 Layer 1 — SDK (namespaced API client, auth, RBAC, events, pipe)
0.37.4 Layer 2 — Shell (no auth gate, temp bypass for visual validation)
0.37.5 Login surface + Settings surface
0.37.6 Admin surface
0.37.7 Team Admin surface
0.37.8 Chat Pane (reusable component, NOT a surface)
0.37.9 Notes Pane (reusable component, NOT a surface)
0.37.10 Chat surface (composes ChatPane + sidebar + panels)
0.37.11 Notes surface (composes NotesPane + graph)
0.37.12 Extension surface container
0.37.13 Workflow surfaces
0.37.# Tag — functionality restored
```
**Key distinction:** ChatPane and NotesPane (0.37.8–9) are embeddable
components that surfaces compose via `sw.chat(container, opts)` and
`sw.notes(container, opts)`. The Chat surface (0.37.10) and Notes
surface (0.37.11) are the full page layouts that wire panes into
sidebars, panels, and routing.
Each version is a changeset. CI must be green before moving to the next.
Tag at the end when functionality is restored.
---
## 9. Migration Strategy
### Phase 1: Foundation (no user-visible changes)
1. Vendor Preact + htm into the Docker image
2. Build Layer 0 primitives as standalone components in `src/js/sw/`
3. Build Layer 1 SDK (new `src/js/sw/sdk.js`) alongside old SDK
4. Add `GET /profile/permissions` backend endpoint
5. Primitives test page: a hidden `/dev/primitives` route that renders
all primitives for visual verification
**Gate:** All primitives render correctly. SDK can auth, fetch, gate.
### Phase 2: Shell swap
1. Replace `base.html` with minimal shell template
2. Rewrite UserMenu as Preact component
3. Rewrite Banners as Preact components
4. Rewrite ToastContainer / DialogStack as Preact components
5. Extension surface template stays the same (it's already minimal)
**Gate:** Login → shell renders → user menu works → extension surfaces
load and function. Old surfaces (chat, admin, settings) are temporarily
broken.
### Phase 3: Surface rebuild (one at a time)
Rebuild each surface as a Preact component tree using SDK + primitives.
Order by dependency (simplest first):
1. **Settings** — self-contained, no real-time, smallest scope
2. **Admin** — self-contained, no real-time, but large (many sections)
3. **Notes** — moderate complexity, some real-time (graph)
4. **Chat** — highest complexity, real-time streaming, most critical
Each surface rebuild is an independent unit of work. The old JS files
for a surface are deleted when its replacement ships.
**Gate per surface:** All ICD runner tests pass for that surface's
endpoints. Manual smoke test of all sections/features.
### Phase 4: Cleanup
1. Delete all old JS files (`ui-admin.js`, `ui-core.js`, etc.)
2. Delete old CSS files replaced by primitives
3. Remove `sb.js` action registry (replaced by SDK)
4. Remove `API` global (internalized in SDK)
5. Remove all `window[name]` dual-writes
6. Update all Go templates to remove server-rendered DOM
---
## 10. File Structure (new)
```
src/
js/
sw/ # New — all new code lives here
vendor/ # Preact + htm (copied at build)
primitives/
menu.js
dialog.js
confirm.js
prompt.js
toast.js
banner.js
drawer.js
tabs.js
dropdown.js
tooltip.js
form-field.js
button.js
avatar.js
spinner.js
index.js # Re-exports all primitives
sdk/
client.js # Internal REST client
auth.js # Auth + token management
permissions.js # RBAC cache + sw.can()
events.js # Event bus
theme.js # Theme control
pipe.js # Filter pipeline
api/ # Namespaced domain modules
channels.js
personas.js
knowledge.js
notes.js
...
index.js # Assembles the `sw` object
shell/
app.js # Root <App> component
user-menu.js
surface-viewport.js
banner-bar.js
toast-container.js
dialog-stack.js
surfaces/
settings/
index.js
sections/
profile.js
appearance.js
providers.js
...
admin/
index.js
sections/
overview.js
users.js
teams.js
providers.js
...
chat/
index.js
... (big — own design pass)
notes/
index.js
...
css/
variables.css # KEEP — token system
primitives.css # REWRITE — styles for new primitives
shell.css # NEW — shell layout
surfaces/ # NEW — per-surface CSS
settings.css
admin.css
chat.css
notes.css
```
---
## 11. Open Questions
1. **Should the shell use client-side routing?** Currently the Go backend
handles routing (`/admin`, `/settings`, `/notes`). We could keep that
(each is a separate page load) or switch to client-side routing where
the shell persists and surfaces swap in/out without a page reload.
Client-side routing is smoother but adds complexity. The Go template
engine still needs to serve the right initial HTML either way.
2. **Chat surface scope.** Chat is ~55KB of JS (`chat.js`) plus
`chat-pane.js`, `channel-models.js`, `pane-container.js`, `panels.js`,
and significant chunks of `ui-core.js`. This is by far the largest
surface. Should it get its own detailed design pass before Phase 3?
3. **Workflow surfaces.** Workflow has its own template engine, stage
modes, and form rendering. How much of this moves into the new
primitive/SDK system vs. stays as workflow-specific code?
4. **CM6 / editor integration.** The editor surface uses esbuild-bundled
CodeMirror 6. This is already a somewhat isolated component. Does it
stay as-is with a thin Preact wrapper, or get deeper integration?
5. **Test strategy.** The current `__tests__/` directory has ~120KB of
tests (api-contracts, auth-resilience, extensions, model-processing,
policy-gating, user-journeys). These test against the old globals.
Rewrite tests in parallel, or accept a gap during transition?
---
## 12. Risks
| Risk | Mitigation |
|------|------------|
| Big-bang rewrite takes too long | Phase 1-2 can ship without breaking existing surfaces. Phase 3 is incremental per-surface. |
| Preact+htm too unfamiliar for me (Claude) to get right | htm uses tagged templates, not JSX — no build step, no transpiler surprises. Preact's API is a strict subset of React's hooks API, which is the most-documented frontend pattern in existence. |
| Old and new code coexisting | Phase 2 is the hard cut. Old surfaces can load old JS files until their Phase 3 rebuild. The shell is the only thing that must be new from Phase 2 onward. |
| Permission caching goes stale | Refresh permissions on token refresh (every 15 min). Emit `auth.permissions.changed` event so components can re-render. |
| Extension surfaces depend on old globals | Extension surfaces use `sw.*` (SDK). During transition, the SDK can expose backward-compat shims for `API`, `Events`, `Theme` globals. Shims emit deprecation warnings. |

756
docs/ARCHITECTURE.md
View File

@@ -1,590 +1,176 @@
# Architecture — Chat Switchboard v0.30.2
# Switchboard Core — Architecture
## Deployment Modes
Three Docker images support different deployment scenarios:
| Image | Dockerfile | Contents | Use Case |
|-------|-----------|----------|----------|
| **Unified** | `Dockerfile` | nginx + Go backend | Dev, docker-compose, single-node |
| **Backend** | `server/Dockerfile` | Go binary only | K8s — scale API independently |
| **Frontend** | `Dockerfile.frontend` | nginx + static files + CM6 bundle | K8s — scale FE independently |
**Unified** bundles everything in one container (4-stage Docker build: vendor libs → CM6 bundle → Go backend → nginx runtime). Nginx serves static files and proxies `/api/*` and `/ws` to the Go backend running on `:8080`. Good for development and small deployments.
**Split (Backend + Frontend)** for production K8s: Ingress routes `/api/*` and `/ws` to the backend Service, everything else to the frontend Service. The frontend entrypoint (`docker-entrypoint-fe.sh`) handles `BASE_PATH` injection into `index.html` and dynamic nginx config generation at startup. Supports branding volume mounts at `/branding/`.
```
┌─────────────────────────┐
│ Ingress / Traefik │
│ ├─ /api/* → be-svc:8080 │
│ ├─ /ws → be-svc:8080 │
│ └─ /* → fe-svc:80 │
└─────────────────────────┘
│ │
┌──────────▼──┐ ┌───────▼────────┐
│ Backend │ │ Frontend │
│ (Go :8080) │ │ (nginx :80) │
│ replicas:N │ │ replicas:M │
└──────┬──────┘ └────────────────┘
│
┌──────▼──────┐
│ PostgreSQL │ (or SQLite for
└─────────────┘ single-node)
```
Switchboard Core is a self-hosted extension platform. It provides identity,
teams, permissions, storage, workflows, notifications, and a package system.
Everything else — chat, AI providers, personas, knowledge bases, notes,
tools — ships as installable extensions.
## Design Principles
1. **Persona-as-Trust-Boundary**: A persona (model + config + prompt) is the unit of access control. Users interact with personas, not raw provider configs. Admins control which models are visible; team admins control which personas their team can use.
2. **Roles vs Teams**: Clean separation between vertical permissions (Roles: admin, user) and horizontal visibility (Teams: organizational units). A user's role determines what they can *do*; their team membership determines what they can *see*.
3. **Store Layer**: All database access goes through typed Go interfaces (`store.Stores`). Handlers never write raw SQL. The store layer has two implementations — `store/postgres/` and `store/sqlite/` — selected at startup via `DB_DRIVER`. This enables Postgres for production and SQLite for single-node or air-gapped deployments.
4. **Scope Model**: Provider configs, personas, and model settings all use a three-value `scope` column: `global` (admin-managed, visible to all), `team` (team-admin-managed, visible to team), `personal` (user-managed, visible to owner). The `owner_id` column points to the owning user or team depending on scope.
5. **Capabilities Resolution**: Model capabilities (vision, tool calling, thinking, context window) are resolved through a three-tier priority chain: catalog DB (provider API sync, per-provider authoritative) → heuristic inference (regex patterns on model ID) → admin overrides (highest priority, from `capability_overrides` table). No static model table — the same model can have different capabilities on different providers. The catalog is populated by auto-fetch on provider creation and manual refresh. Admins can correct any field via the override endpoints.
6. **Channels as Execution Context**: Channels are the universal container for conversation state — messages, tool activity, notes, and artifacts all hang off a channel. Today channels are single-user direct chats. The architecture anticipates multi-participant channels (team members, anonymous visitors, AI personas) for workflow execution, without requiring changes to the message tree, tool framework, or streaming infrastructure.
## Workflow Architecture (v0.26.0 — Shipped)
The platform's existing primitives (teams, personas, channels, tools, notes)
compose into a workflow engine where the channel is the execution context
that moves through defined stages.
### Data Model
```
Team
└─ Workflow (admin-defined, team-scoped or global)
├─ name, slug, description, branding (JSONB)
├─ entry_mode (authenticated | public_link)
├─ is_active, version (auto-incremented on edit)
└─ Stages[] (ordered)
├─ persona_id (which AI drives this stage)
├─ system_prompt (stage-specific override)
├─ form_template (JSONB — fields to collect)
├─ history_mode (full | summary | fresh)
└─ assignment_team_id (who can be assigned)
Channel (workflow instance)
├─ workflow_id + workflow_version + current_stage
├─ stage_data (JSONB — accumulated form data across stages)
├─ workflow_status (active | completed | stale | cancelled)
├─ participants[]
│ ├─ anonymous visitor (session token)
│ ├─ AI persona (per-stage, from workflow definition)
│ └─ assigned team member (claimed from assignment queue)
├─ messages (existing tree structure)
└─ notes (stage data persisted as channel-scoped notes)
WorkflowAssignment (queue entry)
├─ channel_id + stage + team_id
├─ assigned_to (nullable — claimed by team member)
└─ status (unassigned | claimed | completed)
```
### How Existing Primitives Map
| Existing Primitive | Workflow Role |
|-------------------|---------------|
| **Persona** (system prompt + model) | Drives a workflow stage — the AI knows what to collect, when to route |
| **Team** (members + roles) | Owns the workflow definition; members are assignable to channels |
| **Channel** (messages + tree) | Execution instance of a workflow; full conversation history |
| **Notes + Tools** | Structured data collection; `workflow_advance` tool triggers transitions, notes persist form data |
| **EventBus + WebSocket** | Real-time notifications for assignment, stage transitions, new messages |
| **Anonymous Sessions** (v0.24.3) | Visitors participate without user accounts |
| **DenyVisitor** predicate | Scopes tool availability per participant type |
### Stage Transition Flow
```
Visitor starts → Channel created (stage 0, persona bound)
│
▼
Persona collects form data via conversation
│
▼
Persona calls workflow_advance(data) → Stage notes created
│ │
▼ ▼
Next stage: new persona bound If assignment_team_id set:
(or workflow completes) → workflow_assignment created
→ team member claims
→ member + persona collaborate
```
### Key Implementation Details
- **Form template injection.** When `channelType == "workflow"`, the completion
handler loads the current stage's `form_template` JSONB and injects a system
prompt telling the persona what fields to collect and when to call
`workflow_advance`.
- **Dialect-safe stage data merge.** `MergeWorkflowStageData()` reads existing
JSON from the `stage_data` column, merges in Go, writes back. No
Postgres-specific JSONB operators — works on both Postgres and SQLite.
- **Optimistic claim lock.** `UPDATE workflow_assignments SET assigned_to = $1
WHERE id = $2 AND status = 'unassigned'` — if rows_affected == 0, another
team member already claimed it.
- **Staleness sweep.** Background goroutine (1h tick) marks workflow instances
as `stale` when `last_activity_at` exceeds `WORKFLOW_STALE_HOURS`.
## Package Structure
```
server/
├── main.go # Wiring: stores → handlers → routes
├── config/config.go # Env-based configuration
├── database/
│ ├── database.go # Connection management (PG + SQLite)
│ ├── migrate.go # Auto-migration on startup
│ └── migrations/
│ ├── 001_v016_schema.sql # Consolidated schema (PG)
│ └── 002_v017_persona_kb.sql
├── store/
│ ├── interfaces.go # Store interfaces + shared types
│ ├── postgres/ # Postgres implementations (22 stores)
│ │ ├── stores.go # NewStores() constructor
│ │ ├── provider.go # ProviderStore
│ │ ├── catalog.go # CatalogStore
│ │ ├── persona.go # PersonaStore
│ │ ├── channel.go # ChannelStore
│ │ ├── message.go # MessageStore
│ │ ├── user.go # UserStore
│ │ ├── team.go # TeamStore
│ │ ├── note.go # NoteStore
│ │ ├── knowledge.go # KnowledgeStore
│ │ ├── memory.go # MemoryStore (CRUD + recall)
│ │ ├── memory_hybrid.go # RecallHybrid (pgvector cosine)
│ │ └── ...
│ └── sqlite/ # SQLite implementations (22 stores)
│ ├── stores.go # NewStores() constructor
│ └── ... # Mirror of postgres/ with dialect adaptations
├── models/models.go # Shared domain types
├── capabilities/
│ ├── intrinsic.go # Heuristic detection + three-tier resolution (catalog → heuristic → admin override)
│ └── resolver.go # ModelsForUser() unified resolver
├── compaction/ # Conversation summarization engine
├── crypto/ # AES-256-GCM API key encryption
├── events/ # EventBus + WebSocket hub
├── extraction/ # Document text extraction pipeline
├── health/ # Provider health tracking (v0.22.0), tool health + auto-disable (v0.22.4)
├── routing/ # Policy-based request routing (v0.22.2)
│ ├── types.go # Policy, Candidate, Context, Decision types
│ ├── evaluator.go # Policy evaluation engine (4 policy types)
│ ├── evaluator_test.go # 12 tests
│ ├── fallback.go # RunWithFallback candidate dispatcher
│ └── convert.go # DB model → routing type conversion
│ ├── accumulator.go # In-memory counters, 60s flush, rate limits, tool health, auto-disable
│ └── accumulator_test.go # Concurrent recording, flush isolation, threshold tests
├── memory/ # Long-term memory extraction + scanning
│ ├── extractor.go # Conversation analysis → fact extraction
│ └── scanner.go # Background job: find channels needing extraction
├── mentions/ # @mention parsing for multi-model routing
│ └── parser.go # Parse() extracts + resolves @mentions against roster
├── notifications/ # Notification service + email transport
│ ├── service.go # Notify/NotifyMany, preference resolution, dispatch
│ ├── email.go # SMTP transport (TLS/STARTTLS, multipart MIME)
│ └── templates.go # HTML + plaintext email templates
├── handlers/ # HTTP handlers (Gin)
│ ├── auth.go # Login, register, refresh, logout
│ ├── admin.go # User/config/model management
│ ├── channels.go # Channel CRUD
│ ├── messages.go # Message CRUD + forking
│ ├── completion.go # Chat completions (SSE streaming)
│ ├── stream_loop.go # Shared streaming + tool execution loop
│ ├── capabilities.go # ModelHandler: model list + health enrichment + ResolveModelCaps (v0.22.3)
│ ├── health_admin.go # Provider health + capability override admin endpoints
│ ├── export.go # Markdown → PDF/DOCX conversion via pandoc (v0.22.4)
│ ├── routing_admin.go # Routing policy CRUD + dry-run test (v0.22.2)
│ ├── presets.go # Persona CRUD (all scopes)
│ ├── apiconfigs.go # User provider config CRUD (BYOK)
│ ├── teams.go # Team management
│ ├── notes.go # Notes CRUD + search + graph + backlinks
│ ├── knowledge.go # Knowledge base management
│ ├── memory.go # Memory CRUD + review (user + admin)
│ ├── memory_inject.go # BuildMemoryHint() for completion injection
│ ├── notifications.go # Notification CRUD + preference endpoints
│ ├── admin_email.go # Admin SMTP test email endpoint
│ ├── channel_models.go # Channel model roster CRUD (multi-model)
│ ├── model_prefs.go # User model visibility preferences
│ └── ...
├── notelinks/ # Wikilink extraction (regex → NoteLink structs)
├── knowledge/ # KB chunking, embedding, search
├── providers/ # LLM provider adapters
│ ├── provider.go # Provider interface + ExtraBody/mergeExtraBody
│ ├── registry.go # Data-driven type registry with metadata (v0.22.1)
│ ├── profile.go # Profile schemas per provider type (v0.22.1)
│ ├── hooks.go # PreRequest/PostStreamEvent transforms (v0.22.1)
│ ├── hooks_test.go # 17 tests: all hooks, schemas, merge, nil safety
│ ├── anthropic.go # Anthropic Messages API (+ extended thinking)
│ ├── openai.go # OpenAI-compatible (+ ExtraBody merge)
│ ├── openrouter.go
│ └── venice.go
├── middleware/ # Auth, admin, CORS, rate limiting
├── storage/ # Blob storage (PVC + S3)
└── tools/ # Built-in tool definitions
```
## Store Layer Pattern
Every store follows the same pattern:
```go
// Interface in store/interfaces.go
type FooStore interface {
Create(ctx context.Context, f *models.Foo) error
GetByID(ctx context.Context, id string) (*models.Foo, error)
Update(ctx context.Context, id string, patch models.FooPatch) error
Delete(ctx context.Context, id string) error
List(ctx context.Context, opts ListOptions) ([]models.Foo, int, error)
}
// Implementation in store/postgres/foo.go
type fooStore struct{ db *sql.DB }
func (s *fooStore) Create(ctx context.Context, f *models.Foo) error { ... }
```
Handlers receive `store.Stores` (a bundle of all store interfaces):
```go
type AdminHandler struct { stores store.Stores }
func (h *AdminHandler) CreateUser(c *gin.Context) {
// ...
err := h.stores.Users.Create(c.Request.Context(), user)
}
```
## Capabilities Resolution Chain
When the system needs to know what a model can do (vision? tools? thinking?):
```
1. model_catalog DB (exact match: model_id + provider_config_id)
↓ miss
2. model_catalog DB (any provider: same model_id)
↓ miss
3. Heuristic inference (name-based: "gpt-4-vision" → vision=true)
```
The `capabilities.ModelsForUser()` function combines catalog entries, team personas, and user preferences into a single unified model list for the frontend.
## Scope / Ownership Model
```
scope='global' → owner_id=NULL → Admin-managed, visible to all
scope='team' → owner_id=team.id → Team-admin-managed, visible to team
scope='personal' → owner_id=user.id → User-managed, visible to owner
```
Used by: `provider_configs`, `personas`, `model_catalog` (visibility column adds `enabled`/`disabled`/`team` states on top), `projects`.
## Personas
Personas are named AI configurations: a system prompt, base model, provider
config, and behavioral parameters (temperature, max tokens, thinking budget).
Each persona has a unique `handle` field (e.g. `veronica-sharpe`) used for
@mention routing.
**Handle Generation:** Auto-generated from name on create (`HandleFromName()`):
lowercase, spaces→hyphens, alphanumeric only, max 50 chars. Editable via the
persona form. Unique index prevents collisions.
**Scope Model:** Same as other resources — global, team, personal. Personal
personas are only visible to the creator. Team and global personas are visible
to all users with the appropriate access.
**Resolution Chain (completion):**
```
Explicit persona (req.PersonaID) → Project persona → @mention persona → dropdown selection
```
**Memory:** Personas have `memory_enabled` and `memory_extraction_prompt` fields.
When memory is enabled, the persona's memory scope is independent — facts
extracted in conversations with Persona A are not visible to Persona B.
## Projects (v0.19.0)
Projects are organizational containers that group channels, knowledge bases,
and notes. They follow the same scope model as other resources.
**KB Resolution Chain:**
```
Persona KBs → Project KBs → Channel KBs → Personal KBs
```
When a channel belongs to a project, all KBs bound to that project are
automatically included in both the system prompt hint (`BuildKBHint`) and
tool-time search (`kbsearch`). This means adding a KB to a project makes it
available to every channel in that project without per-channel configuration.
**Channel Association:** Channels have an optional `project_id` FK.
The `project_channels` junction table maintains ordered membership with a
UNIQUE constraint on `channel_id` (a channel can only belong to one project).
`AddChannel` performs an atomic move: single transaction deletes from the
old project, inserts into the new one, and updates the denormalized FK.
**Store Pattern:** `ProjectStore` interface with Postgres and SQLite
implementations following the same conventions as other stores (see
Store Layer Pattern above). Access checks via `UserCanAccess` enforce
owner-or-team-member visibility.
## Schema Migration
Migrations are handled by the backend on startup (no separate migration job):
1. Creates `schema_migrations` table if absent
2. Checks which migration files have been applied
3. Applies any new `.sql` files in order
Postgres uses consolidated schemas (`001_v016_schema.sql` + incremental). SQLite uses application-generated UUIDs and `datetime()` instead of `gen_random_uuid()` and `now()`. Both drivers share the same migration tracking table.
## Notifications (v0.20.0)
Centralized notification infrastructure with real-time WebSocket delivery
and optional email transport.
**Service Pattern:** All notification sources go through `notifications.Service`,
never insert directly. `Notify()` persists to store, checks user preferences,
pushes via WebSocket (if online), and dispatches email (if configured and
opted-in). `NotifyMany()` fans out to multiple users (best-effort).
**Preference Resolution:**
```
Specific type pref → User wildcard '*' pref → System default (in_app=true, email=false)
```
**Email Transport:** Optional SMTP via `EmailTransport`. Supports implicit TLS
(port 465) and STARTTLS (port 587). Multipart MIME (HTML + plaintext).
Templates rendered via Go `html/template` with instance branding. Async
delivery via goroutine (failures logged, never block notification creation).
Admin configures SMTP in platform settings; users toggle per-type delivery
in Settings → Notifications.
**Cleanup:** Background goroutine deletes notifications older than retention
period (default 90 days, configurable via admin settings).
## @mention Routing (v0.23.0)
Type `@handle` or `@model-id` in any chat to route the completion to a
different persona or model. No roster, group, or channel setup required.
**Resolution (`resolveMention`):** Extracts the first `@token` from message
content and queries the DB directly:
1. Persona handle — exact match against `personas.handle` (case-insensitive)
2. Persona handle — prefix match (unambiguous only)
3. Model ID — exact match against `model_catalog.model_id` (enabled + active provider)
4. Model ID — prefix match (unambiguous only)
Persona resolution returns the persona's model, provider config, and system
prompt. Model resolution returns the raw model ID and config — no persona
character, no system prompt override.
**Context Boundaries:** When @mentioning a persona, a system message is
injected before the user's message telling the target to ignore previous
personas' styles. When no @mention is used but persona responses exist in
history, a boundary tells the default model to respond in its own style.
**Participant Hint:** `buildParticipantHint()` injects a system message
listing all @mentionable personas (handle + name + description). This tells
the LLM who it can invoke, enabling AI-to-AI chaining.
**AI-to-AI Chaining:** After each completion, `chainIfMentioned()` runs
`resolveMention()` on the assistant's response. If it @mentions another
persona, a follow-up completion fires asynchronously. Result delivered via
WebSocket (`message.created`). Self-mention blocked. Depth capped at 5.
Same resolution function for user→LLM and LLM→LLM routing.
**Autocomplete (Frontend):** `ChannelModels.onInput()` triggers on `@` in
the chat input, matching against all `App.models` (enabled, non-hidden) by
persona handle, model ID, and display name. Popup shows avatar, name,
`@handle` hint, and provider. Selection inserts `@handle` into the input.
## Frontend Architecture
Vanilla JavaScript, no framework. The frontend ships as static files served by nginx. The only build step is the CM6 editor bundle, compiled at Docker build time via esbuild (IIFE output, no runtime bundler).
### File Structure
```
src/
├── index.html # Redirect stub → server-rendered surfaces
├── sw.js # Service worker (offline cache, BUILD_HASH)
├── manifest.json # PWA manifest
├── css/
│ ├── variables.css # Theme vars, reset, base styles
│ ├── layout.css # App shell, workspace, sidebar, responsive
│ ├── primitives.css # Buttons, forms, toasts, toggles, badges, icon-btn
│ ├── modals.css # Modal system, debug, command palette, confirm
│ ├── chat.css # Chat area, markdown, input, files, lightbox
│ ├── panels.css # Side panel, preview, notes, graph
│ ├── surfaces.css # Admin, settings, editor, projects, notes layouts
│ ├── splash.css # Auth splash, PWA banner
│ ├── chat-pane.css # ChatPane component styles
│ ├── pane-container.css # PaneContainer resize/tab styles
│ ├── user-menu.css # UserMenu flyout styles
│ ├── editor-surface.css # Editor surface 3-pane layout
│ ├── channel-models.css # @mention autocomplete, model roster
│ ├── memory.css # Memory settings + admin review
│ ├── notifications.css # Notification dropdown + list
│ ├── notification-prefs.css # SMTP config, notification pref table
│ ├── persona-kb.css # Persona KB picker panel
│ ├── admin-surfaces.css # Admin surface management
│ └── tool-grants.css # Persona tool grants checkbox list
├── js/
│ ├── ── Primitives (base.html, all surfaces) ──
│ ├── app-state.js # Global state (App, API refs)
│ ├── api.js # HTTP client with token refresh
│ ├── events.js # Labeled event bus + WebSocket bridge
│ ├── ui-primitives.js # esc(), componentMixin, Providers, Roles,
│ │ # renderCapBadges, renderProviderForm/List,
│ │ # renderRoleConfig, renderUsageDashboard,
│ │ # showConfirm, showPrompt, openModal/closeModal,
│ │ # Theme, mountAvatarUpload
│ ├── ui-format.js # Markdown rendering, code blocks, time formatting
│ ├── ui-core.js # DOM rendering, sidebar, navigation, toast
│ ├── pages.js # Page routing, splash, login
│ │
│ ├── ── Components (base.html, all surfaces) ──
│ ├── user-menu.js # UserMenu template+factory component
│ ├── model-selector.js # ModelSelector template+factory component
│ ├── file-tree.js # FileTree template+factory component
│ ├── code-editor.js # CodeEditor template+factory component (CM6)
│ ├── note-editor.js # NoteEditor template+factory component
│ ├── chat-pane.js # ChatPane template+factory component
│ ├── pane-container.js # PaneContainer (resizable multi-pane layout)
│ ├── debug.js # Debug panel + diagnostics
│ ├── repl.js # Browser REPL console
│ │
│ ├── ── Surface: Chat ──
│ ├── app.js # Chat surface boot, auth, WebSocket
│ ├── chat.js # Chat message send/receive, stream
│ ├── extensions.js # Extension framework + sandbox
│ ├── panels.js # Side panel registry
│ ├── ui-settings.js # Settings modal tabs (chat surface)
│ ├── ui-admin.js # Admin navigation + section routing
│ ├── admin-handlers.js # Admin CRUD + extension editors
│ ├── admin-scaffold.js # Admin section HTML scaffolding
│ ├── settings-handlers.js # User settings CRUD, command palette
│ ├── tokens.js # Input token counting + context budget
│ ├── files.js # File upload, paste-to-file, lightbox
│ ├── notes.js # Notes panel CRUD + graph + daily notes
│ ├── note-graph.js # Canvas force-directed graph visualization
│ ├── projects-ui.js # Project sidebar, context menu, DnD
│ ├── channel-models.js # Channel model roster + @mention AC
│ ├── tools-toggle.js # Tool enable/disable popup
│ ├── knowledge-ui.js # Knowledge base picker + management
│ ├── persona-kb.js # Persona KB assignment panel
│ ├── memory-ui.js # Memory settings + admin review
│ ├── notifications.js # Notification bell + dropdown
│ ├── notification-prefs.js # SMTP config + notification prefs
│ │
│ ├── ── Surface: Editor ──
│ ├── editor-surface.js # Editor 3-pane boot, workspace management
│ │
│ ├── ── Surface: Admin ──
│ ├── admin-surfaces.js # Surface management (admin only)
│ │
│ ├── ── Surface: Login ──
│ ├── pages-splash.js # Login/register splash page
│ │
│ └── __tests__/ # Node.js test suite (node --test)
├── editor/ # CM6 source (compiled at build time)
│ ├── package.json # CM6 + language mode dependencies
│ ├── build.mjs # esbuild script → IIFE bundle
│ ├── index.mjs # Bundle entrypoint (window.CM)
│ ├── chat-input.mjs # Chat input factory (markdown, code blocks)
│ ├── code-editor.mjs # Code editor factory (syntax, keybindings)
│ ├── note-editor.mjs # Note editor factory (markdown + wikilinks)
│ ├── wikilink.mjs # CM6 wikilink extension
│ └── theme.mjs # CM6 themes (dark/light)
└── vendor/ # Vendored libraries (loaded in base.html)
├── marked.min.js # Markdown renderer
├── purify.min.js # DOMPurify (XSS protection)
└── codemirror/
└── codemirror.bundle.js # CM6 bundle (~295KB min, ~90KB gzip)
```
### CM6 Integration
The CodeMirror 6 bundle provides three factory functions exposed on `window.CM`:
**`CM.chatInput(target, opts)`** — Markdown-mode editor for the chat input area. Features: auto-growing height, Enter=send / Shift+Enter=newline, WYSIWYG fenced code block decorations (visual container with monospace font and accent border), inline code shortcut (Ctrl/Cmd+E), spell check. Always uses standard keybindings.
**`CM.codeEditor(target, opts)`** — Full-featured code editor for the admin extension panel (JSON manifests, JavaScript scripts). Features: line numbers, bracket matching, search/replace, fold gutter, 10 bundled language modes. Supports Vim and Emacs keybindings via user preference (reconfigurable at runtime via compartments).
**`CM.noteEditor(target, opts)`** — Rich markdown editor for the notes panel. Features: heading size rendering (h1–h3), blockquote styling, fenced code block decorations, `[[wikilink]]` autocomplete (triggered by `[[`), wikilink chip rendering (clickable, styled by link/transclusion type), search/replace. `onLink` callback for navigating to linked notes; `linkCompleter` async callback for autocomplete results.
Both factories return a clean API: `getValue()`, `setValue()`, `focus()`, `destroy()`. The `ChatInput` abstraction in `chat.js` wraps the CM6 instance with a textarea fallback — all callsites use `ChatInput.getValue()` etc., never raw DOM access.
**Graceful degradation**: Every integration point checks `window.CM` before using CM6. If the bundle fails to load (air-gapped without the build step, CDN issues), the app falls back to native `<textarea>` with zero breakage.
### Theme System
CSS variables define the color palette in `:root` (dark, default) and `[data-theme="light"]` (light override). All UI components — including CM6 editors — reference these variables, so theme switches propagate instantly without editor reconfiguration.
The appearance settings offer three modes: Light, Dark, System. System mode listens to `prefers-color-scheme` and re-evaluates on OS theme change. Theme changes emit `theme.changed` on the EventBus; CM6 code editors toggle the `oneDark` syntax theme via compartment reconfiguration.
### Communication Pattern
`app.js` calls `API.*` methods, updates state, then calls `UI.*` methods to render. `events.js` handles real-time updates via WebSocket. No framework, no virtual DOM, no reactive bindings.
## Security Model
- **Auth**: JWT access tokens (short-lived) + refresh tokens (DB-stored, revocable)
- **Admin Bootstrap**: `SWITCHBOARD_ADMIN_USERNAME`/`PASSWORD` env vars create/update admin on every startup (K8s secret pattern)
- **API Key Storage**: Provider API keys encrypted with AES-256-GCM. Two-tier: org keys use `ENCRYPTION_KEY` env var, personal BYOK keys use per-user encryption keys (admins cannot recover)
- **Policies**: Boolean flags in `global_settings` table control registration, BYOK, team providers, etc.
- **Audit**: All admin operations logged to `audit_log` with actor, action, resource type/ID, and diff
- **Banner**: Environment classification banner configurable via admin settings (text, color, position)
- **CORS**: `CORS_ALLOWED_ORIGINS` env var controls which origins can make
cross-origin requests (HTTP and WebSocket). Comma-separated list of
allowed origins, e.g. `https://switchboard.corp,https://admin.corp`.
Behavior by environment:
- **Production** (`ENVIRONMENT=production`): if unset, defaults to
same-origin only (no `Access-Control-Allow-Origin` header). Set
explicitly to the domain(s) serving the frontend.
- **Development/test**: if unset, defaults to `*` (all origins).
- **Explicit `*`**: allowed but logs a startup warning. Not recommended
for production — restricts nothing and exposes the API to any origin.
The WebSocket upgrader's `CheckOrigin` respects the same setting.
Connections from origins not in the list are rejected at upgrade time.
- **No sensitive terminology**: Banner system avoids classification-related terms for security compliance
## File Storage
Blob storage for attachments, extracted text, and (future) knowledge base documents.
**Interface**: `storage.ObjectStore` — `Put`, `Get`, `Delete`, `DeletePrefix`, `Exists`, `Healthy`, `Stats`, `Backend`. All handlers use the interface; backend is selected at startup.
**Backends**:
| Backend | Config | Use Case |
|---------|--------|----------|
| **PVC** | `STORAGE_BACKEND=pvc` + `STORAGE_PATH` | Single-node, dev, docker-compose. Local filesystem with atomic writes (temp+rename). |
| **S3** | `STORAGE_BACKEND=s3` + `S3_ENDPOINT`, `S3_BUCKET`, credentials | Multi-node production. MinIO, Ceph RGW, AWS S3. Uses minio-go v7. |
| **Auto** | `STORAGE_BACKEND=` (empty) | Tries PVC at `STORAGE_PATH`; disables if not writable. |
**Key layout**: `attachments/{channel_id}/{attachment_id}_{filename}`. S3 prefix (`S3_PREFIX`) prepended for shared buckets.
**PVC always mounted**: Even with S3 backend, the PVC mount at `STORAGE_PATH` is needed for the extraction queue's local scratch directory (`processing/{id}/status.json`). With S3, the PVC can be small (1Gi) — only transient coordination state, not blobs.
**Admin panel**: Settings → Storage tab shows backend type, health status, file count, total size, endpoint/bucket (S3) or path (PVC), orphan detection and cleanup.
## Backward Compatibility
v0.9 maintains backward-compatible API routes:
| Old Route | New Handler | Notes |
|-----------|-------------|-------|
| `/api/v1/presets` | PersonaHandler | Returns both `personas` and `presets` keys |
| `/api/v1/api-configs` | ProviderConfigHandler | Unchanged route, new implementation |
| `/api/v1/models` | ModelHandler.ListEnabledModels | Alias for `/models/enabled` |
**KISS.** Every kernel feature earns its place by being required by two or more
extensions or by being impossible to implement outside the kernel. When in
doubt, leave it out and let an extension handle it.
JSON field rename: `api_config_id` → `provider_config_id` in channel and completion request/response bodies. Frontend updated to match.
**Store interface discipline.** All database access flows through the store
interface layer. No raw SQL in handlers, middleware, or sandbox code.
PostgreSQL and SQLite are first-class peers — every query compiles and
passes tests on both.
**Two-track execution.** Go for platform operations (auth, migrations,
package lifecycle). Starlark for custom admin/team logic (extension hooks,
workflow automation, triggers). The boundary is permanent.
**Extensions all the way down.** The platform ships with a kernel and a
package installer. Surfaces (UI pages), tools, filters, triggers, and
providers are all packages. The editor, chat, and admin UI will themselves
be installable surface packages.
## Kernel Components
### Identity & Auth
Users, refresh tokens, OIDC, mTLS, builtin password auth. The kernel owns
the user lifecycle because extensions need a stable identity to key
permissions, settings, and data against.
Auth modes: `builtin` (password), `mtls` (client cert), `oidc` (Keycloak
et al.). Mode is set at deploy time via `AUTH_MODE` env var.
### Teams & Groups
Teams provide horizontal isolation — users see only their team's resources.
Groups provide vertical permissions — what actions a user can perform.
The `Everyone` group (well-known UUID) grants baseline permissions to all
authenticated users without an explicit membership row. Current kernel
permissions: `extension.use`, `extension.install`, `workflow.create`,
`workflow.submit`, `admin.view`, `token.unlimited`.
### Packages
The unified registry for all installable content. A package is a surface
(routable UI page), an extension (Starlark hooks/tools/pipes), a library
(shared dependency), or a workflow definition. Package types:
| Type | What it provides |
|------------|-----------------------------------------------|
| `surface` | A routable page rendered in the shell viewport |
| `extension`| Starlark hooks, tools, API routes, DB tables |
| `full` | Both surface and extension |
| `workflow` | Bundled workflow definition |
| `library` | Shared code imported by other packages |
Tiers: `browser` (JS only), `starlark` (sandboxed server-side),
`sidecar` (container, future).
### Starlark Sandbox
Extensions declare capabilities in their manifest. The admin grants or
denies each capability. At runtime the sandbox injects only the modules
the extension has been granted:
- `db` — namespaced table CRUD (`ext_{pkg_id}_{table}`)
- `http` — SSRF-safe outbound HTTP
- `notifications` — push notifications to users
- `secrets` — read connection credentials from the vault
- `api` — register extension HTTP routes (`/s/:slug/api/*path`)
The sandbox cannot spawn goroutines, access the filesystem, or import
arbitrary Go packages. It runs with a CPU budget and memory ceiling.
### Workflows
Staged processes with form collection, human review, and webhooks.
Workflow definitions live in the kernel because they orchestrate teams,
permissions, and notifications — all kernel concerns.
Stages support four modes: `form_only`, `form_chat`, `review`, `custom`.
The `custom` mode delegates rendering to a surface package, proving the
extension stack end-to-end.
Workflow *instances* are being redesigned. The old channel-based model
(v0.38) is removed. New instance storage will use extension data tables
or a dedicated kernel table — TBD in v0.2.0.
### Connections
Scoped credential storage for extensions. Global (admin-managed), team,
or personal scope. Secrets are AES-256-GCM encrypted at rest using the
platform encryption key or the user's vault key (BYOK).
Connection types are declared by packages. Multiple packages can share
a connection type (e.g., both a GitHub extension and a CI extension
declare type `github`).
### Triggers (planned — v0.2.0)
Three trigger types invoke extension Starlark handlers:
| Trigger | Source | Kernel primitive |
|-----------|-------------------------|----------------------------|
| `time` | Cron expression | Ticker goroutine + dispatch |
| `webhook` | Inbound HTTP | Ext API route sugar |
| `event` | Internal event bus | Subscription registry |
All three converge to `sandbox.Call(handler, triggerContext)`. The
extension doesn't know or care how it was invoked.
Tasks (the old scheduler) are rebuilt as a Starlark extension on top
of these three primitives. This validates the extension stack and
removes ~3,400 lines from the kernel.
### Event Bus
Server-sent events to WebSocket clients. Kernel prefixes:
`user.*`, `team.*`, `workflow.*`, `notification.*`, `presence.*`,
`extension.*`, `admin.*`, `system.*`.
Extensions will subscribe to event patterns at install time (v0.2.0).
Match expressions start as exact strings, grow to globs later.
### Storage & Notifications
**Object storage**: PVC (local disk) or S3-compatible. Used by extensions
for file uploads, package assets, and blob storage.
**Notifications**: In-app notification bell with per-type preferences.
Extension packages can push notifications via the sandbox module.
## Data Layer
Dual-store: PostgreSQL (production) and SQLite (dev/test/edge).
27 kernel tables across 9 migration files per dialect:
| Migration | Tables |
|-----------|--------|
| 001_core | users, refresh_tokens, platform_policies, global_settings, user_presence, oidc_auth_state |
| 002_teams | teams, team_members, groups, group_members |
| 003_packages | packages, package_user_settings, extension_permissions, ext_data_tables |
| 004_connections | ext_connections, ext_dependencies, resource_grants |
| 005_notifications | notifications, notification_preferences |
| 006_audit | audit_log |
| 007_workflows | workflows, workflow_stages, workflow_versions |
| 008_ha | ws_tickets, rate_limit_counters |
SQLite parity rules: `boolToInt` for boolean binding, `store.NewID()` for
INSERT RETURNING, no `NULLS FIRST`, no boolean literals, no `$N` reuse,
`database.ST()`/`database.SNT()` wrappers for time scanning.
## Frontend
Preact (3KB) + htm (tagged template literals). No build step, no bundler
(except CM6 via esbuild). IIFE/global-namespace pattern with
`sb.register()`/`sb.ns()`.
The shell loads surfaces into a viewport. Extensions use `window.html`
and `window.preact` directly. Hooks via `window.hooks`. Vendor libs
(marked.js, DOMPurify, KaTeX, CodeMirror 6) baked into the image.
## Deployment
Single Docker image: Go binary + migrations + frontend assets + vendor
libs. Kubernetes deployment with 3-node PG cluster. CI via Gitea Actions
with DaemonSet DinD runners testing both PG and SQLite pipelines.
Registry: `registry.gobha.me:5000/xcaliber/switchboard-core`
Namespace: `gobha-ai-chat`

532
docs/DESIGN-EXT-CONNECTIONS-LIBRARIES.md
View File

@@ -1,532 +0,0 @@
# DESIGN — Extension Connections & Library Packages
Two platform primitives. Extensions that talk to external services
need scoped credential management. Extensions that share logic and
data need a dependency mechanism with clean boundaries.
---
## Part 1: Extension Connections
### Problem
Extensions integrating with external services need credentials and
endpoint config. The current `settings` mechanism is flat key-value.
It breaks when a user works with two instances of the same service,
when teams share bot tokens alongside personal tokens, or when
multiple extensions need the same credentials.
### Design
A **connection** concept. Same scope/resolution pattern as LLM
providers — scoped CRUD, resolution chain — but generic, owned by
package manifests, and sharable across packages.
#### Manifest Declaration
```json
{
"id": "git-board",
"connections": [
{
"type": "gitea",
"label": "Gitea Instance",
"fields": {
"base_url": {"type": "url", "required": true, "label": "Server URL"},
"api_token": {"type": "secret", "required": true, "label": "API Token"},
"org": {"type": "string", "required": false, "label": "Default Org"}
},
"scopes": ["global", "team", "personal"]
},
{
"type": "github",
"label": "GitHub",
"fields": {
"api_token": {"type": "secret", "required": true, "label": "Personal Access Token"},
"org": {"type": "string", "required": false, "label": "Default Org"}
},
"scopes": ["global", "personal"]
}
]
}
```
| Field | Description |
|-------|-------------|
| `type` | Connection type identifier. Shared by name — if two packages declare `type: "gitea"`, users configure once and both packages resolve it. Prefix to namespace: `"git-board:internal"`. |
| `label` | Human-readable name for the UI. |
| `fields` | Config form schema. Types: `string`, `url`, `secret`, `number`, `boolean`, `select`. |
| `scopes` | Allowed scopes. Subset of `["global", "team", "personal"]`. |
#### Storage
```sql
CREATE TABLE ext_connections (
id TEXT PRIMARY KEY,
type TEXT NOT NULL,
package_id TEXT NOT NULL, -- declaring package (UI attribution)
scope TEXT NOT NULL, -- global | team | personal
owner_id TEXT NOT NULL, -- '' for global, team_id, or user_id
name TEXT NOT NULL, -- user label: "Work Gitea"
config TEXT NOT NULL, -- JSON, secrets encrypted at rest
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMP,
updated_at TIMESTAMP,
UNIQUE(type, scope, owner_id, name)
);
```
Secrets use the same vault pattern as `provider_configs.api_key`.
#### Resolution Chain
Personal → Team → Global. Same walk as provider resolution.
```python
conn = connections.get("gitea") # scope chain
conn = connections.get("gitea", name="Work Gitea") # explicit
all = connections.list("gitea") # all available
```
#### API Endpoints
```
# Admin (global)
POST/GET/PUT/DELETE /api/v1/admin/connections[/:id]
# Team
POST/GET/PUT/DELETE /api/v1/teams/:teamId/connections[/:id]
# Personal
POST/GET/PUT/DELETE /api/v1/connections[/:id]
# Resolution
GET /api/v1/connections/resolve?type=gitea&name=Work+Gitea
```
#### UI
Three surfaces matching the provider config pattern:
- **Admin → Connections** — global. Type picker from installed packages.
- **Team Admin → Connections** — team-scoped.
- **Settings → Connections** — personal.
---
## Part 2: Library Packages
### Problem
Extensions duplicate everything. Two packages talking to Gitea write
the same API client, the same auth logic, the same pagination. If
they cache data, each creates its own tables with the same schema.
There's no mechanism for sharing code, data access, or connection
types across packages.
### Core Principle
**Libraries are services, not shared databases.**
A library owns its data privately. Consumers access it through
exported Starlark functions or REST endpoints — never by querying
the library's tables directly. The library controls validation,
access patterns, schema evolution, and data integrity. Consumers
are decoupled from the physical storage.
Same principle as microservices vs shared databases: share the
schema, share the coupling. Share the API, share the contract.
### Design
A new package type: `library`. Libraries can provide:
- **Starlark functions** — server-side consumers call via `lib.load()`
- **REST endpoints** — browser-side consumers call via HTTP
- **Connection types** — inherited by consumers
- **Private DB tables** — optional, never exposed directly
- Any combination of the above
Libraries do **not** have: surfaces, LLM tools, or pipe filters.
#### Package Type Taxonomy
| Type | Surface | Starlark exports | REST endpoints | LLM tools | Private DB | Depends on libs |
|------|---------|-----------------|----------------|-----------|------------|-----------------|
| `surface` | ✓ | — | — | — | — | via REST |
| `extension` | — | — | — | ✓ | ✓ | ✓ |
| `library` | — | ✓ | ✓ | — | ✓ | ✓ |
| `full` | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
#### Library Manifest
```json
{
"id": "gitea-client",
"title": "Gitea API Client",
"type": "library",
"tier": "starlark",
"version": "1.0.0",
"description": "Shared Gitea client — connections, API, and data caching.",
"permissions": ["api.http", "db.read", "db.write"],
"exports": [
"get_repos", "sync_repos",
"get_issues", "get_issue", "create_issue", "update_issue",
"get_prs", "get_ci_status",
"search_cached_issues"
],
"api_routes": [
{"method": "GET", "path": "/repos"},
{"method": "GET", "path": "/issues"},
{"method": "GET", "path": "/issues/*"},
{"method": "POST", "path": "/issues"},
{"method": "GET", "path": "/prs"},
{"method": "GET", "path": "/ci/*"},
{"method": "POST", "path": "/sync"}
],
"connections": [
{
"type": "gitea",
"label": "Gitea Instance",
"fields": {
"base_url": {"type": "url", "required": true},
"api_token": {"type": "secret", "required": true},
"org": {"type": "string", "required": false}
},
"scopes": ["global", "team", "personal"]
}
],
"db_tables": [
{
"name": "repos",
"columns": {
"connection_id": "text", "full_name": "text",
"owner": "text", "name": "text",
"description": "text", "html_url": "text",
"open_issues": "integer", "synced_at": "text"
}
},
{
"name": "issues",
"columns": {
"connection_id": "text", "repo_full_name": "text",
"number": "integer", "title": "text", "state": "text",
"labels": "text", "assignee": "text", "body": "text",
"created_at": "text", "synced_at": "text"
}
}
],
"schema_version": 1
}
```
Physical tables: `ext_gitea_client_repos`, `ext_gitea_client_issues`.
Private to the library. The db module remains package-scoped.
`physicalTable()` is unchanged — no cross-package table access.
#### script.star (library)
Two entry points. `exports` for Starlark consumers. `on_request`
for REST consumers. Both use the same internal logic. The DB is an
implementation detail.
```python
# ═══════════════════════════════════════════
# Exported functions (Starlark consumers)
# ═══════════════════════════════════════════
def get_repos(conn):
"""Fetch repos live from Gitea API."""
url = conn["base_url"] + "/api/v1/repos/search?limit=50"
resp = http.get(url=url, headers=_auth(conn))
if int(resp["status"]) >= 400:
return None
return json.decode(resp["body"])
def sync_repos(conn):
"""Fetch repos and cache locally. Returns count."""
repos = get_repos(conn)
if repos == None:
return 0
conn_id = conn.get("id", "")
for row in db.query("repos", filters={"connection_id": conn_id}):
db.delete("repos", row["id"])
for r in repos:
db.insert("repos", {
"connection_id": conn_id,
"full_name": r.get("full_name", ""),
"owner": r.get("owner", {}).get("login", ""),
"name": r.get("name", ""),
"description": r.get("description", ""),
"html_url": r.get("html_url", ""),
"open_issues": int(r.get("open_issues_count", 0)),
"synced_at": "",
})
return len(repos)
def get_issues(conn, owner, repo, state="open"):
"""Fetch issues live from Gitea API."""
url = (conn["base_url"] + "/api/v1/repos/" + owner + "/" + repo
+ "/issues?state=" + state + "&limit=50&type=issues")
resp = http.get(url=url, headers=_auth(conn))
if int(resp["status"]) >= 400:
return None
return json.decode(resp["body"])
def search_cached_issues(conn_id, repo_full_name, state=None):
"""Query locally cached issues. No API call."""
filters = {"connection_id": conn_id, "repo_full_name": repo_full_name}
if state:
filters["state"] = state
return db.query("issues", filters=filters, order="-synced_at", limit=200)
def create_issue(conn, owner, repo, title, body="", labels=None):
"""Create issue via Gitea API."""
payload = {"title": title}
if body:
payload["body"] = body
if labels:
payload["labels"] = labels
url = conn["base_url"] + "/api/v1/repos/" + owner + "/" + repo + "/issues"
resp = http.post(url=url, body=json.encode(payload),
headers=_auth_json(conn))
if int(resp["status"]) >= 400:
return None
return json.decode(resp["body"])
# ... update_issue, get_prs, get_ci_status same pattern ...
# ═══════════════════════════════════════════
# REST endpoints (browser / external)
# ═══════════════════════════════════════════
def on_request(req):
path = req["path"]
method = req["method"]
conn = connections.get("gitea")
if not conn:
return _resp(400, {"error": "no gitea connection configured"})
if method == "GET" and path == "/repos":
return _resp(200, {"data": get_repos(conn) or []})
if method == "POST" and path == "/sync":
return _resp(200, {"synced": sync_repos(conn)})
if method == "GET" and path == "/issues":
q = req.get("query", {})
owner = _str(q.get("owner", ""))
repo = _str(q.get("repo", ""))
if owner and repo:
return _resp(200, {"data": get_issues(conn, owner, repo) or []})
return _resp(200, {"data": db.query("issues", order="-synced_at", limit=200)})
return _resp(404, {"error": "not found"})
def _auth(conn):
return {"Authorization": "token " + conn.get("api_token", "")}
def _auth_json(conn):
return {"Authorization": "token " + conn.get("api_token", ""),
"Content-Type": "application/json"}
def _resp(status, data):
return {"status": status, "body": json.encode(data),
"headers": {"Content-Type": "application/json"}}
def _str(v):
return str(v) if v != None else ""
```
The DB calls are all in the library's script. Consumers never see
the physical tables.
#### Consumer: Starlark Path
```json
{
"id": "ci-monitor",
"type": "extension",
"tier": "starlark",
"dependencies": {"gitea-client": ">=1.0.0"},
"tools": [{"name": "check_ci", "description": "Check CI status",
"parameters": {"owner": {"type": "string", "required": true},
"repo": {"type": "string", "required": true},
"ref": {"type": "string", "required": true}}}]
}
```
```python
gitea = lib.load("gitea-client")
def on_tool_call(tool_name, params):
conn = connections.get("gitea")
if tool_name == "check_ci":
return gitea.get_ci_status(conn, params["owner"],
params["repo"], params["ref"])
return {"error": "unknown tool"}
```
No `db_tables`. No `connections` declaration. No `permissions` for
db access. The consumer calls functions.
#### Consumer: REST Path (browser-only)
A pure `type: "surface"` package calls the library's REST endpoints:
```js
// git-board/js/main.js — no Starlark, no lib.load
var repos = await fetch(BASE + '/s/gitea-client/api/repos', {
headers: { 'Authorization': 'Bearer ' + token }
}).then(r => r.json());
await fetch(BASE + '/s/gitea-client/api/sync', {
method: 'POST',
headers: { 'Authorization': 'Bearer ' + token }
});
```
This means `git-board` can be `type: "surface"` (browser-only).
Tools come from the library. Data comes from the library's REST API.
Connections come from the library. The surface just renders.
#### Permission Model
Libraries have their **own** permissions, granted by the admin:
- `gitea-client` gets `api.http` + `db.read` + `db.write`
- `ci-monitor` gets nothing beyond its own tools
When `ci-monitor` calls `gitea.get_ci_status(conn, ...)`, the
library function runs with the **library's** permissions. The
library can make HTTP calls and write to its own tables regardless
of what the consumer has.
The admin trusts the library (by granting permissions). The consumer
trusts the library (by declaring a dependency). The library trusts
nobody — it validates inputs and controls its own data.
#### `lib.load()` Mechanics
1. Check `ext_dependencies`: caller must declare dependency.
2. Load library's `script.star`.
3. Build library's module set using **library's own** permissions.
4. Execute in sandboxed context.
5. Extract globals listed in `exports`.
6. Return as frozen `starlarkstruct.Struct`.
7. Cache per sandbox invocation.
The library's `db` module resolves to `ext_gitea_client_*`.
The consumer's `db` module (if any) resolves to `ext_ci_monitor_*`.
No cross-contamination. `physicalTable()` unchanged.
#### Dependencies
```sql
CREATE TABLE ext_dependencies (
consumer_id TEXT NOT NULL,
library_id TEXT NOT NULL,
version_spec TEXT NOT NULL,
resolved_ver TEXT NOT NULL,
PRIMARY KEY (consumer_id, library_id),
FOREIGN KEY (consumer_id) REFERENCES package_registry(id)
ON DELETE CASCADE,
FOREIGN KEY (library_id) REFERENCES package_registry(id)
ON DELETE RESTRICT
);
```
`ON DELETE RESTRICT`: cannot uninstall a library with consumers.
Libraries can depend on libraries. Circular deps rejected at install.
#### Schema Migrations
Libraries own their migrations. Consumers are unaffected. If a
migration changes internal tables, the library's exported functions
adapt. Private schema is NOT part of the API surface.
#### Breaking Changes
**API surface = exports + REST endpoints + connection types.**
- Remove/rename an export or endpoint → major version bump.
- Change a function's return shape → major version bump.
- Change connection type fields → major version bump.
- Add new exports, endpoints, fields → minor version bump.
- Internal DB changes, bug fixes → patch version bump.
Enforcement deferred. Social contract for now.
---
## The Full Stack
```
┌──────────────────────────────────────────────┐
│ gitea-client (library) │
│ │
│ connections: gitea │
│ exports: get_repos, sync_repos, ... │
│ REST: /repos, /issues, /ci/*, /sync │
│ private DB: repos, issues (untouchable) │
│ │
│ permissions: api.http, db.read, db.write │
└────────┬────────────────┬────────────────┬───┘
│ │ │
lib.load() lib.load() fetch()
│ │ │
┌─────┴──────┐ ┌──────┴───────┐ ┌────┴──────┐
│ ci-monitor │ │ code-review │ │ git-board │
│ extension │ │ extension │ │ surface │
│ (starlark) │ │ (starlark) │ │ (browser) │
└────────────┘ └──────────────┘ └───────────┘
```
One library. N consumers. Zero schema coupling.
---
## Implementation Phases
### Phase 1: Extension Connections
Prerequisite: none.
- `ext_connections` table + store + handlers + UI
- `connections` Starlark module
- Three management surfaces (admin / team / personal)
### Phase 2: Library Packages
Prerequisite: none. Independent of Phase 1.
- `library` package type in installer
- `ext_dependencies` table
- `lib.load()` with per-library permission context
- Library `api_routes` (already works)
- Dependency resolution + uninstall protection
- Admin UI: dependency tree
### Phase 3: Full Composition
Prerequisite: Phase 1 + 2.
- Libraries declare connection types for consumers
- Build `gitea-client` reference library
- Migrate `git-board` to library-backed surface
---
## Not Covered
- **OAuth flows.** Static credentials only.
- **Package registry / marketplace.** `.pkg` archives, no central registry.
- **Runtime version pinning.** Installed version wins.
- **Cross-package events.** Functions and REST, not event subscriptions.
- **Library UI.** No surface. Config via package settings + connections.

456
docs/DESIGN-MULTI-FILE-STARLARK.md
View File

@@ -1,456 +0,0 @@
# DESIGN — Multi-File Starlark Packages
Promotes Starlark scripts from a single inline blob to a proper
file tree with `load()` support. Prerequisite for library packages.
---
## Current State
Three problems:
1. **Scripts live in the DB.** The installer injects `script.star`
contents into `manifest["_starlark_script"]`. The runner reads
the script from the manifest JSONB column at runtime. This means
the entire script — no matter how large — is serialized into a
JSON string inside a JSON object inside a database row.
2. **`load()` is disabled.** The sandbox callback hard-returns an
error. There's no way to split code across files.
3. **Only static assets are extracted.** `extractableRelPath()`
allows `js/`, `css/`, `assets/` — nothing else. Starlark files
land in the DB, not on disk.
The hotfix in v0.37.14 (reading `script.star` from the archive and
injecting it as `_starlark_script`) made single-file clean but
didn't fix the underlying architecture.
---
## Design
### Archive Structure
```
gitea-client/
├── manifest.json ← metadata only, no _starlark_script
├── script.star ← entry point (required for starlark tier)
├── star/ ← optional: additional modules
│ ├── repos.star
│ ├── issues.star
│ ├── auth.star
│ └── ci.star
├── js/ ← optional: surface assets
│ └── main.js
└── css/
└── main.css
```
Convention: `script.star` is always the entry point. Submodules live
in `star/`. This parallels `js/main.js` as the JS entry point with
supporting files alongside it.
### Manifest Change
The manifest gains an optional `entry_point` field for packages that
want a different entry script name. Default: `script.star`.
```json
{
"id": "gitea-client",
"type": "library",
"tier": "starlark",
"entry_point": "script.star"
}
```
`_starlark_script` is no longer written to the manifest. The field
is still read for backward compatibility (existing packages that have
it in their DB row continue to work until reinstalled).
### Extraction
**`extractableRelPath()`** adds `star/` and bare `*.star` files:
```go
var staticPrefixes = []string{"js/", "css/", "assets/", "star/"}
func extractableRelPath(name string) string {
// Existing prefix matching for directories...
// Also extract bare .star files at archive root
base := filepath.Base(name)
if strings.HasSuffix(base, ".star") {
// Strip leading directory (package-id/) if present
if idx := strings.Index(name, "/"); idx >= 0 {
rest := name[idx+1:]
if rest == base || strings.HasPrefix(rest, "star/") {
return rest
}
}
if name == base {
return name
}
}
return ""
}
```
After install, the packages directory looks like:
```
/data/packages/gitea-client/
├── script.star
├── star/
│ ├── repos.star
│ ├── issues.star
│ ├── auth.star
│ └── ci.star
├── js/
│ └── main.js
└── css/
└── main.css
```
### Installer Changes
**`packages.go` `InstallPackage()`:**
1. Stop injecting `_starlark_script` into the manifest. Remove the
hotfix code that reads `script.star` from the archive and stuffs
it into the manifest JSON.
2. Let `extractableRelPath()` handle `.star` files naturally — they
get extracted to disk alongside `js/` and `css/`.
3. Validate: if `tier == "starlark"`, confirm the archive contains
`script.star` (or the manifest's `entry_point` value). Return
400 if missing.
### Runner Changes
**`runner.go` `ExecPackage()`:**
Replace manifest-based script loading with file-based:
```go
func (r *Runner) ExecPackage(ctx context.Context, pkg *store.PackageRegistration, rc *RunContext) (*Result, error) {
if pkg.Status != models.PackageStatusActive {
return nil, fmt.Errorf("package %q is %s, not active", pkg.ID, pkg.Status)
}
if pkg.Tier != models.ExtTierStarlark {
return nil, fmt.Errorf("package %q is tier %s, not starlark", pkg.ID, pkg.Tier)
}
// ── Load script from disk (primary) or manifest (legacy) ──
script, err := r.loadScript(pkg)
if err != nil {
return nil, err
}
modules, err := r.buildModules(ctx, pkg.ID, pkg.Manifest, rc)
if err != nil {
return nil, fmt.Errorf("failed to build modules for %q: %w", pkg.ID, err)
}
// Build package-scoped load callback
loader := r.packageLoader(pkg.ID, modules)
log.Printf(" 🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules))
return r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader)
}
func (r *Runner) loadScript(pkg *store.PackageRegistration) (string, error) {
// Primary: read from disk
if r.packagesDir != "" {
entryPoint := "script.star"
if ep, ok := pkg.Manifest["entry_point"].(string); ok && ep != "" {
entryPoint = ep
}
path := filepath.Join(r.packagesDir, pkg.ID, entryPoint)
data, err := os.ReadFile(path)
if err == nil && len(data) > 0 {
return string(data), nil
}
// Fall through to legacy
}
// Legacy: inline in manifest
script, ok := pkg.Manifest["_starlark_script"].(string)
if ok && script != "" {
return script, nil
}
return "", fmt.Errorf("package %q: no script.star on disk and no _starlark_script in manifest", pkg.ID)
}
```
### Sandbox Changes
**`sandbox.go`** gains `ExecWithLoader()` — same as `Exec()` but
accepts a `load` callback:
```go
// LoadFunc resolves load("path") calls to Starlark source.
// Returns the module's globals. The sandbox caches results per
// thread (Starlark handles this via thread.Load dedup).
type LoadFunc func(thread *starlark.Thread, module string) (starlark.StringDict, error)
func (s *Sandbox) ExecWithLoader(ctx context.Context, filename, source string,
modules map[string]starlark.Value, loader LoadFunc) (*Result, error) {
// ... same as Exec() but thread.Load = loader instead of error ...
}
```
The existing `Exec()` becomes a thin wrapper that passes a
nil/error loader for backward compatibility.
### Package-Scoped Loader
**`runner.go`** builds the load callback scoped to the package's
directory:
```go
func (r *Runner) packageLoader(pkgID string, modules map[string]starlark.Value) LoadFunc {
if r.packagesDir == "" {
return nil // no disk = no load support
}
pkgDir := filepath.Join(r.packagesDir, pkgID)
cache := make(map[string]*loadEntry) // dedup + cycle detection
return func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
// ── Security: reject traversal ──
if strings.Contains(module, "..") || filepath.IsAbs(module) {
return nil, fmt.Errorf("load: path traversal not allowed: %q", module)
}
// ── Resolve to package directory ──
resolved := filepath.Join(pkgDir, module)
if !strings.HasPrefix(filepath.Clean(resolved), filepath.Clean(pkgDir)) {
return nil, fmt.Errorf("load: path escapes package directory: %q", module)
}
// ── Must be a .star file ──
if !strings.HasSuffix(resolved, ".star") {
return nil, fmt.Errorf("load: only .star files can be loaded: %q", module)
}
// ── Cache / cycle detection ──
if entry, ok := cache[module]; ok {
if entry.loading {
return nil, fmt.Errorf("load: circular dependency: %q", module)
}
return entry.globals, nil
}
entry := &loadEntry{loading: true}
cache[module] = entry
// ── Read and execute ──
data, err := os.ReadFile(resolved)
if err != nil {
return nil, fmt.Errorf("load: %q not found in package %q", module, pkgID)
}
globals, err := starlark.ExecFile(thread, module, string(data),
r.predeclaredForLoad(modules))
if err != nil {
return nil, fmt.Errorf("load: error in %q: %w", module, err)
}
entry.globals = globals
entry.loading = false
return globals, nil
}
}
type loadEntry struct {
loading bool
globals starlark.StringDict
}
```
Key constraints:
- **Package-scoped.** Can only load files from within the package's
own directory. Path traversal (`..`) and absolute paths are rejected.
One package cannot load another package's files.
- **`.star` files only.** Cannot load `.js`, `.json`, or anything
else. This is a Starlark sandbox, not a file reader.
- **Circular dependency detection.** If A loads B loads A → error.
- **Cached per invocation.** If `script.star` and `repos.star` both
load `auth.star`, it executes once. Starlark's thread-level load
dedup handles this, plus our explicit cache.
- **Same predeclared modules.** Loaded files get the same `db`,
`http`, `json`, `settings`, `connections` modules as the entry
point. They run in the same permission context.
### Starlark Usage
```python
# script.star (entry point)
load("star/auth.star", "auth_headers", "auth_headers_json")
load("star/repos.star", "get_repos", "sync_repos")
load("star/issues.star", "get_issues", "create_issue", "cache_issue")
load("star/ci.star", "get_ci_status")
def on_request(req):
# ... dispatch to imported functions ...
def on_tool_call(tool_name, params):
# ... dispatch to imported functions ...
```
```python
# star/auth.star
def auth_headers(conn):
return {"Authorization": "token " + conn.get("api_token", "")}
def auth_headers_json(conn):
h = auth_headers(conn)
h["Content-Type"] = "application/json"
return h
```
```python
# star/repos.star
load("star/auth.star", "auth_headers")
def get_repos(conn):
url = conn["base_url"] + "/api/v1/repos/search?limit=50"
resp = http.get(url=url, headers=auth_headers(conn))
if int(resp["status"]) >= 400:
return None
return json.decode(resp["body"])
def sync_repos(conn):
repos = get_repos(conn)
# ... db.insert into private tables ...
```
Submodules can load other submodules. The dependency graph is a DAG
(cycles are rejected).
### Build Script
**`packages/build.sh`** already includes `script.star`. Add `star/`:
```bash
local dirs=""
[ -d "$dir/js" ] && dirs="$dirs js/"
[ -d "$dir/css" ] && dirs="$dirs css/"
[ -d "$dir/assets" ] && dirs="$dirs assets/"
[ -f "$dir/script.star" ] && dirs="$dirs script.star"
[ -d "$dir/star" ] && dirs="$dirs star/"
[ -d "$dir/migrations" ] && dirs="$dirs migrations/"
```
### Runner Constructor
**`runner.go`** gains `packagesDir`:
```go
type Runner struct {
sandbox *Sandbox
stores store.Stores
packagesDir string // NEW
notifier NotificationSender
resolver ProviderResolver
db *sql.DB
dbPostgres bool
}
func NewRunner(sb *Sandbox, stores store.Stores) *Runner {
return &Runner{sandbox: sb, stores: stores}
}
func (r *Runner) SetPackagesDir(dir string) {
r.packagesDir = dir
}
```
**`main.go`** wires it:
```go
starlarkRunner := sandbox.NewRunner(starlarkSandbox, stores)
starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages")
```
---
## Backward Compatibility
| Scenario | Behavior |
|----------|----------|
| Existing package with `_starlark_script` in manifest, no files on disk | Works. `loadScript()` falls through to legacy path. |
| Package installed with v0.37.14 hotfix (`script.star` injected into manifest) | Works. Same legacy path. |
| New package with `script.star` on disk, no `_starlark_script` in manifest | Works. Primary path. |
| New package with `script.star` + `star/` submodules | Works. Loader resolves `load()` calls. |
| Package with both `_starlark_script` AND `script.star` on disk | Disk wins. Manifest is legacy fallback only. |
| Package that calls `load()` but has no files on disk | Error: "load() not available (no package directory)". |
No migration needed. Old packages work as-is. New packages benefit
from the file-based path. Reinstalling an old package extracts its
`script.star` to disk (via the updated `extractableRelPath`).
---
## Security
| Threat | Mitigation |
|--------|------------|
| Path traversal (`load("../../etc/passwd")`) | `..` rejected. `filepath.Clean` + prefix check against package dir. |
| Absolute paths (`load("/etc/shadow")`) | `filepath.IsAbs` check → rejected. |
| Loading non-Starlark files (`load("js/main.js")`) | `.star` suffix required. |
| Cross-package load (`load("../other-pkg/script.star")`) | `..` rejected. Resolution is always relative to the package's own directory. |
| Circular loads (`A→B→A`) | `loadEntry.loading` flag → error on re-entry. |
| Symlink escape | `filepath.Clean` resolves symlinks. Production: packages dir should be on a non-symlink-following mount. |
| Infinite load depth | Starlark's thread step limit applies across all loaded files. A package that loads 100 files still runs under the same 1M step budget. |
---
## Changeset Plan
| CS | Scope | Files |
|----|-------|-------|
| CS1 | Extraction | `packages.go`: `extractableRelPath()` adds `star/`, `*.star`. Remove `_starlark_script` injection from installer. Add entry point validation. |
| CS2 | Sandbox | `sandbox.go`: `ExecWithLoader()`. `LoadFunc` type. Existing `Exec()` wraps with nil loader. |
| CS3 | Runner | `runner.go`: `packagesDir` field + setter. `loadScript()` disk-first. `packageLoader()` with security checks. `ExecPackage()` uses `ExecWithLoader()`. |
| CS4 | Wiring | `main.go`: `SetPackagesDir()`. `build.sh`: add `star/` to archive. |
| CS5 | Test | Integration test: install package with `script.star` + `star/` submodules, call `on_request`, verify `load()` resolved correctly. Install package with only `_starlark_script` in manifest, verify legacy path. |
CS1-CS4 can land in a single session. CS5 validates. All five are
backend-only — no FE changes.
---
## What This Enables
- **Library packages** — a `gitea-client` with `script.star` as
entry point, `star/repos.star`, `star/issues.star`, `star/ci.star`
as submodules. Clean separation of concerns.
- **Large extensions** — a workflow automation package with 20+
tool handlers doesn't have to be one 2000-line file.
- **Shared utilities** — within a single package, common helpers
(auth, validation, formatting) live in their own file and are
loaded by multiple entry points.
- **Testable modules** — each `.star` file can be loaded and
tested independently in a dev sandbox.
This does NOT enable cross-package loading. Package A cannot load
Package B's files. That's what `lib.load()` is for (see
DESIGN-EXT-CONNECTIONS-LIBRARIES.md). File-level `load()` is
intra-package. `lib.load()` is inter-package. Different mechanisms,
different trust boundaries.

454
docs/EXTENSION-SURFACES.md
View File

@@ -1,454 +0,0 @@
# Extension Surfaces — Authoring Guide
**Version:** v0.30.2
**Audience:** Developers building custom surfaces for Chat Switchboard
**Prerequisite:** Admin access to install surfaces
**See also:** [PACKAGES.md](PACKAGES.md) for the `.pkg` archive format, [SDK.md](SDK.md) for the `sw.*` API
---
## What Is an Extension Surface?
An extension surface is a custom full-page UI that runs inside the
Chat Switchboard shell. It gets its own URL (`/s/your-surface`), a
link in the sidebar, and full access to the platform's authenticated
API, theme system, and UI components.
Examples of what you can build:
- A monitoring dashboard that polls external APIs
- A custom form builder for workflow intake
- A kanban board backed by channel data
- A cost tracking view using usage data
- A project timeline visualization
Extension surfaces are installed as `.surface` archives via the admin
panel. No code changes to the platform are required.
---
## Quick Start
### 1. Create the Archive
A `.surface` file is a zip archive with this structure:
```
my-dashboard/
├── manifest.json ← required: surface metadata
├── js/
│ └── main.js ← required: entry point script
└── css/
└── main.css ← optional: styles
```
The top-level directory name must match the `id` in `manifest.json`.
### 2. Write the Manifest
```json
{
"id": "my-dashboard",
"title": "My Dashboard",
"route": "/s/my-dashboard",
"auth": "authenticated",
"layout": "single",
"version": "1.0.0",
"description": "A custom dashboard surface."
}
```
| Field | Required | Description |
|---------------|----------|-------------|
| `id` | yes | Unique identifier. Lowercase, alphanumeric + hyphens. Must match the archive's top-level directory name. |
| `title` | yes | Human-readable name shown in the sidebar nav. |
| `route` | no | URL path. Defaults to `/s/{id}` if omitted. Must start with `/s/`. |
| `auth` | no | Auth requirement. Only `"authenticated"` is supported currently. Default: `"authenticated"`. |
| `layout` | no | Layout preset. Only `"single"` is supported currently. Default: `"single"`. |
| `version` | no | Semver string for your own tracking. Not enforced. |
| `description` | no | Shown in the admin surfaces panel. |
### 3. Write the Entry Point
`js/main.js` runs after all platform scripts have loaded. Your code
mounts into the `#extension-mount` container:
```js
(function() {
'use strict';
var mount = document.getElementById('extension-mount');
if (!mount) return;
var manifest = window.__MANIFEST__ || {};
var user = window.sw?.auth?.user || {};
mount.innerHTML = '<h1>Hello, ' + esc(user.display_name || user.username) + '!</h1>';
function esc(s) {
var el = document.createElement('span');
el.textContent = s;
return el.innerHTML;
}
})();
```
### 4. Package and Install
```sh
zip -r my-dashboard.surface my-dashboard/
```
Then in the admin panel: **Admin → Surfaces → Upload** and select the
`.surface` file. The surface is immediately available — no restart
required. A link appears in the sidebar.
---
## Platform Globals
Your `main.js` runs in the full platform context. These globals are
available when your script executes:
### Window Injections
| Global | Type | Description |
|--------|------|-------------|
| `window.__MANIFEST__` | Object | Your surface's manifest with route, layout, etc. Keys are **lowercase** (JSON serialization from Go). Access as `manifest.id`, `manifest.route`, `manifest.title`. |
| `window.sw.auth.user` | Object | Authenticated user (via SDK): `{ username, display_name, role, id, email }`. Requires SDK boot. |
| `window.__BASE__` | String | URL base path (e.g. `"/dev"` or `""`). Prepend to all internal links. |
| `window.__VERSION__` | String | Platform version string. |
| `window.__SURFACE__` | String | Your surface ID. |
### API Module
`API` provides authenticated fetch wrappers. Auth headers and token
refresh are handled automatically.
```js
// GET request — returns parsed JSON
var data = await API._get('/api/v1/channels');
// POST request — body is JSON-serialized automatically
var result = await API._post('/api/v1/channels', {
name: 'my-channel',
type: 'channel'
});
// PUT request
await API._put('/api/v1/channels/' + id, { name: 'renamed' });
```
All methods return a Promise that resolves to parsed JSON. On 401,
tokens are refreshed automatically and the request is retried.
**Important:** The methods are `_get`, `_post`, `_put` — prefixed with
underscore. There is no `API.get()`.
### UI Module
`UI` provides platform UI primitives:
```js
// Toast notification — types: 'success', 'error', 'info', 'warning'
UI.toast('Operation completed', 'success');
UI.toast('Something went wrong', 'error');
// Navigate to settings
UI.openSettings('general');
```
### Theme Module
`Theme` manages dark/light mode:
```js
// Current resolved theme: 'dark' or 'light'
var mode = Theme.resolved();
// Check via DOM attribute
var isDark = document.documentElement.getAttribute('data-theme') === 'dark';
```
Your CSS should use CSS custom properties (see below) rather than
querying Theme directly. Properties update automatically on theme change.
### Events Module
`Events` is the platform event bus (WebSocket-backed):
```js
// Subscribe to events
Events.on('chat.message.created', function(data) {
console.log('New message:', data);
});
// Emit local-only events
Events.emit('my-surface.updated', { key: 'value' });
```
### ChatPane Component
If your surface needs an embedded chat, you can mount a ChatPane:
```js
var container = document.createElement('div');
container.style.height = '400px';
mount.appendChild(container);
ChatPane.create(container, { role: 'assist' });
```
---
## CSS Custom Properties
Your CSS should use the platform's CSS custom properties for consistent
theming. These update automatically when the user switches themes.
### Colors
| Property | Dark Default | Light Default | Usage |
|----------|-------------|---------------|-------|
| `--bg` | `#0e0e10` | `#f7f7fa` | Page background |
| `--bg-surface` | `#18181b` | `#ffffff` | Card/panel background |
| `--bg-raised` | `#222227` | `#eff0f3` | Elevated element background |
| `--bg-hover` | `#2a2a30` | `#e5e6eb` | Hover state background |
| `--border` | `#2e2e35` | `#d5d6dc` | Default borders |
| `--border-light` | `#3a3a42` | `#c2c3cb` | Subtle borders |
| `--text` | `#e8e8ed` | `#1a1a2e` | Primary text |
| `--text-2` | `#9898a8` | — | Secondary text |
| `--text-3` | `#6b6b7b` | — | Tertiary/muted text |
| `--accent` | `#6c9fff` | — | Accent color (links, highlights) |
| `--accent-hover` | `#84b0ff` | — | Accent hover state |
| `--accent-dim` | `rgba(108,159,255,0.12)` | — | Accent tinted background |
| `--danger` | `#ef4444` | — | Error/destructive actions |
| `--success` | `#22c55e` | — | Success indicators |
| `--warning` | `#eab308` | — | Warning indicators |
### Layout
| Property | Default | Usage |
|----------|---------|-------|
| `--radius` | `8px` | Default border radius |
| `--radius-lg` | `12px` | Large border radius |
| `--font` | `'DM Sans', ...` | Primary font stack |
| `--mono` | `'JetBrains Mono', ...` | Monospace font stack |
| `--transition` | `180ms ease` | Default transition timing |
### Example
```css
.my-card {
background: var(--bg-surface);
border: 1px solid var(--border);
border-radius: var(--radius-lg);
padding: 16px;
color: var(--text);
}
.my-card-subtitle {
color: var(--text-2);
font-size: 13px;
}
.my-card:hover {
background: var(--bg-hover);
}
```
**Do not use** `--bg-secondary`, `--text-secondary`, or
`--text-tertiary` — these do not exist. Use `--bg-surface`, `--text-2`,
and `--text-3` respectively.
---
## DOM Structure
When your surface loads, the page DOM looks like this:
```
<body data-surface="my-dashboard">
<div class="surface">
<div class="surface-inner">
<div id="extension-surface" class="extension-surface">
<div class="user-menu-wrap"> ... </div> ← platform nav
<div id="extension-mount" class="extension-mount">
← YOUR CONTENT GOES HERE
</div>
</div>
</div>
</div>
<script>window.__MANIFEST__ = {...};</script>
<!-- User available via sw.auth.user after SDK boot -->
<script src="/js/api.js"></script>
<script src="/js/ui-core.js"></script>
... platform scripts ...
<script src="/surfaces/my-dashboard/js/main.js"></script> ← YOUR SCRIPT
</body>
```
The `#extension-mount` div uses `flex: 1` and `overflow: auto`, so it
fills all available vertical space. You can set its content to anything.
---
## Admin API Reference
Surfaces are managed via the admin API. All endpoints require admin
authentication.
### Install a Surface
```
POST /api/v1/admin/surfaces/install
Content-Type: multipart/form-data
file: <.surface or .zip file>
```
The archive must contain a `manifest.json` with `id` and `title`.
Static assets (`js/`, `css/`, `assets/`) are extracted to the server's
storage directory. The surface is registered in the database and
enabled immediately.
**Response:**
```json
{
"id": "my-dashboard",
"title": "My Dashboard",
"source": "extension",
"enabled": true
}
```
### List All Surfaces
```
GET /api/v1/admin/surfaces
```
Returns all surfaces (core + extension) with their enabled state.
### Enable / Disable
```
PUT /api/v1/admin/surfaces/:id/enable
PUT /api/v1/admin/surfaces/:id/disable
```
Disabled surfaces redirect to the chat surface. The sidebar link is
hidden. Core surfaces (`chat`, `admin`) cannot be disabled.
### Uninstall
```
DELETE /api/v1/admin/surfaces/:id
```
Removes the surface registration and cleans up extracted static assets.
Core surfaces cannot be deleted.
### List Enabled (Non-Admin)
```
GET /api/v1/surfaces
```
Returns enabled surfaces with minimal info (id, title, route). Used by
the sidebar to render nav links. Available to all authenticated users.
---
## Platform CSS Classes
Your extension can use these CSS classes from the platform's
`primitives.css` without importing anything:
### Buttons
```html
<button class="btn-primary">Primary Action</button>
<button class="btn-secondary">Secondary</button>
<button class="btn-danger">Destructive</button>
<button class="btn-ghost">Ghost</button>
<button class="btn-small">Small</button>
```
### Badges
```html
<span class="badge">Default</span>
<span class="badge badge-success">Success</span>
<span class="badge badge-danger">Error</span>
<span class="badge badge-warning">Warning</span>
```
### Toast (via JS)
```js
UI.toast('Message here', 'success'); // green
UI.toast('Message here', 'error'); // red
UI.toast('Message here', 'info'); // blue
UI.toast('Message here', 'warning'); // yellow
```
---
## Tips
**Always wrap in an IIFE.** Your script shares the global scope with
platform scripts. Use `(function() { ... })();` to avoid collisions.
**Use `esc()` for user-generated content.** The platform does not
provide a global escaping function. Define your own:
```js
function esc(s) {
var el = document.createElement('span');
el.textContent = s;
return el.innerHTML;
}
```
**Prepend `__BASE__` to internal links.** The platform may be deployed
under a path prefix (e.g. `/dev`):
```js
var base = window.__BASE__ || '';
link.href = base + '/settings/general';
```
**Check for globals before using them.** If your surface might be
previewed outside the platform:
```js
if (typeof API !== 'undefined' && API._get) {
// safe to make API calls
}
if (typeof UI !== 'undefined' && UI.toast) {
UI.toast('Works!', 'success');
}
```
**Size limit:** Archive upload is capped at 50 MB. Only `js/`, `css/`,
and `assets/` directories are extracted from the archive.
**Cache busting:** Static assets are served with `?v={platform_version}`
query parameter. When the platform is upgraded, caches are invalidated
automatically.
---
## Full Example
See the included `hello-dashboard.surface` archive for a complete
working example that demonstrates:
- Manifest structure
- Mounting into `#extension-mount`
- Reading `__MANIFEST__` and `sw.auth.user`
- Using `UI.toast()` for notifications
- Using `API._get()` for authenticated requests
- Using CSS custom properties for theming
- Proper `esc()` function for XSS safety

630
docs/EXTENSIONS.md
View File

@@ -1,630 +0,0 @@
# Chat Switchboard — Extension System Specification
**Version:** 0.4
**Status:** All tiers implemented (Browser v0.11.0, Starlark v0.25.0, Sidecar v0.26.0). v0.30.2 adds workflow packages and SDK.
**See also:** [ARCHITECTURE.md](ARCHITECTURE.md), [ROADMAP.md](ROADMAP.md), [PACKAGES.md](PACKAGES.md), [SDK.md](SDK.md)
---
## 1. Philosophy
Chat Switchboard is a substrate, not an application. The core provides:
authentication, provider routing, message persistence, an event bus, and a
rendering surface. Everything else — editing, writing, cluster management,
cost tracking, custom renderers — is an extension.
The goal is that someone with a problem and some JS (or Go, or Python) can
solve it without forking the project. The "modes" — Editor, Article, Chat,
Cluster Manager — are extensions that register surfaces, tools, and event
handlers. This project doesn't have to build all of them. It has to make
them possible.
---
## 2. Extension Tiers
| | Tier 0: Browser | Tier 1: Starlark | Tier 2: Sidecar |
|---|---|---|---|
| **Runs in** | User's browser | Go server (embedded) | Separate container |
| **Language** | JavaScript | Starlark (Python subset) | Any |
| **Deployed by** | User or Admin push | Admin | Admin |
| **Latency** | Zero (client-side) | Low (in-process) | Network hop |
| **Can access** | DOM, EventBus, user context | Message data, DB reads (sandboxed) | Anything (HTTP, filesystem, network) |
| **Cannot access** | Server internals, other users' data | Network, filesystem, raw SQL | N/A (full access) |
| **Trust model** | Same-origin; user-scoped or admin-pushed | Starlark sandbox (no I/O) | Container isolation |
| **Use cases** | UI, rendering, shortcuts, client tools, modes | Routing rules, message transforms, logging | RAG, external APIs, webhooks, heavy compute |
All three tiers communicate through the EventBus. A browser extension
publishes `tool.result.{callId}` the same way a sidecar does — the bus
doesn't care where the event originated.
**Admin-pushed vs User-installed (Tier 0):**
Admin-pushed extensions load for all users (like a managed browser
extension policy). User-installed extensions load from user settings
and only affect that user's session. Both use the same runtime, same
manifest format. The difference is governance, not execution.
---
## 3. Manifest Format
Every extension, regardless of tier, is described by a manifest:
```json
{
"id": "cost-tracker",
"name": "Cost Tracker",
"version": "1.0.0",
"tier": "browser",
"author": "jeff",
"description": "Real-time token counting and cost estimation",
"permissions": [
"events:chat.message.*",
"events:model.selected",
"dom:input-area",
"storage:local"
],
"entry": "cost-tracker.js",
"hooks": {
"chat.message.send": { "priority": 10, "async": false },
"chat.message.received": { "priority": 50, "async": true }
},
"tools": [],
"surfaces": [],
"settings": {
"showInline": {
"type": "boolean",
"label": "Show cost inline",
"default": true
}
}
}
```
### 3.1 Fields
- **id**: Unique identifier. Namespaced by convention (`jeff.cost-tracker`).
- **tier**: `browser` | `starlark` | `sidecar`
- **permissions**: What the extension needs. The loader enforces these.
Undeclared access is blocked (Tier 0 via proxy, Tier 1 via sandbox,
Tier 2 via API scoping).
- **entry**: Browser: JS file path. Starlark: `.star` file. Sidecar:
endpoint URL or Docker image.
- **hooks**: EventBus events this extension subscribes to, with priority
(lower = runs first) and whether the hook is async.
- **tools**: LLM-callable tools this extension provides (see §5).
- **surfaces**: UI surfaces this extension registers (see §6).
- **settings**: User-configurable options, rendered as a form in the
extension settings UI.
---
## 4. Browser Extensions (Tier 0)
### 4.1 Lifecycle
```
Install → Load → Init → Active → Disable → Unload
```
**Install**: Admin pushes manifest + JS to server, or user adds from
settings. Stored in `extensions` table with `tier = 'browser'`.
**Load**: On page load, after `events.js` but before `app.js`, the
extension loader injects `<script>` tags for all enabled browser extensions.
Load order respects declared dependencies.
**Init**: Each extension's entry script calls `Extensions.register()`:
```js
Extensions.register({
id: 'cost-tracker',
init(ctx) {
// ctx.events — scoped EventBus (only permitted events)
// ctx.storage — scoped localStorage wrapper
// ctx.settings — this extension's settings values
// ctx.ui — DOM injection points
// ctx.api — proxied fetch() to backend (auth injected)
// ctx.model — current model ID + resolved capabilities
// ctx.user — current user info (id, username, role)
this.ctx = ctx;
ctx.events.on('chat.message.send', (msg) => {
const est = this.estimateTokens(msg.content);
ctx.ui.inject('input-area', this.renderCost(est));
});
},
destroy() {
// Cleanup: remove DOM elements, unsubscribe events
}
});
```
### 4.2 Context Object
The `ctx` object is the extension's API surface. It's scoped — an
extension that didn't declare `dom:sidebar` in permissions gets a `ctx.ui`
that throws on `inject('sidebar', ...)`.
```
ctx.events — EventBus subscribe/publish (filtered by permissions)
ctx.storage — localStorage namespace (extensions::{id}::*)
ctx.settings — Read-only settings values from manifest
ctx.ui — DOM injection into declared surfaces
ctx.api — Proxied fetch() to backend (auth headers injected)
ctx.model — Current model ID + resolved capabilities
ctx.user — Current user info (id, username, role)
```
### 4.3 Admin-Pushed vs User-Installed
- **Admin-pushed**: `extensions` row with `is_system = true`. Loaded for
all users. Users cannot disable.
- **User-installed**: Stored in user settings JSONB. Users can toggle.
- Both use the same runtime.
### 4.4 Security Model
Browser extensions run in the same origin. The security model is:
1. **Permission declaration** — extensions declare what they need; the
loader enforces it via the `ctx` proxy.
2. **Admin review** — admin-pushed extensions are implicitly trusted.
User-installed extensions get a "this extension can access: ..." prompt.
3. **Event scoping** — extensions only see events they declared in
`permissions`.
4. **CSP headers** — strict Content-Security-Policy prevents inline script
injection. Extension scripts are served from known paths only.
---
## 5. Browser-Defined Tools (The Bridge)
This is the critical innovation. The LLM tool call originates from the
provider response, arrives at the Go backend, and needs to execute in the
user's browser. The EventBus + WebSocket bridge makes this transparent.
### 5.1 The Flow
```
User sends message
→ Backend sends to LLM with tools[] from enabled extensions
→ LLM returns tool_call: { name: "read_file", args: {...} }
→ Backend checks tool registry → tool is tier:browser
→ Backend publishes via WebSocket:
event: tool.call.{callId}
data: { tool: "read_file", args: {...}, callId: "uuid" }
→ Browser extension receives event, executes tool handler
→ Extension publishes result via WebSocket:
event: tool.result.{callId}
data: { callId: "uuid", result: "file contents..." }
→ Backend receives result, feeds back to LLM as tool_result
→ LLM continues with the result
→ Response streams to user
```
### 5.2 Tool Registration
Extensions declare tools in their manifest and implement handlers:
```json
{
"tools": [
{
"name": "estimate_cost",
"description": "Estimate the token cost of a given text",
"parameters": {
"type": "object",
"properties": {
"text": { "type": "string", "description": "Text to estimate" }
},
"required": ["text"]
},
"tier": "browser"
}
]
}
```
```js
Extensions.register({
id: 'cost-tracker',
init(ctx) {
ctx.tools.handle('estimate_cost', async (args) => {
const tokens = this.estimateTokens(args.text);
return { tokens, estimated_cost: tokens * this.pricing / 1e6 };
});
}
});
```
### 5.3 Server-Side Tool Router
The completion handler maintains a tool registry. When building the
`tools[]` array for the LLM request:
```go
func (h *CompletionHandler) collectTools(userID string) []ToolSchema {
var tools []ToolSchema
// Tier 1: Starlark tools (server-side, execute inline)
tools = append(tools, h.starlarkTools()...)
// Tier 2: Sidecar tools (server-side, HTTP call)
tools = append(tools, h.sidecarTools()...)
// Tier 0: Browser tools (client-side, routed via WebSocket)
if h.hub.IsConnected(userID) {
tools = append(tools, h.browserTools(userID)...)
}
return tools
}
```
When a `tool_call` arrives and the tool is `tier: browser`:
```go
case "browser":
callId := uuid.New().String()
bus.Publish(events.Event{
Type: "tool.call." + callId,
Data: toolCallData,
Room: "user:" + userID,
})
result, err := bus.WaitFor("tool.result." + callId, 30*time.Second)
```
### 5.4 Timeout and Fallback
Browser tools have a 30-second timeout. If the browser disconnects or the
tool fails, the backend sends a `tool_result` with an error message to the
LLM so it can recover gracefully.
### 5.5 Why This Matters
An extension author can write a tool in 20 lines of JavaScript that the
LLM can call. No Go code. No container. No deployment. The cluster manager
extension defines `kubectl_get`, `ceph_status`, `node_drain` as browser
tools. The editor extension defines `read_file`, `write_file`,
`search_replace`. The LLM doesn't know or care where the tool executes.
---
## 6. Extension Surfaces
*Shipped in v0.27.0.*
Extension surfaces are full-page custom UIs installed as `.surface`
archives and served at `/s/:slug`. They run inside the platform shell
with access to the authenticated API, theme system, and UI components.
See **[EXTENSION-SURFACES.md](EXTENSION-SURFACES.md)** for the complete
authoring guide including manifest format, platform API reference, CSS
custom properties, and a worked example.
### Summary
- **Archive format:** zip with `manifest.json` + `js/main.js` + optional `css/main.css`
- **Install:** Admin panel → Surfaces → Upload (no restart required)
- **Route:** `/s/{id}` — rendered via Go templates, DB lookup at request time
- **Entry point:** `js/main.js` mounts into `#extension-mount`
- **Platform access:** `API._get()`, `UI.toast()`, `Theme.resolved()`, `Events.on()`, `ChatPane.create()`
- **Theming:** CSS custom properties from `variables.css` (`--bg`, `--bg-surface`, `--text`, `--text-2`, `--accent`, etc.)
- **Admin API:** install, enable/disable, uninstall via `/api/v1/admin/surfaces/*`
### Relationship to Other Extension Types
Extension surfaces are the **most capable** extension type. They create
entirely new pages. Other extension types are lighter-weight:
| Type | Creates a page? | Uses |
|------|----------------|------|
| **Surface** (`/s/:slug`) | Yes — full page | Custom dashboards, tools, visualizations |
| **Block renderer** (Hook 3) | No — transforms message content | Mermaid diagrams, KaTeX math, CSV tables |
| **Tool** (Hook 5) | No — LLM-invocable function | Calculator, web search, file operations |
| **Post-render** (Hook 3) | No — enhances rendered messages | Syntax highlighting, link previews |
A single extension can combine multiple types — e.g. a surface for
configuration plus a tool for LLM interaction plus a renderer for
custom output formatting.
---
## 7. Extension Loader Architecture
### 7.1 Load Order
```html
<!-- Core -->
<script src="js/events.js"></script>
<script src="js/extensions.js"></script> <!-- loader + registry -->
<!-- Extensions (injected by loader) -->
<script src="/api/v1/extensions/cost-tracker/assets/main.js"></script>
<script src="/api/v1/extensions/editor-mode/assets/main.js"></script>
<!-- App (runs after extensions registered) -->
<script src="js/api.js"></script>
<script src="js/ui.js"></script>
<script src="js/app.js"></script>
```
### 7.2 extensions.js (Core)
The extension loader and registry. ~200 lines. Responsibilities:
- Fetch enabled extensions from `/api/v1/extensions?tier=browser`
- Inject script tags in dependency order
- Provide `Extensions.register()` API
- Build scoped `ctx` objects per extension (permission enforcement)
- Manage surface activation/deactivation
- Collect browser tool schemas for the completion handler
- Route `tool.call.*` events to the correct handler
### 7.3 Backend Support
New tables (migration):
```sql
CREATE TABLE extensions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
ext_id VARCHAR(100) NOT NULL UNIQUE, -- manifest id
name VARCHAR(200) NOT NULL,
tier VARCHAR(20) NOT NULL DEFAULT 'browser',
manifest JSONB NOT NULL,
assets_path TEXT, -- filesystem path for browser JS
endpoint TEXT, -- URL for sidecar
script TEXT, -- inline Starlark source
installed_by UUID REFERENCES users(id),
is_system BOOLEAN DEFAULT false,
is_enabled BOOLEAN DEFAULT true,
scope VARCHAR(20) DEFAULT 'global', -- global, team, personal
team_id UUID REFERENCES teams(id),
created_at TIMESTAMPTZ DEFAULT now(),
updated_at TIMESTAMPTZ DEFAULT now()
);
CREATE TABLE extension_user_settings (
extension_id UUID REFERENCES extensions(id) ON DELETE CASCADE,
user_id UUID REFERENCES users(id) ON DELETE CASCADE,
settings JSONB DEFAULT '{}',
is_enabled BOOLEAN DEFAULT true,
PRIMARY KEY (extension_id, user_id)
);
```
New endpoints:
```
GET /api/v1/extensions — list enabled for current user
GET /api/v1/extensions/:id/manifest — get manifest
GET /api/v1/extensions/:id/assets/*path — serve browser JS
POST /api/v1/extensions/:id/settings — update user settings
POST /api/v1/admin/extensions — install extension
DELETE /api/v1/admin/extensions/:id — uninstall
PUT /api/v1/admin/extensions/:id — enable/disable, config
```
---
## 8. EventBus Integration
The routing table from `events/types.go` expands:
```go
var Routes = map[string]Direction{
// Core
"chat.message.*": DirBoth,
"user.presence": DirToClient,
// Tool execution
"tool.call.*": DirToClient, // Server → specific client
"tool.result.*": DirFromClient, // Client → server
// Surfaces
"surface.activated": DirLocal, // Client-only
"surface.deactivated": DirLocal,
// Extension lifecycle
"extension.loaded": DirLocal,
"extension.error": DirLocal,
// Cross-extension (cluster manager example)
"cluster.node.*": DirLocal, // Between browser extensions
"cluster.alert.*": DirBoth, // Could notify server too
}
```
Browser extensions use `Events.on()` and `Events.emit()` — the same API
the core app uses. `DirLocal` events stay in the browser. `DirBoth` and
`DirFromClient` events cross the WebSocket.
---
## 9. Built-in Tools
These ship with core because multiple modes and services depend on them:
| Tool | Tier | Description |
|---|---|---|
| `web_search` | Sidecar | Search provider abstraction (SearXNG, Brave, DuckDuckGo) |
| `url_fetch` | Server | Retrieve and extract content from a URL |
| `note_create` | Server | Create a note with title, content, folder, tags |
| `note_search` | Server | Full-text search across user's notes |
| `note_update` | Server | Update note content |
| `note_list` | Server | List notes with optional folder filter |
| `kb_search` | Server | Semantic search across knowledge bases (future) |
| `task_create` | Server | Schedule a new task (future) |
Extension-provided tools (not core, expected early extensions):
- `read_file`, `write_file`, `search_replace` (Editor mode)
- `kubectl_get`, `ceph_status`, `node_ssh` (Cluster manager)
- `generate_image` (Image gen, browser or sidecar)
- `speak_text`, `transcribe_audio` (STT/TTS, browser via Web Speech API)
---
## 10. Design Principles
1. **Extensions are first-class.** A mode or feature implemented as an
extension is indistinguishable from one built into core. No second-class
citizens.
2. **The EventBus is the spine.** Extensions don't import each other.
They publish and subscribe to events. This is how cluster manager
extensions cooperate without knowing about each other at build time.
3. **Tools are location-transparent.** The LLM sees a tool schema. It
doesn't know if the tool runs in the browser, in a Starlark sandbox,
or in a container. The routing is the platform's problem.
4. **Permissions are declared, not discovered.** An extension says what it
needs upfront. The loader enforces it. No ambient authority.
5. **The core stays small.** Auth, provider routing, message persistence,
event bus, extension loader. That's core. Everything else is an
extension — even if it ships with the project.
6. **Progressive capability.** A browser extension with zero tools and
zero surfaces is just an EventBus subscriber. Add a tool and the LLM
can call it. Add a surface and it becomes a mode. The same manifest
format scales from "show token count" to "full IDE."
---
## Appendix A: Custom Renderers
Custom renderers are the simplest useful browser extension. They intercept
message content rendering to handle specific content types — no tools, no
surfaces, just a pattern match and a render function.
```js
Extensions.register({
id: 'collapsible-code',
init(ctx) {
ctx.renderers.register('collapsible-code', {
// Match fenced code blocks with >10 lines
pattern: /^```(\w+)?\n([\s\S]{10,}?)```$/gm,
render(match, container) {
const lang = match[1] || 'text';
const code = match[2];
const lines = code.split('\n');
const details = document.createElement('details');
details.innerHTML = `
<summary>${lang} — ${lines.length} lines</summary>
<pre><code class="language-${lang}">${escapeHtml(code)}</code></pre>
`;
container.replaceWith(details);
}
});
}
});
```
Expected first-party renderer extensions (ship with core or as official):
| Renderer | Matches | Renders |
|---|---|---|
| `collapsible-code` | Code blocks > N lines | `<details>` with expand/collapse |
| `html-preview` | ```html blocks | Sandboxed iframe with live preview |
| `mermaid` | ```mermaid blocks | SVG diagram via mermaid.js |
| `latex-math` | `$...$` and `$$...$$` | Rendered math via KaTeX |
| `doc-preview` | Generated HTML/PDF content | Preview panel with download link |
These are all ~30-50 line browser extensions. They don't need tool calling
or surfaces. They just pattern-match content in rendered messages.
---
## Appendix B: Model Roles
Extensions (and core services like compaction) may need to call LLMs for
their own purposes — summarization, embedding, classification — independent
of the user's selected chat model. **Model roles** are named slots that
resolve to a specific provider+model at runtime.
### B.1 Role Definition
```
┌──────────────────────────────────────────────────────────────┐
│ Role │ Purpose │ Typical Model │
├──────────────────────────────────────────────────────────────┤
│ utility │ Summarization, routing │ gpt-4o-mini │
│ │ classification, triage │ deepseek-chat │
│ embedding │ Vector generation for │ text-embedding-3 │
│ │ KB search, note search │ nomic-embed-text │
└──────────────────────────────────────────────────────────────┘
```
> **Note:** The `generation` role (image/media) was removed in v0.10.2.
> Image generation will be extension-managed with its own provider config
> and API surface, not a named role slot.
### B.2 Configuration
Admin configures roles in global settings, with optional fallback chain:
```json
{
"model_roles": {
"utility": {
"primary": { "provider_config_id": "abc", "model_id": "gpt-4o-mini" },
"fallback": { "provider_config_id": "def", "model_id": "deepseek-chat" }
},
"embedding": {
"primary": { "provider_config_id": "abc", "model_id": "text-embedding-3-small" },
"fallback": null
}
}
}
```
Teams can override roles for their scope. Personal overrides TBD.
### B.3 Extension API
```js
// Browser extension requests a utility completion
const result = await ctx.models.complete('utility', {
messages: [{ role: 'user', content: 'Summarize this in 2 sentences: ...' }]
});
// Server-side: extension requests an embedding
vec, err := models.Embed(ctx, "embedding", "text to embed")
```
The caller doesn't know which model or provider is being used. The role
system resolves it, tries fallback if primary fails, and logs usage
against the role for cost tracking.
### B.4 Core Consumers
| Consumer | Role | Purpose |
|---|---|---|
| Compaction service | `utility` | Summarize long conversations |
| Smart routing | `utility` | Classify intent, pick best model |
| Summarize & Continue | `utility` | Conversation compaction (v0.10.2) |
| Knowledge bases | `embedding` | Generate vectors for similarity search |
| Note search (future) | `embedding` | Semantic search across notes |
| Image gen extension | _(extension-managed)_ | Uses own provider config, not a role |
### B.5 Relationship to Smart Routing
Model roles and smart routing are complementary:
- **Model roles**: "I need *a* cheap model for summarization" → resolves
to a specific provider+model based on admin config.
- **Smart routing**: "Route *this user's chat* to the best model" → policy
engine that picks based on capabilities, cost, latency, team rules.
Roles are for background/system tasks. Routing is for user-facing chat.
Both use the same provider infrastructure and cost tracking.

203
docs/ICD/README.md
View File

@@ -1,203 +0,0 @@
# ICD-API — Chat Switchboard Backend API Contract
**Version:** 0.28.3
**Updated:** 2026-03-13
**Audience:** Frontend developers, integrators, API consumers
> **Organization principle:** Each file is one domain — the complete story.
> Every endpoint that touches Knowledge Bases is in `knowledge.md`, every
> endpoint that touches Personas is in `personas.md`, etc.
## Files
| File | Domain | Endpoints |
|------|--------|-----------|
| [auth.md](auth.md) | Auth — login, register, refresh, OIDC, mTLS | 6 |
| [channels.md](channels.md) | Channels, messages, completions, participants, folders, presence | ~35 |
| [personas.md](personas.md) | Personas, persona groups, avatars, KB bindings, tool grants | ~20 |
| [knowledge.md](knowledge.md) | Knowledge bases, documents, search, channel/persona bindings | ~12 |
| [notes.md](notes.md) | Notes, wikilinks, graph, search, folders | ~10 |
| [workspaces.md](workspaces.md) | Workspaces, file ops, git, credentials | ~20 |
| [projects.md](projects.md) | Projects, channel/KB/note associations | ~14 |
| [memory.md](memory.md) | Memory extraction, review, admin | 7 |
| [providers.md](providers.md) | Provider configs, health, capabilities, routing policies | ~25 |
| [models.md](models.md) | Enabled models, user preferences | 4 |
| [notifications.md](notifications.md) | Notifications, preferences | 7 |
| [extensions.md](extensions.md) | Extensions, user settings, admin | 8 |
| [profile.md](profile.md) | User profile, avatar, password, app settings | 7 |
| [teams.md](teams.md) | Teams, members, groups, permissions, grants | ~25 |
| [admin.md](admin.md) | Platform admin — users, settings, stats, audit, usage, vault | ~25 |
| [utility.md](utility.md) | Health, export, files, storage | ~12 |
| [workflows.md](workflows.md) | Workflow definitions, stages, instances, assignments, entry | ~20 |
| [tasks.md](tasks.md) | Task definitions, runs, scheduler, team tasks | ~14 |
| [surfaces.md](surfaces.md) | Surface registry, extension surfaces | 6 |
| [websocket.md](websocket.md) | WebSocket protocol, events, rooms | — |
| [enums.md](enums.md) | All enum values, policies | — |
## Conventions
### Base URL
```
{scheme}://{host}{BASE_PATH}/api/v1
```
`BASE_PATH` is empty by default. In path-routed K8s deployments (e.g.
`/staging/`), all routes shift accordingly. The frontend reads `BASE_PATH`
from a `<meta>` tag injected by the Go template engine.
### Authentication
JWT bearer tokens. Every request to a `protected` or `admin` route requires:
```
Authorization: Bearer {access_token}
```
**Access token:** HS256 JWT, 15-minute TTL. Claims: `user_id`, `email`,
`role`, `exp`, `iat`, `jti`.
**Refresh token:** Opaque, DB-stored, 7-day TTL. Used to obtain new
access tokens without re-login.
**WebSocket fallback:** `?token={access_token}` query parameter when
the `Authorization` header isn't available.
**Cookie sync:** On every token save/refresh, the frontend writes
`sb_token` as a cookie. Go template page routes read this cookie via
`AuthOrRedirect` middleware for server-rendered surfaces.
### Auth Modes
| Mode | `AUTH_MODE` | How it works |
|------|------------|--------------|
| Builtin | `builtin` (default) | Username/password, bcrypt, JWT |
| mTLS | `mtls` | Reverse proxy cert headers, auto-provision |
| OIDC | `oidc` | Authorization code flow (Keycloak etc.) |
All three modes issue the same JWT after authentication. Downstream
middleware is auth-mode-agnostic.
### Authorization Tiers
| Tier | Middleware | Who |
|------|-----------|-----|
| Public | none | Anyone (health, login, register, public settings) |
| Session | `AuthOrSession()` | JWT user OR anonymous session cookie (workflow visitors) |
| Authenticated | `Auth()` | Any logged-in user |
| Permission | `RequirePermission(perm)` | User whose groups grant `perm` |
| Team Admin | `RequireTeamAdmin()` | Admin of the specific team |
| Platform Admin | `RequireAdmin()` | Users with `role = "admin"` |
### Error Envelope
All errors return:
```json
{ "error": "human-readable message" }
```
Standard HTTP status codes: 400 (bad request), 401 (not authenticated),
403 (not authorized), 404 (not found), 409 (conflict), 500 (server error),
502 (upstream provider failure).
### Response Envelopes
Three patterns. Every endpoint uses exactly one.
**List (returns an array)** → always wrap in `{"data": []}`:
```json
{
"data": [...]
}
```
If paginated, pagination fields sit alongside `data`:
```json
{
"data": [...],
"page": 1,
"per_page": 50,
"total": 142
}
```
Query params: `?page=1&per_page=50`. Not all list endpoints paginate —
unpaginated lists still use `{"data": [...]}`.
This is a hard rule: `data` is the only array wrapper key. No
domain-specific keys (`teams`, `members`, `configs`, etc.) for arrays.
Clients parse every list response identically.
**Single object (GET by ID, profile, health)** → return the object directly:
```json
{
"id": "uuid",
"name": "...",
...
}
```
No wrapping. The response *is* the resource.
**Composite (multiple named values, not an array)** → named keys:
```json
{
"active": 5,
"pending": 2
}
```
Used for stats, counts, status endpoints — anything returning
multiple named scalars or heterogeneous data. The key names are
domain-specific and documented per endpoint.
**Empty arrays** must serialize as `[]`, never `null`. Go handlers
must guard nil slices before serialization:
```go
if result == nil {
result = []MyType{}
}
c.JSON(http.StatusOK, gin.H{"data": result})
```
### ID Format
All resource IDs are UUIDv4 strings, generated application-side via
`store.NewID()` (Go `uuid.New()`). This ensures compatibility across
both Postgres and SQLite backends.
### Timestamps
ISO 8601 with timezone: `"2025-06-15T14:30:00Z"`. Stored as
`TIMESTAMPTZ` (Postgres) or `TEXT` (SQLite).
## Page Routes (Non-API)
Server-rendered Go template surfaces. Not REST endpoints — return HTML.
| Route | Surface | Description |
|-------|---------|-------------|
| `/login` | Login | Standalone login/register page |
| `/` | Chat | Main chat interface |
| `/chat/:chatID` | Chat | Chat with specific channel loaded |
| `/editor` | Editor | Workspace file editor |
| `/editor/:wsId` | Editor | Editor with specific workspace |
| `/notes` | Notes | Notes interface |
| `/notes/:noteId` | Notes | Notes with specific note loaded |
| `/admin` | Admin | Platform administration |
| `/admin/:section` | Admin | Admin with specific section |
| `/settings` | Settings | User settings |
| `/settings/:section` | Settings | Settings with specific section |
| `/w/:id` | Workflow Landing | Branded workflow entry page |
| `/w/:id/:slug` | Workflow Landing | Slug-suffixed workflow entry |
| `/s/:slug` | Extension Surface | Dynamic surface from registry |
All page routes (except `/login`, `/w/`) require authentication via
`AuthOrRedirect` middleware. Admin routes additionally require
`RequireAdminPage()`. Workflow landing pages use `AuthOrSession`.

251
docs/ICD/admin.md
View File

@@ -1,251 +0,0 @@
# Platform Administration
Cross-cutting admin operations that don't belong to a specific domain.
### User Management
```
GET /admin/users → { "users": [...], "total": N }
POST /admin/users ← { "username", "password", "email", "role" }
PUT /admin/users/:id/role ← { "role": "admin|user" }
PUT /admin/users/:id/active ← { "is_active": false }
DELETE /admin/users/:id
POST /admin/users/:id/reset-password ← { "password": "..." }
POST /admin/users/:id/vault/reset → destroys user's UEK (BYOK keys lost)
```
`PUT /admin/users/:id/role` refuses to demote the last remaining admin
(returns 409). `DELETE /admin/users/:id` also refuses if the target is
the last admin, and destroys the user's vault before deleting the row.
### Global Settings & Policies
Settings are key-value pairs in `global_settings`. Policies are
boolean flags that gate features.
```
GET /admin/settings → all settings
GET /admin/settings/:key → single setting
PUT /admin/settings/:key ← { "value": ... }
```
The handler auto-detects: if the value is a string and the key is a
known policy name, it writes to the `policies` table. Otherwise it
writes to `global_config` as JSON.
**Public settings** (non-admin, for FE bootstrapping):
```
GET /settings/public
```
```json
{
"banner": { "enabled": true, "text": "DEVELOPMENT", "bg": "#007a33", "fg": "#ffffff", "position": "both" },
"branding": { "instance_name": "Switchboard", "logo_url": "...", "tagline": "..." },
"has_admin_prompt": true,
"storage_configured": true,
"paste_to_file_chars": 2000,
"policies": {
"allow_registration": "true",
"allow_user_byok": "true",
"allow_user_personas": "true",
"retention_ttl_days": 0
}
}
```
Note: policy values are strings (`"true"` / `"false"`), not booleans.
`retention_ttl_days` is an integer (days). When > 0, channels using
global/team providers are archived on delete and purged after TTL.
Personal (BYOK) provider channels are always immediately deleted.
### Banner Configuration
The environment banner is stored as a single `global_config` entry
under the key `"banner"`. It's a simple object:
```json
{
"enabled": true,
"text": "DEVELOPMENT",
"position": "both|top|bottom",
"bg": "#007a33",
"fg": "#ffffff"
}
```
Set via `PUT /admin/settings/banner` with `{ "value": { ... } }`.
The admin UI provides a color picker, position selector, and preset
dropdown. The Go template base layout reads the banner from
`PageData` and renders top/bottom strips with CSS custom properties.
### Platform Stats
```
GET /admin/stats
```
```json
{
"users": 42,
"channels": 156,
"messages": 12847
}
```
### Storage & Vault
**Storage status:**
```
GET /admin/storage/status → { "backend", "healthy", "file_count", "total_bytes", ... }
GET /admin/storage/orphans → { "count": 3 }
POST /admin/storage/cleanup → removes orphaned blobs
GET /admin/storage/extraction → extraction queue status
```
**Vault:**
```
GET /admin/vault/status → { "encryption_key_set", "user_vaults_count", ... }
```
### Audit Log
```
GET /admin/audit?page=1&per_page=50&action=user.create&actor_id=uuid&resource_type=...
GET /admin/audit/actions → { "actions": ["user.create", "policy.update", ...] }
```
The list endpoint returns a paginated envelope:
```json
{
"data": [...],
"total": 128,
"page": 1,
"per_page": 50
}
```
Audit entry shape:
```json
{
"id": "uuid",
"actor_id": "uuid",
"actor_name": "admin",
"action": "user.create",
"resource_type": "user",
"resource_id": "uuid",
"metadata": "{}",
"ip_address": "10.0.0.1",
"created_at": "..."
}
```
### Usage & Pricing
```
GET /admin/usage → { "totals", "results" }
GET /admin/usage/teams/:id → { "results": [...] }
GET /admin/usage/users/:id → { "results": [...] }
GET /admin/pricing → [ ... ]
PUT /admin/pricing ← { "provider_config_id", "model_id", "input_per_m", "output_per_m" }
DELETE /admin/pricing/:provider_config_id/:model_id
```
`GET /admin/pricing` returns a bare array (no envelope).
`PUT /admin/pricing` accepts the full `PricingEntry` shape:
```json
{
"provider_config_id": "uuid",
"model_id": "gpt-4o",
"input_per_m": 2.50,
"output_per_m": 10.00,
"cache_create_per_m": 1.25,
"cache_read_per_m": 0.50,
"currency": "USD"
}
```
Pricing cannot be set for personal BYOK providers (403).
**Personal usage** (non-admin):
```
GET /usage → { "totals", "results" }
```
Scoped to the user's BYOK provider usage only.
### Roles (Platform)
```
GET /admin/roles → { "roles": [...] }
GET /admin/roles/:role
PUT /admin/roles/:role ← { "permissions": {...} }
POST /admin/roles/:role/test ← test role permissions
```
### Email Test
```
POST /admin/notifications/test-email
```
No request body required. Sends a test email to the authenticated
admin's own email address using the configured SMTP settings.
---
### Archived Channels
```
GET /admin/channels/archived → { "channels": [...], "total", "page", "per_page", "retention_ttl_days" }
DELETE /admin/channels/:id/purge → permanent delete (ignores retention)
```
Both endpoints require platform admin auth. `GET` accepts
`?page=1&per_page=50` query parameters (max 100 per page).
`DELETE` verifies the channel is archived before purging and cleans
up associated file storage blobs.
### Surfaces Admin
See [surfaces.md](surfaces.md) for the full surface management API.
```
GET /admin/surfaces → all surfaces (including disabled)
GET /admin/surfaces/:id → surface with manifest
POST /admin/surfaces/install ← multipart .surface archive
PUT /admin/surfaces/:id/enable
PUT /admin/surfaces/:id/disable
DELETE /admin/surfaces/:id
```
### Tasks Admin
See [tasks.md](tasks.md) for the full task management API.
```
GET /admin/tasks → all tasks across all users/teams
POST /admin/tasks/:id/run → force run
POST /admin/tasks/:id/kill → cancel active run
DELETE /admin/tasks/:id → delete task
```
### Notifications Admin (v0.28.6)
```
POST /admin/notifications/broadcast ← { "title", "message", "level?" }
→ { "message": "broadcast sent", "count": N }
```
Sends a `system.announcement` notification to all active users.
Also emits a `system.broadcast` WebSocket event for real-time delivery.
`level` is optional (`info` | `warning` | `critical`, default `info`).
See [notifications.md](notifications.md#admin-broadcast-v0286) for full details.

180
docs/ICD/auth.md
View File

@@ -1,180 +0,0 @@
# Auth
Three auth modes, configured via `AUTH_MODE` env var. All three issue the
same JWT after authentication — downstream middleware is auth-mode-agnostic.
## Builtin Auth (default)
Username/password authentication with bcrypt hashing.
### Register
```
POST /auth/register
```
```json
{
"username": "jdoe",
"password": "...",
"email": "jdoe@example.com",
"display_name": "Jane Doe"
}
```
Gated by `allow_registration` policy. A unique `handle` is
auto-generated from `username` (collision-safe with `-2`, `-3` suffixes).
**Response (201):** Same token shape as Login. If the user requires admin
approval (`is_active` = false), returns `201` with:
```json
{ "message": "Account created but requires admin approval", "user_id": "uuid" }
```
### Login
```
POST /auth/login
```
```json
{ "username": "jdoe", "password": "..." }
```
**Response (200):**
```json
{
"access_token": "eyJ...",
"refresh_token": "opaque-string",
"token_type": "Bearer",
"expires_in": 900,
"user": {
"id": "uuid",
"username": "jdoe",
"email": "jdoe@example.com",
"display_name": "Jane Doe",
"handle": "jdoe",
"role": "user",
"auth_source": "builtin"
}
}
```
### Refresh
```
POST /auth/refresh
```
```json
{ "refresh_token": "opaque-string" }
```
Returns new `access_token` and `refresh_token`. Old refresh token is
invalidated (rotation).
### Logout
```
POST /auth/logout
```
```json
{ "refresh_token": "opaque-string" }
```
Revokes the refresh token. Returns `{ "message": "logged out" }`.
## mTLS Auth
Mutual TLS via reverse proxy headers. The proxy terminates TLS and
forwards cert DN fields in headers. The backend auto-provisions
users on first connection.
**Configured by:** `AUTH_MODE=mtls` + `MTLS_HEADER_*` env vars.
No explicit login/register endpoints. User identity is extracted from
the TLS certificate on every request. `password_hash` is NULL for
mTLS users.
**Headers parsed:**
- `MTLS_HEADER_CN` (default: `X-SSL-Client-CN`) — common name → username
- `MTLS_HEADER_DN` (default: `X-SSL-Client-DN`) — distinguished name → metadata
- `MTLS_HEADER_VERIFY` (default: `X-SSL-Client-Verify`) — must be `SUCCESS`
- `MTLS_HEADER_FINGERPRINT` (default: `X-SSL-Client-Fingerprint`) — stable identity
On first request with a valid cert, the system creates a user with
`auth_source=mtls` and `external_id=fingerprint`.
## OIDC Auth
OpenID Connect authorization code flow. Tested with Keycloak but
compatible with any OIDC-compliant IdP.
**Configured by:** `AUTH_MODE=oidc` + `OIDC_*` env vars.
### OIDC Login (redirect)
```
GET /auth/oidc/login
```
Redirects to the IdP authorization endpoint. Stores `state` + `nonce`
in `oidc_auth_state` table (cleaned up after callback).
### OIDC Callback
```
GET /auth/oidc/callback?code=...&state=...
```
Exchanges authorization code for tokens, validates ID token signature
via JWKS, extracts claims. Auto-provisions user on first login with
`auth_source=oidc` and `external_id=sub`. Returns HTML fragment that
passes tokens to the frontend via URL fragment.
**Claim mapping:**
- `sub` → `external_id`
- `preferred_username` → `username` + `handle`
- `email` → `email`
- `name` → `display_name`
- `groups` → synced to internal groups with `source=oidc`
**Split-horizon issuer:** `OIDC_EXTERNAL_ISSUER_URL` can differ from
`OIDC_ISSUER_URL` when the IdP's internal address (Docker/K8s service)
differs from its external address.
**OIDC env vars:**
| Var | Required | Description |
|-----|----------|-------------|
| `OIDC_ISSUER_URL` | Yes | IdP issuer URL for discovery |
| `OIDC_EXTERNAL_ISSUER_URL` | No | External issuer (if different from internal) |
| `OIDC_CLIENT_ID` | Yes | Client ID |
| `OIDC_CLIENT_SECRET` | Yes | Client secret |
| `OIDC_REDIRECT_URL` | No | Callback URL (auto-derived if not set) |
## Login Page Adaptation
The `/login` page adapts by `AUTH_MODE`:
| Mode | UI |
|------|-----|
| `builtin` | Username/password form + register link |
| `mtls` | Certificate status display |
| `oidc` | "Sign in with SSO" button |
## User Identity Fields
All auth modes produce users with:
| Field | Description |
|-------|-------------|
| `auth_source` | `builtin`, `mtls`, or `oidc` |
| `external_id` | IdP subject (OIDC) or cert fingerprint (mTLS). NULL for builtin. |
| `handle` | Unique @mention handle. Auto-generated, collision-safe. |
`handle` is the canonical identifier for @mentions (replacing username
in the resolution chain since v0.24.0).

765
docs/ICD/channels.md
View File

@@ -1,765 +0,0 @@
# Channels & Conversations
The **channel** is the universal conversation container. Messages,
tool activity, attachments, and KB bindings all hang off a channel.
Channels support multiple participants — users, personas, and
sessions — making them the foundation for collaborative and
multi-model workflows.
### Channel CRUD
**List channels** — paginated, sorted by last activity.
```
GET /channels?page=1&per_page=50
```
Optional query filters: `type`, `types` (comma-separated), `folder`,
`folder_id`, `search`, `project_id` (UUID or `"none"`), `archived`.
Returns pagination envelope. Each channel:
```json
{
"id": "uuid",
"user_id": "uuid",
"title": "My Chat",
"type": "direct|dm|group|channel|workflow|service",
"ai_mode": "auto|mention_only|off",
"topic": "string|null",
"description": "",
"model": "claude-sonnet-4-20250514",
"provider_config_id": "uuid|null",
"system_prompt": "",
"is_archived": false,
"is_pinned": false,
"folder": "string|null",
"folder_id": "uuid|null",
"project_id": "uuid|null",
"workspace_id": "uuid|null",
"tags": ["tag1", "tag2"],
"settings": {},
"message_count": 0,
"unread_count": 0,
"created_at": "...",
"updated_at": "..."
}
```
**Channel types:**
| Type | Description |
|------|-------------|
| `direct` | Single-user chat (default, backward compatible) |
| `dm` | Human-to-human direct message, AI silent unless @mentioned |
| `group` | Multi-user/multi-model collaborative channel |
| `channel` | Named persistent space, configurable ai_mode |
| `workflow` | Staged channel driven by workflow definitions |
| `service` | Autonomous task execution, no human participant |
**Create channel:**
```
POST /channels
```
```json
{
"title": "New Chat",
"type": "direct",
"description": "",
"model": "claude-sonnet-4-20250514",
"provider_config_id": "uuid",
"system_prompt": "",
"folder": null,
"folder_id": null,
"tags": [],
"participants": ["user-uuid"]
}
```
On create, the authenticated user is automatically added as a
participant with `role: "owner"`. If `model` and `provider_config_id`
are provided, the channel model roster (`channel_models`) is
auto-populated with an initial entry.
`participants` is only used for `dm` type — array of exactly one
other user UUID. The handler enforces DM dedup: if a DM already
exists between the two users, the existing channel is returned
(HTTP 200, not 201).
**Get channel:**
```
GET /channels/:id
```
Returns single channel object. Accessible by the channel owner
**or** any user who is a participant in the channel (via
`channel_participants`).
**Update channel:**
```
PUT /channels/:id
```
Accepts partial updates — only fields present in the body are changed.
Same field set as create, plus `is_archived`, `is_pinned`, `settings`
(JSONB merge), `workspace_id`, `ai_mode`, `topic`, `folder_id`.
**Auth:** Owner only (keyed on `user_id` column).
**Delete channel:**
```
DELETE /channels/:id
```
**Retention policy (v0.37.14):** When `retention_ttl_days > 0`, all
channels are archived on delete and purged after the TTL. The only
exception is channels using a Personal (BYOK) provider — those are
always hard-deleted immediately.
| Provider scope | TTL > 0 | TTL = 0 |
|---------------|---------|---------|
| `personal` (BYOK) | Hard delete | Hard delete |
| `global` | Archive + purge after TTL | Hard delete |
| `team` | Archive + purge after TTL | Hard delete |
| NULL (no provider) | Archive + purge after TTL | Hard delete |
When retention applies, the channel is archived (`is_archived = true`)
and stamped with `purge_after`. A background scanner purges channels
past their `purge_after` hourly. Archived channels are hidden from
the user's sidebar.
Non-owners who call DELETE are removed as participants ("leave channel")
instead of deleting.
**Response (immediate delete):** `{"message": "channel deleted"}`
**Response (retention):** `{"message": "channel archived for retention", "purge_after": "..."}`
**Response (leave):** `{"message": "left channel"}`
Hard-delete cascades: messages, files, channel_models, channel_kbs,
channel_participants. Storage cleanup runs asynchronously.
**Auth:** Owner for delete/archive; any participant for leave.
### Message Tree
Switchboard uses a **tree** model for messages, not a flat list. Each
message has a `parent_id` (NULL for root). Editing creates a sibling
branch; regeneration creates an alternative sibling. The **cursor**
tracks which child is "active" at each branch point.
**Get active path** — the primary endpoint for loading chat history:
```
GET /channels/:id/path
```
Returns the messages along the active branch from root to leaf,
following the cursor at each fork. This is what the UI renders.
```json
{
"messages": [
{
"id": "uuid",
"channel_id": "uuid",
"parent_id": "uuid|null",
"role": "user|assistant|system|tool",
"content": "...",
"model": "claude-sonnet-4-20250514|null",
"provider_config_id": "uuid|null",
"participant_id": "uuid|null",
"participant_type": "user|persona|session|null",
"thinking": "...|null",
"tool_calls": [...],
"tool_results": [...],
"attachments": [...],
"has_siblings": true,
"sibling_index": 0,
"sibling_count": 2,
"created_at": "..."
}
]
}
```
`participant_id` and `participant_type` identify who authored the
message. For `user` messages, this is the authenticated user. For
`assistant` messages, this is the persona (if set) or null for raw
model responses. Existing messages without participants return null.
**List all messages** — includes all branches, flat:
```
GET /channels/:id/messages
```
Legacy/debug endpoint. Returns every message in the channel regardless
of branch. Not used by the standard frontend.
**Create message:**
```
POST /channels/:id/messages
```
```json
{
"role": "user",
"content": "Hello",
"parent_id": "uuid|null",
"attachments": ["attachment-uuid-1"]
}
```
**Edit message** — creates a sibling at the same level:
```
POST /channels/:id/messages/:msgId/edit
```
```json
{ "content": "Revised question" }
```
Creates a new message with the same `parent_id` as the original.
Automatically updates the cursor to point to the new sibling.
**Regenerate** — creates an alternative assistant response:
```
POST /channels/:id/messages/:msgId/regenerate
```
```json
{
"model": "claude-sonnet-4-20250514",
"provider_config_id": "uuid"
}
```
Creates a new empty assistant message as a sibling of `msgId`,
then streams the completion into it. The cursor is updated.
**Update cursor** — switch to a different branch:
```
PUT /channels/:id/cursor
```
```json
{
"active_leaf_id": "uuid"
}
```
Sets which leaf message is active. The server walks the tree from
this leaf to the root and returns the full path.
Subsequent `GET /channels/:id/path` will follow the new branch.
**List siblings** — all children of a message's parent:
```
GET /channels/:id/messages/:msgId/siblings
```
Returns `{ "siblings": [...] }` with abbreviated message objects
(id, role, content preview, created_at).
### Completions & Streaming
The single most complex endpoint. Sends a message to an LLM provider
and streams the response via Server-Sent Events.
```
POST /chat/completions
```
**Request:**
```json
{
"channel_id": "uuid",
"message": "User's message text",
"model": "claude-sonnet-4-20250514",
"provider_config_id": "uuid",
"persona_id": "uuid|null",
"parent_id": "uuid|null",
"tools_enabled": true,
"tool_ids": ["web_search", "calculator"],
"attachments": ["attachment-uuid"],
"extra_body": {}
}
```
`channel_id` is required.
`extra_body` is a freeform JSON object merged into the provider request
(provider-specific parameters like `temperature`, `reasoning`, etc.).
**Participant scoping:** The requesting user must be a participant in
the channel. Messages are attributed to the authenticated user (or
session). In `group` channels, all participants see all messages in
real time via WebSocket.
**Persona resolution:** The primary completion path. If `persona_id`
is set (explicitly or via `@mention` resolution from message content),
the handler loads the persona's system prompt, model, provider config,
and KB bindings. Per-message `model`/`provider_config_id` override the
persona's defaults. In multi-persona channels, `@mention` in the
message content is parsed to resolve the target persona from the
channel's participant list.
**Routing resolution:** After config resolution, the routing policy
evaluator (`evaluateRouting()`) may redirect to a different provider
based on active policies (see §10.4). The winning provider config's
credentials are loaded and used for the actual API call.
**SSE Stream Format:**
The response is `Content-Type: text/event-stream`. Content and
reasoning deltas use OpenAI-compatible envelope format. Tool events
use named SSE events.
```
data: {"choices":[{"delta":{"content":"Hello"},"finish_reason":null}],"model":"claude-sonnet-4-20250514"}
data: {"choices":[{"delta":{"content":" world"},"finish_reason":null}],"model":"claude-sonnet-4-20250514"}
data: {"choices":[{"delta":{"reasoning_content":"Let me think..."},"finish_reason":null}],"model":"claude-sonnet-4-20250514"}
event: tool_use
data: [{"id":"call_1","name":"web_search","input":{"query":"..."}}]
event: tool_result
data: [{"id":"call_1","content":"Search results..."}]
data: {"choices":[{"delta":{"content":"Based on the search..."},"finish_reason":null}],"model":"claude-sonnet-4-20250514"}
data: {"choices":[{"delta":{},"finish_reason":"stop"}],"model":"claude-sonnet-4-20250514"}
data: [DONE]
```
| Event | `event:` field | Payload |
|-------|---------------|---------|
| Content delta | _(unnamed)_ | OpenAI envelope: `choices[0].delta.content` |
| Reasoning delta | _(unnamed)_ | OpenAI envelope: `choices[0].delta.reasoning_content` |
| Tool invocation | `tool_use` | `[{ "id", "name", "input" }]` |
| Tool result | `tool_result` | `[{ "id", "content" }]` |
| Stream complete | _(unnamed)_ | OpenAI envelope: `choices[0].finish_reason` = `"stop"` or `"budget_exceeded"` |
| End sentinel | _(unnamed)_ | `[DONE]` (literal string) |
| Error mid-stream | _(unnamed)_ | `{ "error": "message" }` |
**Tool cycle:** Tool use and tool result events can repeat up to
`max_tool_iterations` (configured server-side). The handler executes
tools, feeds results back to the model, and continues streaming.
**Response headers:**
```
X-Switchboard-Provider: providerID/configID
X-Switchboard-Route: policy-name (when routing policy is active)
X-Switchboard-Fallback: depth (when fallback chain was triggered)
```
**List available tools:**
```
GET /tools
```
Returns `{ "tools": [...] }` where each tool has `name`, `description`,
`parameters` (JSON Schema), and `category`.
### Multi-Model Roster (Raw Access)
The primary way to add AI to a channel is by adding a **persona as a
participant** (§3.7). Adding a persona automatically populates the
channel model roster with the persona's underlying model.
For the rare case where a user needs raw model access without a
persona wrapper, the `channel_models` table provides direct model
roster management. This is the 1% escape hatch — most users should
never need it.
```
GET /channels/:id/models → { "models": [...] }
POST /channels/:id/models ← { "model": "...", "provider_config_id": "...", "display_name": "..." }
PATCH /channels/:id/models/:modelId ← { "display_name": "..." }
DELETE /channels/:id/models/:modelId
```
Each roster entry:
```json
{
"id": "uuid",
"channel_id": "uuid",
"model": "claude-sonnet-4-20250514",
"provider_config_id": "uuid",
"display_name": "Claude",
"is_default": true,
"persona_id": "uuid|null",
"created_at": "..."
}
```
`persona_id`: set when this roster entry was auto-created by adding a
persona participant. Null for manually-added raw models.
**`@mention` resolution:** When a user types `@` in a channel, the
autocomplete shows persona participants first (by display name), then
raw model roster entries. The target of a `@mention` is always
resolved to a specific `provider_config_id` + `model` pair — never
an ambiguous bare model name.
### Channel Utilities
**Summarize channel** (compaction):
```
POST /channels/:id/summarize
```
Triggers conversation summarization via the utility model role.
Returns `{ "message": "summarized", "summary_id": "uuid" }`.
**Generate title:**
```
POST /channels/:id/generate-title
```
Uses the utility model to auto-generate a title from the first
exchange. Returns `{ "title": "Generated Title" }`.
### Files
All binary content associated with channels — user uploads, tool-
generated artifacts, system files — lives in the unified `files`
table backed by the ObjectStore (see §X). The old `attachments`
table and routes are removed.
**Upload (user):**
```
POST /channels/:id/files
Content-Type: multipart/form-data
```
Field: `file`. Sets `origin: "user_upload"`. Returns the file object.
**Create (tool output):** When `workspace_write` or `workspace_patch`
tools succeed during a completion, the handler auto-records a file
reference with `origin: "tool_output"` linked to the assistant message
(v0.37.18). No public endpoint — created internally by the completion
handler via the store layer.
**List by channel:**
```
GET /channels/:id/files?origin=user_upload
```
Returns `{ "files": [...] }`. Filterable by `origin`, `content_type`.
**List by message:**
```
GET /messages/:id/files
```
Returns `{ "files": [...] }`. How the frontend discovers generated
artifacts to render inline.
**List by user (file manager):**
```
GET /files?page=1&per_page=50
```
Returns `{ "files": [...], "total": N, "page": N, "per_page": N }`.
All files owned by the authenticated user, paginated.
**Get metadata:**
```
GET /files/:id
```
```json
{
"id": "uuid",
"channel_id": "uuid",
"message_id": "uuid|null",
"user_id": "uuid",
"origin": "user_upload|tool_output|system",
"filename": "report.pdf",
"content_type": "application/pdf",
"size_bytes": 1048576,
"display_hint": "inline|download|thumbnail",
"extracted_text": "...|null",
"metadata": {},
"created_at": "...",
"updated_at": "..."
}
```
**Download:**
```
GET /files/:id/download
```
Returns the file with appropriate `Content-Type` and
`Content-Disposition` headers.
**Thumbnail** _(planned — v0.28.0+):_
```
GET /files/:id/thumbnail
```
**Delete:**
```
DELETE /files/:id
```
Deletes metadata + blob. Channel deletion cascades via FK +
`DeletePrefix`.
### Channel Participants
The participant system is how users and AI interact in a channel.
**Personas are the primary way to add AI** — adding a persona as a
participant brings its full identity (name, avatar, system prompt,
model, provider config, KB bindings) into the channel. This is the
99% path. Raw model access without a persona is available via the
model roster (§3.4) for power users.
Every channel has at least one participant (the owner). `direct`
channels have exactly one user participant and optionally one or more
persona participants. `group` and `workflow` channels support multiple
participants of mixed types.
**List participants:**
```
GET /channels/:id/participants
```
Returns `{ "participants": [...] }`:
```json
{
"id": "uuid",
"channel_id": "uuid",
"participant_type": "user|persona|session",
"participant_id": "uuid",
"role": "owner|member|observer",
"display_name": "Jane Doe",
"avatar_url": "...|null",
"joined_at": "..."
}
```
**Participant types:**
| Type | Description |
|------|-------------|
| `user` | Authenticated user. `participant_id` = `users.id` |
| `persona` | AI persona added to channel. `participant_id` = `personas.id`. Brings model, system prompt, KB bindings |
| `session` | Anonymous/session participant (workflow intake). `participant_id` = session token |
**Participant roles:**
| Role | Capabilities |
|------|-------------|
| `owner` | Full control: add/remove participants, delete channel, all member capabilities |
| `member` | Send messages, trigger completions, view history |
| `observer` | Read-only: view messages, no send |
**Add participant:**
```
POST /channels/:id/participants
```
```json
{
"participant_type": "user|persona",
"participant_id": "uuid",
"role": "member"
}
```
Requires `owner` role in the channel. Adding a `persona` participant:
- Makes the persona available as an `@mention` target
- Adds the persona's model to the channel model roster automatically
- Scopes the persona's KB bindings into the channel's KB resolution chain
- The persona's avatar and display name appear in the participant list
**Update participant role:**
```
PATCH /channels/:id/participants/:participantId
```
```json
{ "role": "observer" }
```
**Remove participant:**
```
DELETE /channels/:id/participants/:participantId
```
Cannot remove the last owner. Removing a `persona` participant also
removes its auto-created model roster entry.
**Completion routing:** When `@PersonaName` appears in message
content, the completion handler resolves to that persona's model,
provider config, and system prompt. When no `@mention` is present in
a multi-persona channel, the channel's default persona (first added)
handles the response. The `persona_id` field in the completion
request (§3.3) can also be set explicitly to override `@mention`
resolution.
**Backward compatibility:** Existing `direct` channels that predate
the participant system are auto-migrated on first access — a single
`user` participant with `role: "owner"` is created from the channel's
legacy `user_id` column.
### Presence
Heartbeat-based online status. Not channel-scoped — tracks global user presence.
**Heartbeat:**
```
POST /presence/heartbeat
```
Client sends every 30 seconds. Server upserts `user_presence` row.
No request body needed. Returns `{ "ok": true }`.
**Auth:** Authenticated.
**Bulk query:**
```
GET /presence?users=uuid1,uuid2,uuid3
```
Returns online/offline status for listed user UUIDs. Threshold:
online if last heartbeat < 90 seconds ago.
```json
{
"presence": {
"uuid1": "online",
"uuid2": "offline"
}
}
```
Values are flat status strings. Capped at 100 user IDs per request.
Subsequent updates delivered via WebSocket `presence.changed` events.
**Typing indicators** are sent via `POST /channels/:id/typing`
(authenticated, broadcasts `typing.user` WebSocket event to other
user participants) and via WebSocket (see websocket.md). The typing
event payload includes `channel_id`, `user_id`, and `display_name`.
### Chat Folders
User-scoped grouping for chats. Folders have no semantic weight —
they are named drawers. The schema supports nesting via `parent_id`
(DnD nesting not yet implemented in the UI).
```
GET /folders → { "folders": [...] }
POST /folders ← { "name": "Research", "sort_order": 0 }
PUT /folders/:id ← { "name": "Renamed", "sort_order": 1 }
DELETE /folders/:id → chats become unfiled
```
**Folder object:**
```json
{
"id": "uuid",
"name": "Research",
"parent_id": "uuid|null",
"sort_order": 0,
"created_at": "...",
"updated_at": "..."
}
```
Moving a chat into/out of a folder is done via `PUT /channels/:id`
with `{ "folder_id": "uuid" }` or `{ "folder_id": "" }` to unbind.
The channel list also supports `?folder_id=uuid` as a query filter.
**Auth:** Authenticated (user-scoped).
### Channel Configuration
Additional channel fields (set via `PUT /channels/:id`):
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `ai_mode` | string | `auto` | `auto`, `mention_only`, or `off` |
| `topic` | string | null | Short description shown in header |
Fields managed internally (not settable via channel update):
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `allow_anonymous` | boolean | false | Session participants can join. Set by workflow instance handlers on channel creation. |
| `kb_auto_inject` | boolean | false | Top-K KB chunks auto-prepend to context _(v0.28.0 — schema exists, not yet wired)_. |
`ai_mode` behavioral matrix:
| Channel type | Default ai_mode | Notes |
|-------------|----------------|-------|
| `direct` | `auto` | Existing behavior |
| `dm` | `mention_only` | AI silent unless @mentioned |
| `group` | `auto` | Configurable per channel |
| `channel` | `auto` | Configurable per channel |
### Mark Read
```
POST /channels/:id/mark-read
```
Updates the user's `last_read_at` cursor. Used for unread count
calculation.
**Auth:** Authenticated.
### User Search
```
GET /users/search?q=alice
```
Returns `{ "users": [{ "id", "username", "display_name", "handle" }] }`.
Matches on username, display_name, and handle (case-insensitive
substring). Used for DM creation and participant picker. Excludes the
calling user. Max 20 results.
**Auth:** Authenticated.
---

282
docs/ICD/enums.md
View File

@@ -1,282 +0,0 @@
# Appendix: Enums & Constants
All enum values used across the API. Definitive source of truth.
## Channel Types
| Value | Description |
|-------|-------------|
| `direct` | Single-user chat (default, backward compatible) |
| `dm` | Human-to-human direct message, AI silent unless @mentioned |
| `group` | Multi-user/multi-model collaborative channel |
| `channel` | Named persistent space, configurable ai_mode |
| `workflow` | Staged channel driven by workflow definitions |
| `service` | Autonomous task execution, no human participant |
## Channel AI Modes
| Value | Description |
|-------|-------------|
| `auto` | AI responds to every message (default) |
| `mention_only` | AI responds only when @mentioned (default for `dm`) |
| `off` | AI disabled entirely |
## Participant Types
| Value | Description |
|-------|-------------|
| `user` | Authenticated user. `participant_id` = `users.id` |
| `persona` | AI persona. `participant_id` = `personas.id` |
| `session` | Anonymous visitor. `participant_id` = session token |
## Participant Roles
| Value | Description |
|-------|-------------|
| `owner` | Full control: add/remove participants, delete channel |
| `member` | Send messages, trigger completions, view history |
| `observer` | Read-only: view messages, no send |
| `visitor` | Anonymous session participant (workflow intake) |
## Presence Statuses
`online`, `away`, `offline`
## Message Roles
`user`, `assistant`, `system`, `tool`
## Scopes
`personal`, `team`, `global`
## Visibility (Model Catalog)
`enabled`, `disabled`, `team`
## User Roles
`admin`, `user`
## Team Member Roles
`admin`, `member`
## Provider Types
`openai`, `anthropic`, `openrouter`, `venice` (extensible via registry)
## Model Types
`chat`, `embedding`, `image`
## Memory Scopes
`user`, `persona`, `persona_user`
## Memory Statuses
`active`, `pending_review`, `archived`
## Workspace Owner Types
`user`, `project`, `channel`, `team`
## Workspace Statuses
`active`, `archived`, `deleting`
## File Index Statuses
`pending`, `indexing`, `ready`, `error`, `skipped`
## KB Document Statuses
`pending`, `chunking`, `embedding`, `ready`, `error`
## Provider Health Statuses
`healthy`, `degraded`, `down`
## Routing Policy Types
| Value | Description |
|-------|-------------|
| `provider_prefer` | Ordered provider fallback list |
| `team_route` | Restrict team to specific providers |
| `cost_limit` | Heuristic cost cap per request |
| `model_alias` | Alias → provider+model rewrite |
| `capability_match` | Match cheapest model with required capabilities |
## Extension Tiers
`browser` (implemented), `starlark` (implemented, v0.29.0), `sidecar` (future)
## Grant Types
| Value | Visibility |
|-------|-----------|
| `team_only` | Only the owning team |
| `global` | All authenticated users |
| `groups` | Members of specified groups |
## Resource Grant Resource Types
`persona`, `knowledge_base`, `project`
## Group Sources
| Value | Description |
|-------|-------------|
| `manual` | Admin-created |
| `oidc` | Synced from IdP groups claim |
| `system` | Platform-seeded (e.g., Everyone group) |
## Auth Sources
| Value | Description |
|-------|-------------|
| `builtin` | Username/password, bcrypt |
| `mtls` | Mutual TLS via reverse proxy |
| `oidc` | OpenID Connect (Keycloak etc.) |
## Workflow Statuses
| Value | Description |
|-------|-------------|
| `active` | Instance is running |
| `completed` | All stages finished |
| `stale` | No activity within threshold |
| `cancelled` | Manually cancelled |
## Workflow Entry Modes
`public_link`, `team_only`
## Workflow History Modes
| Value | Description |
|-------|-------------|
| `full` | Next stage sees complete chat history |
| `summary` | Utility-role summary of previous stages |
| `fresh` | Clean slate — no history carried forward |
## Task Types
`prompt`, `workflow`
## Task Run Statuses
| Value | Description |
|-------|-------------|
| `queued` | Waiting to execute |
| `running` | Currently executing |
| `completed` | Finished successfully |
| `failed` | Error during execution |
| `budget_exceeded` | Hit token/tool/wall-clock limit |
| `cancelled` | Manually killed |
## Task Scopes
`personal`, `team`, `global`
## Task Output Modes
`channel`, `note`, `webhook`
## Assignment Statuses
`unassigned`, `claimed`, `completed`
## Surface Sources
`core`, `extension`
## Notification Types
Convention: `domain.action`. Free-form strings — new types don't
require migration.
| Type | Description |
|------|-------------|
| `role.fallback` | Model role fell back to secondary provider |
| `kb.ready` | Knowledge base finished indexing |
| `kb.error` | Knowledge base indexing failed |
| `grant.changed` | User added to or removed from a group |
| `memory.extracted` | New memories extracted from conversation |
| `user.mentioned` | User was @mentioned in a channel |
| `workflow.assigned` | Workflow stage assigned to team/user |
| `workflow.claimed` | Workflow assignment claimed by a user |
| `task.completed` | Scheduled task finished successfully |
| `task.failed` | Scheduled task failed |
| `task.budget_exceeded` | Task hit token/tool/wall-clock budget |
**Planned (not yet implemented):**
`system.announcement` (v0.28.4 — admin broadcast to all users)
## Git Auth Types
`https_pat`, `https_basic`, `ssh_key`
## File Origins
`user_upload`, `tool_output`, `system`
## File Display Hints
`inline`, `download`, `thumbnail`
## Proxy Modes
`system`, `direct`, `custom`
## Permissions
12 permission constants, `domain.action` convention:
| Permission | Description |
|-----------|-------------|
| `model.use` | Use AI models for completions |
| `kb.read` | Search knowledge bases |
| `kb.create` | Create knowledge bases |
| `kb.write` | Upload documents to KBs |
| `channel.create` | Create new channels |
| `channel.invite` | Add participants to channels |
| `persona.create` | Create personal personas |
| `persona.manage` | Edit/delete own personas |
| `workflow.create` | Create/edit workflow definitions |
| `tasks.create` | Create/manage tasks |
| `tasks.admin` | Manage all tasks (admin) |
| `admin.access` | Access admin panel |
Permissions are granted via groups. The `Everyone` group
(ID `00000000-0000-0000-0000-000000000001`) provides default
permissions to all authenticated users. It is system-seeded
and editable by admins.
## Extension Permissions (v0.29.0+)
7 extension permission constants, `domain.action` convention.
Declared in package manifests, granted by admin review.
| Permission | Module | Since |
|-----------|--------|-------|
| `secrets.read` | `secrets` | v0.29.0 |
| `notifications.send` | `notifications` | v0.29.0 |
| `filters.pre_completion` | — | v0.29.0 |
| `api.http` | `http` | v0.29.0 |
| `provider.complete` | `provider` | v0.29.1 |
| `db.read` | `db` | v0.29.2 |
| `db.write` | `db` | v0.29.2 |
## Policies
| Key | Default | Description |
|-----|---------|-------------|
| `allow_registration` | `"true"` | Allow new user self-registration |
| `allow_user_byok` | `"false"` | Allow users to add personal API keys |
| `allow_user_personas` | `"false"` | Allow users to create personal Personas |
| `allow_team_providers` | `"true"` | Allow team admins to configure team providers |
| `allow_raw_model_access` | `"true"` | Allow raw model selection without persona |
| `kb_direct_access` | `"true"` | Show KB picker in channel (false = strict enterprise) |
| `default_user_active` | `"false"` | New users active immediately (vs admin approval) |

581
docs/ICD/extensions.md
View File

@@ -1,581 +0,0 @@
# Extensions
Plugin system with three tiers: Browser JS (client-side), Starlark
sandbox (server-side, v0.29.0), Sidecar containers (server-side, future).
### User Extensions
**Auth:** Authenticated user
```
GET /extensions → {"data": [...UserExtension]}
?tier=browser (optional filter by tier)
POST /extensions/:id/settings ← {"is_enabled": bool, "settings": {...}}
:id = extension UUID → {"ok": true}
GET /extensions/:id/manifest → {"data": <manifest JSON>}
:id = ext_id (manifest id)
GET /extensions/tools → {"data": [...tool schema objects]}
```
**Notes:**
- `GET /extensions` returns `UserExtension` objects: the base extension
fields plus `user_enabled` and `user_settings` overrides.
- System extensions (`is_system: true`) cannot be disabled by users.
Attempting to set `is_enabled: false` on a system extension returns 403.
- `GET /extensions/tools` returns raw tool schema JSON from all enabled
browser extensions' `manifest.tools[]` arrays.
### Extension API Routes (v0.29.1)
Starlark packages serve custom JSON endpoints. Mounted at
`/s/:slug/api/*path` with JWT authentication.
**Auth:** Authenticated user (JWT — returns 401, not redirect)
```
ANY /s/:slug/api/*path → Starlark on_request(req) response
```
**Route:** `slug` is the package ID. `path` is matched against the
manifest's `api_routes` declaration.
**Manifest `api_routes` field:**
```json
{
"api_routes": [
{"method": "GET", "path": "/status"},
{"method": "POST", "path": "/webhook"},
{"method": "*", "path": "/proxy/*"}
]
}
```
- `method`: exact match (case-insensitive), or `"*"` for any method
- `path`: exact match, or trailing `"*"` for prefix match
- Boolean `true` shorthand: all routes forwarded
- Missing or `false`: no routes served
**Request dict passed to `on_request(req)`:**
```python
{
"method": "POST",
"path": "/webhook",
"headers": {"content-type": "application/json", ...},
"query": {"page": "1", ...},
"body": "{...}",
"user_id": "uuid"
}
```
**Expected return dict:**
```python
{"status": 200, "headers": {"X-Custom": "val"}, "body": "{...}"}
```
- `status`: int (default 200). Return `None` for 204 No Content.
- `headers`: dict (optional). Content-Type auto-detected if not set.
- `body`: string (default "").
**Validation pipeline:**
1. Package exists and is enabled
2. Status is `active` (not `pending_review` or `suspended`)
3. Tier is `starlark`
4. `api.http` permission is granted
5. Method+path matches `api_routes` in manifest
**Errors:**
- `401` — no/invalid JWT
- `403` — package suspended or missing `api.http` permission
- `404` — package not found, disabled, or route not declared
- `400` — package is not starlark tier
- `500` — Starlark execution error
### Admin Extension Management
**Auth:** Admin role
```
GET /admin/extensions → {"data": [...Extension]}
POST /admin/extensions ← {ext_id*, name*, version?, tier?,
description?, author?, manifest?,
is_system?, is_enabled?}
→ {"data": Extension} (201)
PUT /admin/extensions/:id ← {name?, version?, description?,
:id = extension UUID author?, is_system?, is_enabled?,
manifest?}
→ {"data": Extension}
DELETE /admin/extensions/:id → {"ok": true}
:id = extension UUID
```
**Install defaults:** `version` → `"0.0.0"`, `tier` → `"browser"`,
`manifest` → `{}`, `scope` → `"global"`.
**Tier validation:** `tier` must be one of: `browser`, `starlark`, `sidecar`.
**Duplicate rejection:** If `ext_id` is already installed, returns 409.
### Asset Serving
**Auth:** None (public — script tags can't send Authorization headers)
```
GET /extensions/:id/assets/*path → application/javascript
:id = ext_id (manifest id)
```
Returns the inline `_script` field from the extension's manifest.
The `*path` segment is accepted but currently ignored (all requests
return the same script). Disabled extensions return 404.
### Extension Permissions (v0.29.0+)
Starlark packages declare capabilities in `manifest.permissions`.
Admin must grant each before the package activates.
| Permission | Module | Description |
|-----------|--------|-------------|
| `secrets.read` | `secrets` | Read extension secrets via GlobalConfig |
| `notifications.send` | `notifications` | Send in-app notifications |
| `filters.pre_completion` | — | Register pre-completion filter |
| `api.http` | `http` | Outbound HTTP requests (v0.29.0: module, v0.29.1: also required for API routes) |
| `provider.complete` | `provider` | LLM completion calls via BYOK chain (v0.29.1) |
| `db.read` | `db` | Read extension tables and platform views (v0.29.2) |
| `db.write` | `db` | Write extension tables (v0.29.2) |
### Starlark Modules (v0.29.0+)
Modules injected into the script namespace based on granted permissions:
**`secrets`** (requires `secrets.read`):
- `secrets.get(key)` → string or None
- `secrets.list()` → list of key names
**`notifications`** (requires `notifications.send`):
- `notifications.send(user_id, title, body?, type?)` → True
**`http`** (requires `api.http`, v0.29.1):
- `http.get(url, headers?)` → `{"status": int, "headers": dict, "body": string}`
- `http.post(url, body?, headers?)` → response dict
- `http.put(url, body?, headers?)` → response dict
- `http.delete(url, headers?)` → response dict
- `http.request(method, url, body?, headers?)` → response dict
- SSRF protection: private/loopback/link-local IPs blocked after DNS
- Manifest `network_access`: `{"allow": [...]}` or `{"block": [...]}`
**`provider`** (requires `provider.complete`, v0.29.1):
- `provider.complete(messages, model?, max_tokens?, temperature?)` → response dict
- Response: `{"content", "model", "finish_reason", "input_tokens", "output_tokens"}`
- Provider resolved via BYOK chain; pinnable via `requires_provider.provider_config_id`
**`db`** (requires `db.read` or `db.write`, v0.29.2):
- `db.query(table, filters=None, order=None, limit=100)` → list of dicts
- `table`: logical name (physical: `ext_{pkg_slug}_{table}`)
- `filters`: dict of `{column: value}` equality filters (optional)
- `order`: `"col"` or `"-col"` for descending (optional)
- `limit`: max rows (default 100)
- `db.insert(table, row_dict)` → inserted row dict with auto-generated `id` (`db.write`)
- `db.update(table, id, partial_dict)` → True (`db.write`)
- `db.delete(table, id)` → True (`db.write`)
- `db.list_tables()` → list of logical table names for this package
- `db.view(view_name, filters=None, limit=100)` → list of dicts
- Allowed view names: `"users"` → `ext_view_users`, `"channels"` → `ext_view_channels`
- `ext_view_users`: `id`, `display_name`, `email`
- `ext_view_channels`: `id`, `title`, `type`, `team_id`
### Extension Database Tables (v0.29.2)
Starlark packages declare owned tables in `manifest.db_tables`. Tables
are created on install and dropped on uninstall.
**Manifest `db_tables` field:**
```json
{
"db_tables": {
"logs": {
"columns": {
"message": "text",
"user_id": "text",
"count": "int",
"score": "real",
"active": "bool",
"created_at": "timestamp"
},
"indexes": [
["user_id"],
["user_id", "created_at"]
]
}
}
}
```
- **Physical name**: `ext_{pkg_slug}_{logical_name}` (hyphens → underscores)
- **Auto-columns**: `id TEXT PRIMARY KEY` (UUID generated on insert),
`created_at` (dialect-correct timestamp default)
- **Column types**: `text`, `int`/`integer`, `real`/`float`,
`bool`/`boolean`, `timestamp` — mapped to dialect-correct SQL
- **Indexes**: each entry is a list of columns for a composite index
- **Dialect**: PG uses `TIMESTAMPTZ`/`BOOLEAN`; SQLite uses `TEXT`/`INTEGER`
- **Catalog**: tracked in `ext_data_tables` per package for lifecycle management
### Extension Tools (v0.29.2)
Starlark packages declare server-side tools in `manifest.tools`. These
are included in `BuildToolDefs` alongside server tools. The completion
tool loop dispatches matched calls to the `on_tool_call` entry point.
**Manifest `tools` field (starlark tier only):**
```json
{
"tier": "starlark",
"permissions": ["db.read"],
"tools": [
{
"name": "search_logs",
"description": "Search extension log entries",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"user_id": {"type": "string"}
},
"required": ["query"]
}
}
]
}
```
**`on_tool_call(call)` entry point:**
Called by the completion tool loop when a tool declared in `tools` is
invoked by the LLM.
```python
def on_tool_call(call):
# call dict:
# {
# "tool_name": "search_logs",
# "tool_call_id": "call_abc123",
# "arguments": {"query": "hello", "user_id": "u1"}
# }
if call["tool_name"] == "search_logs":
rows = db.query("logs", filters={"user_id": call["arguments"]["user_id"]})
return {"results": rows}
return {"error": "unknown tool"}
```
- Return value is serialized to JSON and returned as the tool result
- All sandbox modules (including `db`) are available per granted permissions
- No additional permission is required beyond package being `active`
### Multi-File Starlark Packages (v0.38.0)
Starlark packages support `load()` for splitting code across multiple
files. The entry point script is `script.star` (or the `entry_point`
manifest field). Submodules live in `star/`.
**Archive structure:**
```
my-extension/
├── manifest.json ← metadata only (no _starlark_script)
├── script.star ← entry point (required for starlark tier)
├── star/ ← optional submodules
│ ├── auth.star
│ ├── repos.star
│ └── helpers.star
├── js/ ← optional surface assets
└── css/
```
**Manifest `entry_point` field (optional):**
```json
{
"tier": "starlark",
"entry_point": "script.star"
}
```
Default is `script.star`. The installer validates the entry point exists
in the archive for starlark-tier packages (returns 400 if missing).
**`load()` in Starlark scripts:**
```python
# script.star
load("star/auth.star", "auth_headers")
load("star/repos.star", "get_repos")
def on_request(req):
headers = auth_headers(conn)
repos = get_repos(conn)
return {"status": 200, "body": json.encode(repos)}
```
**Constraints:**
- Package-scoped: can only load files within the package's own directory
- `.star` files only — cannot load `.js`, `.json`, etc.
- Path traversal (`..`) and absolute paths are rejected
- Circular dependencies are detected and rejected
- Loaded files get the same injected modules (`db`, `http`, etc.) as the entry point
- All loaded files share the same step limit budget (1M ops total)
**Backward compatibility:** Existing packages with `_starlark_script`
in their manifest continue to work. The runner tries disk first, then
falls back to the manifest field. Reinstalling extracts `script.star`
to disk.
**Install errors:**
- `400` — `starlark package missing entry point "script.star"` (or
custom `entry_point` value)
## 5. Extension Connections (v0.38.1)
Scoped credential management for extensions that integrate with
external services. Same scope/resolution pattern as provider configs
(personal → team → global).
### Personal Connections
**Auth:** Authenticated user
```
GET /connections → {"data": [...connection summaries]}
POST /connections ← {"type","package_id","name","config"} → {"id","type","name"}
GET /connections/:id → connection summary (owned only)
PUT /connections/:id ← {"name?","config?"} → {"message":"connection updated"}
DELETE /connections/:id → {"message":"connection deleted"}
GET /connections/resolve → resolved connection (decrypted config)
?type=gitea&name=Work+Gitea
```
### Team Connections
**Auth:** Team admin (RequireTeamAdmin middleware)
```
GET /teams/:teamId/connections → {"data": [...connection summaries]}
POST /teams/:teamId/connections ← {"type","package_id","name","config"} → {"id","type","name"}
PUT /teams/:teamId/connections/:id ← {"name?","config?"} → {"message":"connection updated"}
DELETE /teams/:teamId/connections/:id → {"message":"connection deleted"}
```
### Admin (Global) Connections
**Auth:** Admin
```
GET /admin/connections → {"data": [...connection summaries]}
POST /admin/connections ← {"type","package_id","name","config"} → {"id","type","name"}
PUT /admin/connections/:id ← {"name?","config?"} → {"message":"connection updated"}
DELETE /admin/connections/:id → {"message":"connection deleted"}
```
**Connection summary** (list/get responses — secrets masked):
```json
{
"id": "uuid", "type": "gitea", "package_id": "git-board",
"scope": "personal", "name": "Work Gitea",
"is_active": true, "has_config": true,
"created_at": "2026-03-25T...", "updated_at": "2026-03-25T..."
}
```
**Scope resolution:** `GET /connections/resolve?type=gitea` resolves
via personal → team → global chain. Returns full connection with
decrypted config. Optional `name` parameter for named resolution.
**Config encryption:** Fields with `type: "secret"` in the package
manifest's `connections[].fields` are encrypted at rest using the
same vault pattern as provider API keys. The handler layer encrypts
on write and decrypts on read. The store is encryption-agnostic.
**Unique constraint:** `(type, scope, owner_id, name)` — a user cannot
have two connections of the same type with the same name.
**Starlark module:** Extensions with `connections.read` permission
get a `connections` module:
```python
conn = connections.get("gitea") # scope chain
conn = connections.get("gitea", name="Work Gitea") # named
all = connections.list("gitea") # all accessible
```
Each returned dict contains `id`, `type`, `name`, `scope` plus all
config fields with secrets decrypted.
### Connection Type Discovery (v0.38.4)
**Auth:** Authenticated user (no specific permission required)
```
GET /connection-types → {"data": [...connection type entries]}
```
Returns the merged set of connection types declared by all active packages.
When multiple packages declare the same type name, library declarations
take precedence over non-library declarations.
**Response entry:**
```json
{
"type": "gitea",
"label": "Gitea Instance",
"package_id": "gitea-client",
"package_title": "Gitea API Client",
"fields": {
"base_url": {"type": "url", "required": "true", "label": "Server URL"},
"api_token": {"type": "secret", "required": "true", "label": "API Token"},
"org": {"type": "string", "required": "false", "label": "Default Org"}
},
"scopes": ["global", "team", "personal"]
}
```
Used by all three Connections management UIs (Settings, Admin, Team Admin)
to populate the connection type dropdown. Replaces client-side manifest
scanning which required admin permissions.
---
### Library Composition (v0.38.4)
Libraries declare connection types in their manifest `connections[]` field.
Consumers depend on the library and use its exported functions, passing
connection dicts obtained from `connections.get()`.
**End-to-end pattern:**
1. Library declares `connections: [{ "type": "gitea", ... }]` in manifest
2. Library exports functions that accept a `conn` parameter
3. Admin installs library and grants permissions (`api.http`, `connections.read`, etc.)
4. User creates a connection of the library's type via Settings > Connections
5. Consumer declares `dependencies: { "gitea-client": ">=1.0.0" }` in manifest
6. Consumer loads library: `gitea = lib.load("gitea-client")`
7. Consumer resolves connection: `conn = connections.get("gitea")`
8. Consumer calls library: `gitea.get_repos(conn)`
The library function executes with the **library's** permission context.
The consumer does not need `api.http` or `db.*` permissions — those are
the library's concern.
**Browser-side consumers** can also call the library's REST endpoints
directly (`GET /s/gitea-client/api/repos`) without Starlark, enabling
pure `type: "surface"` packages.
---
### Library Packages (v0.38.2)
Library packages (`type: "library"`) export Starlark functions for other
packages to consume. Libraries run with their own permission context,
isolating consumers from implementation details.
**Manifest fields:**
- `type: "library"` — required
- `tier: "starlark"` — required (libraries are always starlark-tier)
- `exports: ["fn_a", "fn_b"]` — required, names of globals to expose
- `permissions: [...]` — the library's own permissions (not inherited by consumers)
- `dependencies: {"other-lib": ">=1.0.0"}` — optional, libraries can depend on other libraries
**Restrictions:** Libraries cannot have `tools`, `pipes`, or a `route`.
**Consumer usage (Starlark):**
Consumers declare dependencies in their manifest:
```json
{ "dependencies": { "gitea-client": ">=1.0.0" } }
```
Then load at runtime:
```python
gitea = lib.load("gitea-client")
repos = gitea.get_repos(conn)
```
`lib.load()` validates the dependency record, loads the library's script
with the library's own permissions, extracts the declared exports, and
returns them as a frozen struct. Results are cached per invocation.
**Admin endpoints:**
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/admin/packages/:id/dependencies` | Libraries this package depends on |
| GET | `/api/v1/admin/packages/:id/consumers` | Packages that depend on this library |
| GET | `/api/v1/admin/dependencies` | Full dependency graph |
**Uninstall protection:** Libraries with active consumers return 409 on delete.
---
### Config Sections (v0.38.3)
Packages can declare a `config_section` in their manifest to inject a
Preact configuration component into the Settings, Admin, or Team Admin
surfaces. This enables headless extensions and libraries to own their
configuration UX without requiring a full surface.
**Manifest schema:**
```json
{
"config_section": {
"label": "My Extension",
"icon": "M12 2L2 7...",
"component": "js/config.js",
"surfaces": ["admin", "settings", "team-admin"],
"category": "system"
}
}
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `label` | string | Yes | Nav link text. Falls back to package title if empty. |
| `icon` | string | No | SVG path data in compact format (admin CatIcon). |
| `component` | string | No | Relative path within the package archive. Default: `js/config.js`. |
| `surfaces` | string[] | Yes | Target surfaces: `admin`, `settings`, `team-admin`. |
| `category` | string | No | Admin-only: category tab to appear under. Default: `system`. |
**Component contract:**
The config section component is a standard ES module exporting a
default Preact component. It is lazy-loaded via dynamic `import()` from
the existing asset-serving endpoint (`GET /surfaces/:id/:component`).
The component uses `sw.sdk` to read/write its own package settings:
- `sw.api.admin.packages.settings(packageId)` — read
- `sw.api.admin.packages.updateSettings(packageId, data)` — write
Settings are stored in the `package_settings` JSONB column (admin scope)
and `package_user_settings` table (user scope).
**Discovery:** At page load, the backend queries enabled packages for
`config_section` entries targeting the current surface and passes them
as `__CONFIG_SECTIONS__` to the frontend. The surface merges them into
its nav and section module map.
**No new tables or endpoints.** Config sections are purely manifest-driven,
using existing package storage and settings infrastructure.
---
### Builtin Seeding
On startup, `SeedBuiltinExtensions()` scans `extensions/builtin/`
for subdirectories containing `manifest.json` + `script.js`. Each
is upserted as a system extension:
- **New ext_id** → `Create` with `is_system: true`
- **Same version** → skip (idempotent)
- **Different version** → `Update` manifest, name, description, author
---

271
docs/ICD/knowledge.md
View File

@@ -1,271 +0,0 @@
# Knowledge Bases
The **Knowledge Base** (KB) is the RAG system: upload documents → chunk
→ embed (pgvector / app-level cosine for SQLite) → search via
`kb_search` tool during completions.
### KB CRUD
```
GET /knowledge-bases → { "data": [KB objects], "total": N }
POST /knowledge-bases ← { "name", "description", "scope", "team_id" }
GET /knowledge-bases/:id → KB object
PUT /knowledge-bases/:id ← partial update { "name", "description" }
DELETE /knowledge-bases/:id
```
**Auth:** All endpoints require a valid JWT. `POST` additionally requires
the `kb.create` permission (checked via `RequirePermission` middleware).
**Scope rules for `POST`:**
| Scope | Requirement |
|-------|-------------|
| `personal` (default) | Any authenticated user with `kb.create` |
| `team` | `team_id` required; caller must be team admin (or system admin) |
| `global` | Caller must be system admin |
KB object (as returned by CRUD endpoints):
```json
{
"id": "uuid",
"name": "Product Docs",
"description": "Internal product documentation",
"scope": "personal|team|global",
"owner_id": "uuid|null",
"team_id": "uuid|null",
"embedding_config": {},
"document_count": 12,
"chunk_count": 48,
"total_bytes": 245760,
"status": "active|processing|error",
"created_at": "2025-01-15T10:30:00Z",
"updated_at": "2025-01-15T10:30:00Z"
}
```
> **Note:** The `discoverable` field is stored in the DB and present on
> the raw model, but is **not** included in the `kbResponse` struct
> returned by CRUD endpoints. It is managed via the dedicated
> discoverable endpoints below.
### Documents
**Upload:**
```
POST /knowledge-bases/:id/documents
Content-Type: multipart/form-data
```
**Auth:** Requires `kb.write` permission (checked via `RequirePermission`
middleware). Also requires a configured embedding model role — returns
`412 Precondition Failed` if missing.
Field: `file` (max 10 MB). Currently supported types: **TXT, MD, CSV,
HTML**. Binary formats (PDF, DOCX, XLSX, PPTX) are planned but require
an extraction sidecar that is not yet wired in.
Returns the document object with HTTP `202 Accepted`:
```json
{
"id": "uuid",
"kb_id": "uuid",
"filename": "guide.md",
"content_type": "text/markdown",
"size_bytes": 12345,
"chunk_count": 0,
"status": "pending",
"error": null,
"uploaded_by": "uuid",
"created_at": "2025-01-15T10:30:00Z",
"updated_at": "2025-01-15T10:30:00Z"
}
```
**List:**
```
GET /knowledge-bases/:id/documents
```
Returns `{ "data": [document objects], "total": N }`.
**Status** (poll during processing):
```
GET /knowledge-bases/:id/documents/:docId/status
```
Returns a subset of the document object:
```json
{
"id": "uuid",
"status": "chunking",
"chunk_count": 0,
"error": null,
"filename": "guide.md"
}
```
Status progression: `pending → extracting → chunking → embedding → ready | error`.
**Delete:**
```
DELETE /knowledge-bases/:id/documents/:docId
```
Deletes the document, its chunks, and the stored file. Returns
`{ "deleted": true }`.
### Search
```
POST /knowledge-bases/:id/search
```
```json
{
"query": "How do I configure SSO?",
"limit": 5,
"threshold": 0.3
}
```
`limit` defaults to 5, max 20. `threshold` is the minimum cosine
similarity score (defaults to 0.3).
Returns `{ "data": [...], "query": "...", "total": N }`:
```json
{
"data": [
{
"content": "To configure SSO, navigate to...",
"source": "admin-guide.md",
"kb": "Product Docs",
"similarity": 0.87,
"metadata": {}
}
],
"query": "How do I configure SSO?",
"total": 1
}
```
### Rebuild
Re-chunks and re-embeds all documents. Skips documents whose extracted
text is missing (would need the original file from object store).
```
POST /knowledge-bases/:id/rebuild
```
Returns `202 Accepted`:
```json
{
"message": "rebuild started",
"total_docs": 12,
"rebuilding": 10,
"skipped": 2
}
```
Returns `503` if the ingestion pipeline is not configured, or `400` if
the KB has no documents.
### Channel KB Bindings
A channel can have KBs explicitly bound to it (in addition to any KBs
coming from the active Persona or parent Project).
**Get:**
```
GET /channels/:id/knowledge-bases
```
Returns `{ "data": [ChannelKB objects] }`:
```json
{
"kb_id": "uuid",
"kb_name": "Product Docs",
"enabled": true,
"document_count": 12
}
```
**Set** (replace all):
```
PUT /channels/:id/knowledge-bases
```
```json
{ "kb_ids": ["kb-uuid-1", "kb-uuid-2"] }
```
Validates that the caller has access to every referenced KB before
setting the bindings.
### Discoverable KBs
The `discoverable` flag controls whether a KB appears in the user's KB
picker. Non-discoverable KBs are hidden from direct access but still
searchable through Persona bindings (enterprise KB mode).
**List discoverable** (for KB picker UI):
```
GET /knowledge-bases-discoverable
```
Returns `{ "data": [KB objects] }` — only KBs the user can see
(personal + team + discoverable global). Uses the same `kbResponse`
format as the CRUD endpoints.
**Toggle discoverability:**
```
PUT /knowledge-bases/:id/discoverable
```
```json
{ "discoverable": false }
```
**Auth:** Requires `loadAndAuthorize` (KB must exist, caller must have
basic access). Additionally, only the KB **owner** or a **system admin**
may change the flag — all other callers receive `403 Forbidden`.
### KB Resolution Chain
When a completion fires, KBs are resolved in priority order:
```
Persona KBs → Project KBs → Channel KBs → Personal KBs
```
The `BuildKBHint` function assembles all applicable KBs into the
system prompt, and the `kb_search` tool searches across all of them.
The `kb_search` tool additionally resolves project-bound KBs and
personal KBs at search time, so all user-accessible KBs are
searchable even if not explicitly listed in the hint.
### Persona KB Bindings
Managed via the Persona endpoints — see [personas.md](personas.md).
Persona-KB binding is the primary enterprise mechanism for controlled
KB access. The `auto_search` flag on persona KB bindings controls
whether top-K results are prepended to context automatically (planned,
see v0.28.4) vs available only through the `kb_search` tool (current
behavior).
---

95
docs/ICD/memory.md
View File

@@ -1,95 +0,0 @@
# Memory
**Long-term memory** extracted from conversations. The system
identifies facts, preferences, and context about the user and stores
them for injection into future conversations.
**Auth:** All user endpoints require authenticated session.
Admin endpoints require platform admin role.
---
### User Memory
```
GET /memories → { "data": [memory objects] }
GET /memories/count → { "active": 5, "pending": 2 }
PUT /memories/:id ← { "key": "...", "value": "...", "confidence": 0.9 }
DELETE /memories/:id
POST /memories/:id/approve → approve a pending_review memory
POST /memories/:id/reject → reject (archive) a pending_review memory
```
Query params for `GET /memories`:
| Param | Default | Description |
|-------|---------|-------------|
| `status` | `active` | Filter by status: `active`, `pending_review`, `archived` |
| `query` | — | Keyword filter on key+value (PG: full-text, SQLite: LIKE) |
Memory object:
```json
{
"id": "uuid",
"scope": "user|persona|persona_user",
"owner_id": "uuid",
"user_id": "uuid|null",
"key": "preferred language",
"value": "Go with Gin framework for backend services",
"source_channel_id": "uuid|null",
"confidence": 0.85,
"status": "active|pending_review|archived",
"created_at": "...",
"updated_at": "..."
}
```
**Scopes:**
| Scope | `owner_id` | `user_id` | Description |
|-------|-----------|-----------|-------------|
| `user` | user's ID | null | Personal facts, persist across all conversations |
| `persona` | persona's ID | null | Shared across all users of a Persona |
| `persona_user` | persona's ID | user's ID | Per-user facts within a Persona context |
**Ownership:** `PUT` and `DELETE` enforce ownership — a user can only
modify their own user-scope memories (`owner_id == user_id`).
### Admin Memory Review
```
GET /admin/memories/pending → { "data": [memory objects] }
POST /admin/memories/bulk-approve ← { "ids": ["uuid1", "uuid2"] }
→ { "approved": 2 }
```
Platform admin can review and bulk-approve pending memories across
all users.
### AI Tools (non-REST)
Two tools are registered for AI use during completions:
| Tool | Description |
|------|-------------|
| `memory_save` | Save a fact with `key`, `value`, `confidence`. Saves as `active` (bypasses review). |
| `memory_recall` | Search memories by query. Uses hybrid semantic+keyword recall when embedder is configured. |
### Background Extraction
A background scanner periodically finds conversations with enough
new activity and calls the utility model to extract memorable facts.
Extracted memories are saved as `pending_review`.
Controlled by `memory_extraction_enabled` global config key and
per-persona `memory_enabled` flag.
### Memory Injection
Active memories are injected into the system prompt at completion
time via `BuildMemoryHint`. Uses hybrid semantic recall when an
embedder is available, falling back to keyword/confidence ranking.
Budget: ~6000 characters.
---

258
docs/ICD/models.md
View File

@@ -1,258 +0,0 @@
# Models & Preferences
What models are available to the current user and how they control
visibility.
---
### Enabled Models
The primary endpoint for populating model selectors:
```
GET /models/enabled
```
Returns a composite response:
```json
{
"data": [ UserModel, ... ],
"default_model": "claude-sonnet-4-20250514"
}
```
`default_model` is the admin-configured default model ID (from the
`default_model` policy). Empty string if unset.
**UserModel object:**
```json
{
"id": "composite-id",
"display_name": "Claude Sonnet 4",
"model_id": "claude-sonnet-4-20250514",
"model_type": "chat",
"source": "catalog",
"provider_config_id": "uuid",
"config_id": "uuid",
"provider_name": "Anthropic",
"provider_type": "anthropic",
"capabilities": {
"streaming": true,
"tool_calling": true,
"vision": true,
"thinking": true,
"reasoning": false,
"code_optimized": false,
"web_search": false,
"max_context": 200000,
"max_output_tokens": 8192
},
"is_persona": false,
"persona_id": "",
"persona_handle": "",
"persona_scope": "",
"persona_avatar": "",
"persona_team_name": "",
"description": "",
"icon": "",
"avatar": "",
"system_prompt": "",
"temperature": null,
"max_tokens": null,
"tool_grants": [],
"pricing": {
"input_per_m": 3.00,
"output_per_m": 15.00,
"currency": "USD"
},
"scope": "global",
"owner_id": null,
"team_name": "",
"hidden": false,
"sort_order": 0,
"provider_status": "healthy"
}
```
**Field reference:**
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Composite key. Catalog: `{config_id}:{model_id}`. Persona: persona UUID. |
| `display_name` | string | Human-readable name (from catalog or persona). |
| `model_id` | string | Raw model identifier (e.g. `claude-sonnet-4-20250514`). |
| `model_type` | string | `"chat"`, `"embedding"`, `"image"`, etc. |
| `source` | string | `"catalog"` (DB catalog entry) or `"persona"`. |
| `provider_config_id` | string | UUID of the provider config that serves this model. |
| `config_id` | string | Alias of `provider_config_id` (frontend compat). |
| `provider_name` | string | Display name of the provider config. |
| `provider_type` | string | Provider type slug (e.g. `"anthropic"`, `"openai"`). |
| `capabilities` | object | Nested `ModelCapabilities` — see below. |
| `is_persona` | bool | `true` if this entry is a Persona, not a raw model. |
| `persona_id` | string | Persona UUID (omitted if not a persona). |
| `persona_handle` | string | `@handle` for mention routing (omitted if not a persona). |
| `persona_scope` | string | Persona scope: `"global"`, `"team"`, `"personal"` (omitted if not). |
| `persona_avatar` | string | Avatar URL (omitted if not a persona or unset). |
| `persona_team_name` | string | Owning team name for team-scoped personas (omitted if N/A). |
| `description` | string | Persona description (omitted if empty). |
| `icon` | string | Persona icon (omitted if empty). |
| `avatar` | string | Avatar URL (omitted if empty). |
| `system_prompt` | string | Persona system prompt (omitted if empty). |
| `temperature` | float\|null | Persona temperature override. |
| `max_tokens` | int\|null | Persona max-tokens override. |
| `tool_grants` | string[] | Tool IDs granted to this persona. |
| `pricing` | object\|null | Nested `ModelPricing` — see below. Omitted if unavailable. |
| `scope` | string | `"global"`, `"team"`, or `"personal"`. |
| `owner_id` | string\|null | Owner user/team ID for team/personal scoped entries. |
| `team_name` | string | Team name (persona entries only). |
| `hidden` | bool | `true` if the user has hidden this model via preferences. |
| `sort_order` | int | User's custom sort position (0 = default). |
| `provider_status` | string | Health status: `"healthy"`, `"degraded"`, `"down"`, `"unknown"`, or empty. |
**ModelCapabilities:**
| Field | Type | Description |
|-------|------|-------------|
| `streaming` | bool | Supports streaming responses. |
| `tool_calling` | bool | Supports function/tool calling. |
| `vision` | bool | Supports image inputs. |
| `thinking` | bool | Supports extended thinking. |
| `reasoning` | bool | Reasoning-optimized model. |
| `code_optimized` | bool | Code-optimized model. |
| `web_search` | bool | Has built-in web search. |
| `max_context` | int | Context window size in tokens. |
| `max_output_tokens` | int | Maximum output tokens. |
Capabilities are resolved through the three-tier chain: catalog DB →
heuristic inference → admin overrides (see `providers.md` §capabilities).
**ModelPricing:**
| Field | Type | Description |
|-------|------|-------------|
| `input_per_m` | float | Input cost per million tokens. |
| `output_per_m` | float | Output cost per million tokens. |
| `currency` | string | Currency code (e.g. `"USD"`). |
**Persona inheritance:** When `is_persona` is `true`, the `capabilities`
and `pricing` objects are inherited from the persona's underlying base
model. The persona adds identity (name, handle, avatar, system prompt,
tool grants) — not capability restrictions. Persona entries always have
`hidden: false`; persona visibility is controlled by the grant system
and the `is_active` flag, not by model preferences.
**Model allowlist filtering (v0.24.2):** Non-admin users whose groups
define model restrictions will only see models in their allowlist.
Persona entries whose underlying `model_id` is not in the allowlist are
also filtered. Admins bypass this filter entirely.
**Visibility resolution order:**
1. Global catalog: enabled models → all authenticated users
2. Team catalog: enabled models → team members
3. Personal BYOK: enabled models → owner (requires `allow_user_byok` policy)
4. Personas: global + team + personal + shared-via-grants
5. User hidden preferences applied last (sets `hidden` flag, does not remove)
6. Model allowlist filtering (removes entries entirely for non-admins)
---
### User Model Preferences
Users can hide models they don't want to see and set per-model
defaults. Preferences are keyed on the **composite identity**
`provider_config_id:model_id` — the same model from different
providers can have independent visibility.
```
GET /models/preferences → { "data": [PreferenceEntry, ...] }
PUT /models/preferences ← { "model_id": "...", "provider_config_id": "uuid", ... }
POST /models/preferences/bulk ← { "entries": [...], "hidden": bool }
```
**GET /models/preferences**
Returns all preference entries for the authenticated user.
PreferenceEntry:
```json
{
"id": "uuid",
"user_id": "uuid",
"model_id": "grok-4.1-fast",
"provider_config_id": "uuid",
"hidden": true,
"preferred_temperature": 0.7,
"preferred_max_tokens": 4096,
"sort_order": 0,
"created_at": "2025-06-15T14:30:00Z",
"updated_at": "2025-06-15T14:30:00Z"
}
```
**PUT /models/preferences**
Upserts a single preference entry. All fields except `model_id` are
optional — only provided fields are updated. If no entry exists, one
is created.
Request body:
```json
{
"model_id": "claude-sonnet-4-20250514",
"provider_config_id": "uuid",
"hidden": true,
"preferred_temperature": 0.7,
"preferred_max_tokens": 4096,
"sort_order": 1
}
```
`model_id` and `provider_config_id` are required. All other fields are
optional — only provided fields are updated. If no entry exists, one
is created.
Returns `{ "message": "preference updated" }`.
**POST /models/preferences/bulk**
Sets the hidden state for multiple model+provider pairs at once. The
`hidden` value is applied to **all** entries in the batch — this is
not per-entry hidden control.
Request body:
```json
{
"entries": [
{ "model_id": "grok-4.1-fast", "provider_config_id": "uuid" },
{ "model_id": "llama-3.1-70b", "provider_config_id": "uuid" }
],
"hidden": true
}
```
Returns `{ "message": "preferences updated", "count": 2 }`.
**Identity rule:** `provider_config_id` is required on write. The same
bare `model_id` from two different provider configs (e.g. global Venice
vs personal BYOK Venice) are independent preference entries. The
frontend composite ID format is `{provider_config_id}:{model_id}`.
**Persona preferences:** Personas are not in this table. A persona's
visibility is controlled by the grant system and the `is_active` flag,
not by model preferences.
---

105
docs/ICD/notes.md
View File

@@ -1,105 +0,0 @@
# Notes
**Obsidian-style** knowledge graph with `[[wikilinks]]`, backlinks,
graph visualization, and folder organization. Notes are user-scoped
(personal notebook). Future: channel-scoped notes for workflow
artifacts.
### CRUD
```
GET /notes → paginated list of noteListItem
POST /notes ← { "title", "content", "folder" }
GET /notes/:id → full note object
PUT /notes/:id ← { "title", "content", "folder" }
DELETE /notes/:id
```
Note object:
```json
{
"id": "uuid",
"user_id": "uuid",
"title": "Meeting Notes",
"content": "# Meeting Notes\n\nDiscussed [[Project Alpha]]...",
"folder": "work",
"source_message_id": "uuid|null",
"created_at": "...",
"updated_at": "..."
}
```
`source_message_id` links a note back to the chat message that
created it (provenance for AI-generated notes).
`noteListItem` is the same but with `content` truncated or omitted.
### Search
**Full-text search:**
```
GET /notes/search?q=project+alpha
```
Uses `plainto_tsquery` (Postgres) or `LIKE` fallback (SQLite).
Returns `{ "data": [searchResult objects] }` with match highlights.
**Title search** (lightweight, for wikilink autocomplete):
```
GET /notes/search-titles?q=proj
```
Returns `{ "data": [{ "id", "title" }] }`.
### Graph
**Full graph topology:**
```
GET /notes/graph
```
```json
{
"nodes": [{ "id": "uuid", "title": "Note Title" }],
"edges": [{ "source": "uuid", "target": "uuid" }],
"unresolved": [{ "source": "uuid", "target_title": "Missing Note" }]
}
```
Edges are directed: source contains `[[target_title]]`. Unresolved
edges have a `target_title` but no matching note (dangling wikilink).
When a note with that title is created, the edge auto-resolves.
**Backlinks** (who links to this note):
```
GET /notes/:id/backlinks
```
Returns `{ "data": [note objects] }` — all notes containing
`[[this note's title]]`.
### Folders
```
GET /notes/folders
```
Returns `{ "folders": ["work", "personal", "archive"] }` — distinct
folder values across all user notes.
### Bulk Operations
```
POST /notes/bulk-delete
```
```json
{ "ids": ["uuid-1", "uuid-2"] }
```
---

155
docs/ICD/notifications.md
View File

@@ -1,155 +0,0 @@
# Notifications
Real-time notification infrastructure with WebSocket delivery and
optional email transport.
**Auth:** All endpoints require authentication (JWT via `sb_token`
cookie or `Authorization: Bearer` header).
### CRUD
```
GET /notifications → paginated list
GET /notifications/unread-count → { "count": 5 }
PATCH /notifications/:id/read → mark one as read
POST /notifications/mark-all-read → mark all as read
DELETE /notifications/:id
```
**`GET /notifications`** — paginated list for the authenticated user.
Query parameters:
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `limit` | int | 20 | 1–100, clamped |
| `offset` | int | 0 | Pagination offset |
| `unread_only` | bool | false | Filter to unread only |
Response:
```json
{
"data": [ ... ],
"total": 42,
"limit": 20,
"offset": 0
}
```
Notification object:
```json
{
"id": "uuid",
"user_id": "uuid",
"type": "role.fallback",
"title": "Role Fallback Triggered",
"body": "Primary model unavailable, using fallback",
"resource_type": "channel",
"resource_id": "uuid",
"is_read": false,
"created_at": "2025-01-15T12:00:00Z"
}
```
`resource_type` and `resource_id` are optional — used for
click-to-navigate. The resource may be deleted (no FK constraint).
### Preferences
Users control per-type delivery:
```
GET /notifications/preferences → { "data": [...] }
PUT /notifications/preferences/:type ← { "in_app": true, "email": false }
DELETE /notifications/preferences/:type → reset to default
```
**`GET /notifications/preferences`** returns all preferences set by
the user. Empty array if none are configured.
**`PUT /notifications/preferences/:type`** creates or updates a
preference. Both `in_app` and `email` fields are optional (merges
with existing values if set).
**`DELETE /notifications/preferences/:type`** removes the preference,
falling back to the next level in the resolution chain. Idempotent —
deleting a non-existent preference returns 200.
Resolution: specific type pref → user wildcard `*` pref → system
default (`in_app=true`, `email=false`).
Returns 503 if the preference store is not available (unmanaged mode).
### Admin
```
POST /admin/notifications/test-email → send test email to requesting admin
```
Requires admin role. Loads SMTP config from platform settings,
sends a test message to the admin's registered email address.
### Admin Broadcast (v0.28.6)
```
POST /admin/notifications/broadcast
```
**Auth:** Admin only.
Sends a `system.announcement` notification to all active users via
`NotifyMany`. Also emits a `system.broadcast` WebSocket event for
real-time visibility.
**Request body:**
```json
{
"title": "Scheduled Maintenance",
"message": "Servers will be down from 2-4 AM EST.",
"level": "warning"
}
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `title` | string | yes | Short summary shown in bell dropdown |
| `message` | string | yes | Detail text |
| `level` | string | no | `info` (default), `warning`, or `critical` |
**Response:**
```json
{ "message": "broadcast sent", "count": 42 }
```
`count` is the number of active users notified.
### WebSocket Events
| Event | Payload | Description |
|-------|---------|-------------|
| `notification.new` | Full notification object | New notification created |
| `notification.read` | `{"id": "uuid"}` | Single notification marked read |
| `notification.read` | `{"action": "mark_all_read"}` | All notifications marked read |
| `system.broadcast` | `{"title", "message", "level"}` | Admin announcement (v0.28.6) |
These are targeted via `Hub.SendToUser` — no room subscription
required. Used for badge sync across tabs.
### Notification Types
See [enums.md](enums.md#notification-types) for the canonical list.
Types are free-form strings (`domain.action` convention) — new types
don't require migration. Core types include `system.announcement`
(admin broadcast, v0.28.6).
### Retention
Background cleanup prunes notifications older than 90 days (default,
configurable via `WithRetention`). Runs daily after a 5-minute
startup delay.
---

243
docs/ICD/personas.md
View File

@@ -1,243 +0,0 @@
# Personas
A **Persona** is the unit of access control and AI identity: system
prompt + model + provider config + KB bindings + grant scoping. Users
interact with Personas, not raw provider configs. Admins curate which
Personas are available; team admins create team-scoped Personas.
Three scopes, three route namespaces, one domain:
| Scope | Routes | Who manages |
|-------|--------|-------------|
| Personal | `GET/POST/PUT/DELETE /personas` | The user |
| Team | `GET/POST/PUT/DELETE /teams/:teamId/personas` | Team admin |
| Global (admin) | `GET/POST/PUT/DELETE /admin/personas` | Platform admin |
All three return the same Persona object shape:
```json
{
"id": "uuid",
"name": "Code Reviewer",
"handle": "code-reviewer",
"description": "Reviews code for bugs and style",
"icon": "🔍",
"avatar": "data:image/png;base64,...",
"base_model_id": "claude-sonnet-4-20250514",
"provider_config_id": "uuid|null",
"system_prompt": "You are a senior code reviewer...",
"temperature": 0.7,
"max_tokens": 4096,
"thinking_budget": null,
"top_p": null,
"scope": "personal|team|global",
"owner_id": "uuid|null",
"created_by": "uuid",
"is_active": true,
"is_shared": false,
"memory_enabled": false,
"memory_extraction_prompt": null,
"grants": [],
"kb_ids": [],
"created_at": "...",
"updated_at": "..."
}
```
`grants` and `kb_ids` are loaded from junction tables, not stored on the
persona row. All nullable fields (`provider_config_id`, `owner_id`,
`temperature`, `max_tokens`, `thinking_budget`, `top_p`,
`memory_extraction_prompt`) are omitted from JSON when null.
`handle` is auto-generated from `name` on creation if not provided
(e.g. "Code Reviewer" → "code-reviewer"). Used for @mention routing.
### Personal Personas
Gated by `allow_user_personas` policy.
```
GET /personas → { "data": [...] }
POST /personas ← { "name", "base_model_id", ... }
PUT /personas/:id ← partial update
DELETE /personas/:id
```
**Auth:** JWT required. `POST` requires `persona:create` permission.
`PUT`/`DELETE` require `persona:manage` permission and ownership check
(`scope == "personal"` and `owner_id == caller`).
### Team Personas
```
GET /teams/:teamId/personas → { "data": [...] }
POST /teams/:teamId/personas ← same shape
PUT /teams/:teamId/personas/:id ← partial update
DELETE /teams/:teamId/personas/:id
```
**Auth:** JWT required. Team membership for `GET`, team admin for
`POST`/`PUT`/`DELETE`. All mutating endpoints verify the persona belongs
to the team in the URL path (`scope == "team"` and `owner_id == teamId`).
### Admin (Global) Personas
```
GET /admin/personas → { "data": [...] }
POST /admin/personas ← same shape
PUT /admin/personas/:id
DELETE /admin/personas/:id
```
**Auth:** Admin JWT required (middleware enforced).
### Persona KB Bindings
A Persona can have Knowledge Bases bound to it. When a channel uses
that Persona, the bound KBs are automatically scoped into `kb_search`
tool calls. Users never see or manage the underlying KBs directly —
they talk to the Persona.
**Get bindings:**
```
GET /personas/:id/knowledge-bases → { "data": [KB objects] }
GET /teams/:teamId/personas/:id/knowledge-bases → { "data": [...] }
GET /admin/personas/:id/knowledge-bases → { "data": [...] }
```
**Set bindings** (replace all):
```
PUT /personas/:id/knowledge-bases
PUT /teams/:teamId/personas/:id/knowledge-bases
PUT /admin/personas/:id/knowledge-bases
```
```json
{
"kb_ids": ["kb-uuid-1", "kb-uuid-2"],
"auto_search": {
"kb-uuid-1": true,
"kb-uuid-2": false
}
}
```
`auto_search`: when `true`, the KB is always searched (prepended to
context). When `false`, it's available via the `kb_search` tool but
not auto-injected.
Team-scoped KB binding endpoints verify the persona belongs to the
team in the URL path before reading or writing bindings.
### Persona Avatars
Avatars are stored as `data:image/png;base64,...` data URIs in the
`avatar` column. Upload accepts a JSON body (not multipart):
```json
{ "image": "data:image/png;base64,..." }
```
The image is decoded, resized to 128×128 PNG, and stored.
```
POST /personas/:id/avatar ← { "image": "base64..." }
DELETE /personas/:id/avatar
POST /admin/personas/:id/avatar ← same
DELETE /admin/personas/:id/avatar
POST /teams/:teamId/personas/:id/avatar ← same (verify team ownership)
DELETE /teams/:teamId/personas/:id/avatar
```
**Auth:** Personal avatar routes verify the caller owns the persona
(`scope == "personal"` and `owner_id == caller`). Admin routes are
protected by admin middleware.
### Persona Tool Grants
Control which tools a persona can use during completions. The completion
handler applies tool grants as a second-pass allowlist.
```
GET /personas/:id/tool-grants → { "data": ["web_search", "kb_search", ...] }
PUT /personas/:id/tool-grants ← { "tool_names": ["web_search", "calculator"] }
```
Admin equivalents:
```
GET /admin/personas/:id/tool-grants
PUT /admin/personas/:id/tool-grants
```
Team equivalents (verify persona belongs to team):
```
GET /teams/:teamId/personas/:id/tool-grants
PUT /teams/:teamId/personas/:id/tool-grants
```
When a workflow version is published, persona tool grants at that moment
are frozen into the version snapshot.
### Persona Groups
Saved roster templates. A persona group is a named set of personas that
can be stamped onto new conversations.
```
GET /persona-groups → { "data": [...] }
POST /persona-groups ← { "name", "description" }
GET /persona-groups/:id → group with members
PUT /persona-groups/:id ← partial update
DELETE /persona-groups/:id
```
**Members:**
```
POST /persona-groups/:id/members ← { "persona_id", "is_leader": false }
DELETE /persona-groups/:id/members/:memberId
```
`is_leader`: the leader persona responds when no @mention is present in
a group channel. Only one leader per group. Setting a new leader
automatically clears the existing one.
**Persona Group Object:**
```json
{
"id": "uuid",
"name": "Support Team",
"description": "...",
"owner_id": "uuid",
"scope": "personal",
"team_id": null,
"members": [
{
"id": "uuid",
"group_id": "uuid",
"persona_id": "uuid",
"is_leader": true,
"sort_order": 0,
"persona_name": "...",
"persona_handle": "...",
"persona_avatar": "..."
}
],
"created_at": "...",
"updated_at": "..."
}
```
**Note:** Persona groups are currently personal-scope only. The `scope`
and `team_id` fields exist in the schema but `POST` hardcodes
`scope = 'personal'`. Team-scoped persona groups are a future item.
### Resource Grants
See [teams.md](teams.md) §15.3 for the grant system. Personas use grants
to control who can see and use them beyond their base scope.

309
docs/ICD/profile.md
View File

@@ -1,309 +0,0 @@
# User Profile & Settings
User identity, preferences, and credentials.
**All endpoints require:** `Auth()` middleware (JWT bearer token).
---
### Get Profile
```
GET /profile
```
**Auth:** Authenticated user
Returns the current user's profile.
```json
{
"id": "uuid",
"username": "jdoe",
"email": "jdoe@example.com",
"display_name": "Jane Doe",
"role": "user",
"avatar": "data:image/png;base64,...",
"settings": { "theme": "dark" },
"created_at": "2025-06-15T14:30:00Z",
"last_login_at": "2025-06-15T14:30:00Z"
}
```
| Field | Type | Notes |
|-------|------|-------|
| `id` | string | UUIDv4 |
| `username` | string | Unique login name |
| `email` | string | Unique, lowercased |
| `display_name` | string \| null | Optional friendly name |
| `role` | `"user"` \| `"admin"` | Platform role |
| `avatar` | string \| null | Data URI (PNG, 128×128). Omitted if unset. |
| `settings` | object | User preferences (theme, keybindings, etc.) |
| `created_at` | string | ISO 8601 timestamp |
| `last_login_at` | string \| null | ISO 8601 timestamp. Null if never logged in. |
> **Note:** The profile endpoint returns `avatar` (not `avatar_url`) as a
> curated response shape. The `User` model uses `avatar_url` internally.
---
### Update Profile
```
PUT /profile
```
**Auth:** Authenticated user
```json
{
"display_name": "New Name",
"email": "new@example.com"
}
```
Both fields are optional (partial update). Returns the full profile
(same shape as `GET /profile`).
**Errors:**
- `409` — email already taken
---
### Upload Avatar
```
POST /profile/avatar
```
**Auth:** Authenticated user
**Content-Type:** `application/json`
```json
{
"image": "data:image/png;base64,..."
}
```
Accepts a base64 data URI or raw base64 string. The server decodes,
resizes to 128×128 PNG, and stores as a data URI. Max input: 2 MB.
**Response:**
```json
{
"avatar": "data:image/png;base64,..."
}
```
**Errors:**
- `400` — missing `image` field, invalid base64, unsupported format, too large
---
### Delete Avatar
```
DELETE /profile/avatar
```
**Auth:** Authenticated user
**Response:**
```json
{
"message": "avatar removed"
}
```
---
### Change Password
```
POST /profile/password
```
**Auth:** Authenticated user (builtin auth only)
```json
{
"current_password": "old-pass",
"new_password": "new-pass-min-8"
}
```
Validates `current_password` against stored bcrypt hash. `new_password`
must be 8–128 characters.
On success, the UEK (User Encryption Key) is re-wrapped with the new
password so BYOK-encrypted provider keys remain accessible.
**Response:**
```json
{
"message": "password updated"
}
```
**Errors:**
- `400` — missing fields, password too short/long
- `401` — current password incorrect
---
### Get Settings
```
GET /settings
```
**Auth:** Authenticated user
Returns user-level preferences wrapped in a `settings` key.
```json
{
"settings": {
"theme": "dark",
"editor_keybindings": "vim",
"default_model": "claude-sonnet-4-5"
}
}
```
This is a **composite response** (named key wrapping a single object),
not a bare object. The `settings` key is always present; its value is
`{}` if the user has never saved preferences.
---
### Update Settings
```
PUT /settings
```
**Auth:** Authenticated user
```json
{
"theme": "light",
"new_key": "new_value"
}
```
Shallow merge: incoming keys overwrite existing, unmentioned keys are
preserved. Handles edge cases: SQL NULL, JSON `null`, and corrupted
array values are all normalized to `{}` before merge.
Returns the updated settings (same shape as `GET /settings`).
---
### Get My Permissions
```
GET /profile/permissions
```
**Auth:** Authenticated user
Returns the current user's resolved permission set, contributing group
IDs, team memberships, and UI-relevant policies. Admin users receive all
permissions by definition.
**Response:**
```json
{
"permissions": ["model.use", "kb.read", "channel.create"],
"groups": ["group-id-1", "00000000-0000-0000-0000-000000000001"],
"teams": [
{ "id": "uuid", "name": "Engineering", "my_role": "member" }
],
"policies": {
"allow_user_byok": false,
"allow_user_personas": false,
"allow_raw_model_access": true,
"kb_direct_access": true
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `permissions` | string[] | Resolved permission set (union of all group permissions) |
| `groups` | string[] | Contributing group IDs (always includes Everyone) |
| `teams` | object[] | User's active team memberships with `my_role` |
| `policies` | object | Boolean policy flags relevant to UI feature gating |
`permissions` is sorted alphabetically. `teams` includes only active teams
the user is a member of.
This endpoint is the SDK's bootstrap source for `sw.can(permission)`.
Called once at login and on each token refresh.
---
### Bootstrap (v0.37.15)
```
GET /profile/bootstrap
```
**Auth:** Authenticated user
Single-call boot payload for the SDK. Collapses what previously required
3–4 sequential requests (profile, permissions, teams/mine, settings) into
one call. The SDK calls this at startup and on token refresh.
**Response:**
```json
{
"user": {
"id": "uuid",
"username": "jdoe",
"display_name": "Jane Doe",
"email": "jdoe@example.com",
"role": "user",
"avatar": "data:image/png;base64,..."
},
"permissions": ["channel.create", "kb.read", "model.use"],
"groups": ["group-id-1", "00000000-0000-0000-0000-000000000001"],
"teams": [
{ "id": "uuid", "name": "Engineering", "my_role": "member" }
],
"policies": {
"allow_user_byok": false,
"allow_user_personas": false,
"allow_raw_model_access": true,
"kb_direct_access": true
},
"settings": {
"theme": "dark",
"editor_keybindings": "vim"
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `user` | object | Curated profile (same fields as `GET /profile` minus timestamps) |
| `permissions` | string[] | Resolved permission set (sorted). Admins get all. |
| `groups` | string[] | Contributing group IDs (always includes Everyone) |
| `teams` | object[] | Active team memberships with `my_role` |
| `policies` | object | Boolean policy flags for UI feature gating |
| `settings` | object | User preferences (`{}` if never saved) |
`user.avatar` is omitted when unset (not `null`).
This is a **superset** of `GET /profile/permissions` — it includes
everything that endpoint returns plus `user` and `settings`. Clients
that only need permissions can continue using the narrower endpoint.
---

336
docs/ICD/projects.md
View File

@@ -1,336 +0,0 @@
# Projects
**Projects** are organizational containers that group channels,
knowledge bases, notes, and files. They follow the scope model
(personal, team, global).
All endpoints require authentication (`Authorization: Bearer <token>`).
Admin endpoints additionally require `role=admin`.
## Project Object
```json
{
"id": "uuid",
"name": "Q3 Research",
"description": "...",
"color": "#3b82f6",
"icon": "folder",
"scope": "personal|team|global",
"owner_id": "uuid",
"team_id": "uuid|null",
"workspace_id": "uuid|null",
"is_archived": false,
"settings": {},
"channel_count": 5,
"kb_count": 2,
"note_count": 3,
"created_at": "2025-01-15T10:30:00Z",
"updated_at": "2025-01-15T10:30:00Z"
}
```
| Field | Type | Notes |
|-------|------|-------|
| `id` | uuid | PK, auto-generated |
| `name` | string | Required, max 200 chars |
| `description` | string | Optional |
| `color` | string? | CSS hex color, max 7 chars |
| `icon` | string? | Icon identifier, max 50 chars |
| `scope` | enum | `personal`, `team`, `global` |
| `owner_id` | uuid | Set from JWT, immutable |
| `team_id` | uuid? | Set for team-scoped projects |
| `workspace_id` | uuid? | Linked workspace |
| `is_archived` | bool | Default `false` |
| `settings` | object | JSONB, default `{}` |
| `channel_count` | int | Computed, omitted when 0 |
| `kb_count` | int | Computed, omitted when 0 |
| `note_count` | int | Computed, omitted when 0 |
| `created_at` | datetime | Auto-set |
| `updated_at` | datetime | Auto-updated on change |
## Project CRUD
### List Projects
```
GET /projects → { "data": [...] }
```
Query params: `?include_archived=true` to include archived projects.
Returns projects the user can access: personal (own), team (member),
and global. Ordered by name.
### Create Project
```
POST /projects → 201, project object
```
```json
{
"name": "My Project",
"description": "Optional description",
"color": "#3b82f6",
"icon": "folder"
}
```
| Field | Required | Notes |
|-------|----------|-------|
| `name` | yes | Max 200 chars |
| `description` | no | |
| `color` | no | CSS hex |
| `icon` | no | Icon identifier |
Scope is set to `personal` and `owner_id` is taken from the JWT.
Returns the created project object (bare, no envelope).
### Get Project
```
GET /projects/:id → project object
```
Returns bare project object (no envelope). 404 if not found or
not accessible.
### Update Project
```
PUT /projects/:id → updated project object
```
Partial update — only supplied fields are changed.
```json
{
"name": "New Name",
"description": "Updated",
"color": "#ef4444",
"icon": "star",
"is_archived": true,
"workspace_id": "uuid|null",
"settings": { "key": "value" }
}
```
Settings are merged (overlay, not replace). Returns the refreshed
project object after update.
### Delete Project
```
DELETE /projects/:id → { "message": "project deleted" }
```
Only the project owner or an admin can delete. Channels get
`project_id` set to NULL; junction rows cascade.
---
## Channel Association
```
GET /projects/:id/channels → { "data": [...] }
POST /projects/:id/channels ← { "channel_id", "position" }
DELETE /projects/:id/channels/:channelId → { "message": "channel removed" }
PUT /projects/:id/channels/reorder ← { "channel_ids": ["uuid1", "uuid2"] }
```
A channel can only belong to one project. `POST` performs an atomic
move if the channel is already in a different project. The user must
own the channel being added.
**Channel association object:**
```json
{
"project_id": "uuid",
"channel_id": "uuid",
"position": 0,
"folder": "",
"added_at": "2025-01-15T10:30:00Z"
}
```
**POST request:**
| Field | Required | Notes |
|-------|----------|-------|
| `channel_id` | yes | UUID of channel to add |
| `position` | no | Sort order (default 0) |
---
## KB Association
```
GET /projects/:id/knowledge-bases → { "data": [...] }
POST /projects/:id/knowledge-bases ← { "kb_id", "auto_search" }
DELETE /projects/:id/knowledge-bases/:kbId → { "message": "KB removed" }
```
KBs bound to a project are automatically available to every channel
in that project (resolution chain injection at completion time).
Upsert on conflict — re-posting updates `auto_search`.
**KB association object:**
```json
{
"project_id": "uuid",
"kb_id": "uuid",
"auto_search": true,
"added_at": "2025-01-15T10:30:00Z",
"name": "KB Name"
}
```
`name` is enriched via JOIN from `knowledge_bases.name`.
**POST request:**
| Field | Required | Notes |
|-------|----------|-------|
| `kb_id` | yes | UUID of knowledge base |
| `auto_search` | no | Default `false` |
---
## Note Association
```
GET /projects/:id/notes → { "data": [...] }
POST /projects/:id/notes ← { "note_id" }
DELETE /projects/:id/notes/:noteId → { "message": "note removed" }
```
**Note association object:**
```json
{
"project_id": "uuid",
"note_id": "uuid",
"added_at": "2025-01-15T10:30:00Z",
"title": "Note Title"
}
```
`title` is enriched via JOIN from `notes.title`.
Insert uses ON CONFLICT DO NOTHING (idempotent).
---
## Project Files (v0.37.17 — workspace-backed)
Project files are stored in the project's workspace (auto-created on
first upload). The response uses `"files"` key, not `"data"`. Files
are `WorkspaceFile` objects with tree structure (directories + files).
### List Files
```
GET /projects/:id/files → { "files": [...], "count": N }
?path=/docs&recursive=true
```
| Param | Default | Notes |
|-------|---------|-------|
| `path` | `""` | Directory to list (empty = root) |
| `recursive` | `true` | Include subdirectories |
### Upload File
```
POST /projects/:id/files ← multipart/form-data
?path=/docs
```
Multipart form field: `file`. Optional `path` query param to upload
into a subdirectory. Auto-creates the project workspace on first upload.
Returns `201` with `{ path, filename, content_type, size_bytes }`.
**Errors:** `413` if file exceeds size limit or workspace quota.
### Download File
```
GET /projects/:id/files/download?path=/docs/readme.md
```
Streams raw file content with `Content-Disposition: attachment`.
### Delete File
```
DELETE /projects/:id/files?path=/docs/readme.md
&recursive=false
```
`recursive=true` to delete directories with contents.
### Create Directory
```
POST /projects/:id/files/mkdir ← { "path": "/docs/images" }
```
Path also accepted via `?path=` query param. Returns `201`.
### Upload Archive
```
POST /projects/:id/archive/upload ← multipart/form-data
```
Multipart field: `file`. Accepts `.zip`, `.tar.gz`, `.tgz`.
Extracts contents into the workspace root.
Returns `{ ok: true, files_extracted: N }`.
### Download Archive
```
GET /projects/:id/archive/download?format=zip
```
Bundles all project files into a zip (or `tar.gz`).
Streams with `Content-Disposition: attachment`.
---
## Admin Project Management
```
GET /admin/projects → { "data": [...] }
DELETE /admin/projects/:id → { "message": "project deleted" }
```
Query params: `?include_archived=true`.
Cross-instance visibility for platform admins. Admin list returns
an enriched object with extra fields:
```json
{
"id": "uuid",
"name": "...",
"description": "...",
"scope": "personal",
"owner_id": "uuid",
"team_id": null,
"is_archived": false,
"created_at": "...",
"updated_at": "...",
"channel_count": 5,
"kb_count": 2,
"note_count": 3,
"owner_name": "jdoe"
}
```
Note: admin list object omits `color`, `icon`, `workspace_id`,
and `settings` (uses a local struct, not the full model).
---

221
docs/ICD/providers.md
View File

@@ -1,221 +0,0 @@
# Providers & Routing
The **multi-provider** system. One or more LLM providers are configured,
each with their own API keys, endpoints, and model catalogs. The routing
layer decides which provider handles each request.
### User BYOK Provider Configs
Gated by `allow_user_byok` policy. Personal API keys are encrypted with
the user's UEK (per-user encryption key, Argon2id-derived). Platform
admins cannot recover personal keys.
```
GET /api-configs → { "configs": [safeConfig objects] }
POST /api-configs ← { "name", "provider", "endpoint", "api_key", ... }
GET /api-configs/:id → safeConfig
PUT /api-configs/:id ← partial update (api_key optional)
DELETE /api-configs/:id
```
`safeConfig` — API keys are **never** returned:
```json
{
"id": "uuid",
"name": "My OpenAI",
"provider": "openai",
"endpoint": "https://api.openai.com/v1",
"model_default": "gpt-4o",
"scope": "personal",
"owner_id": "uuid",
"is_active": true,
"has_key": true,
"config": {},
"headers": {},
"settings": {},
"created_at": "..."
}
```
**List models for a user config:**
```
GET /api-configs/:id/models
```
Returns `{ "models": [catalog entries] }`.
**Fetch/sync models from provider API:**
```
POST /api-configs/:id/models/fetch
```
Calls the provider's model list API, upserts into the local catalog.
### Admin Global Provider Configs
```
GET /admin/configs → { "configs": [configWithKey objects] }
POST /admin/configs ← { "name", "provider", "endpoint", "api_key", "config", "headers", "settings", "is_private" }
PUT /admin/configs/:id ← partial update
DELETE /admin/configs/:id
```
`configWithKey` is the same as `safeConfig` but comes from
`ListGlobal` — still redacts API keys, just adds the `has_key` flag.
Admin configs use the `ENCRYPTION_KEY` env var (not per-user UEK).
`is_private`: when true, the config is available for admin-created
Personas but not directly selectable by users.
### Model Catalog (Admin)
The catalog is populated by fetching from provider APIs and stores
model metadata (capabilities, context window, pricing).
```
GET /admin/models → { "models": [catalog entries] }
PUT /admin/models/:id ← { "visibility", "display_name", ... }
PUT /admin/models/bulk ← { "provider_config_id", "visibility" }
DELETE /admin/models/:id
POST /admin/models/fetch ← { "provider_config_id": "uuid|empty" }
```
**Fetch** with empty `provider_config_id` syncs ALL active global
providers. Returns:
```json
{
"message": "models synced",
"added": 5,
"updated": 12,
"total": 47
}
```
Or for multi-provider fetch: `{ "added", "updated", "total", "errors": [...] }`.
**Bulk visibility** sets all models for a provider (or all models
globally if no `provider_config_id`) to the specified visibility
(`enabled`, `disabled`, `team`).
### Provider Health
Real-time health tracking per provider config. The health accumulator
records success/failure/latency on every completion, flushes to DB
every 60 seconds.
**Get all:**
```
GET /admin/providers/health
```
Returns `{ "data": [ProviderHealth objects] }`:
```json
{
"provider_config_id": "uuid",
"provider_config_name": "OpenAI Production",
"status": "healthy|degraded|down",
"error_rate": 0.02,
"avg_latency_ms": 450,
"timeout_rate": 0.01,
"rate_limit_count": 3,
"last_check": "...",
"last_error": "...|null"
}
```
**Get single:**
```
GET /admin/providers/:id/health
```
**Auto-disable:** After `PROVIDER_AUTO_DISABLE_THRESHOLD` consecutive
"down" windows (default: 3), the provider is automatically deactivated.
### Capability Overrides
Admin can override any model capability detected by the catalog or
heuristic layer.
```
GET /admin/capability-overrides → { "data": [...] }
GET /admin/models/:id/capabilities → capabilities for one model
PUT /admin/models/:id/capabilities ← { "field": "value" }
DELETE /admin/models/:id/capabilities/:overrideId
```
Override fields: `supports_vision`, `supports_tools`, `supports_thinking`,
`context_window`, `max_output_tokens`, etc.
### Routing Policies
Policy-based request routing. Evaluated after model/config resolution,
before provider dispatch.
**Admin CRUD:**
```
GET /admin/routing/policies → { "data": [...] }
GET /admin/routing/policies/:id → policy object
POST /admin/routing/policies ← { "name", "scope", "team_id", "priority", "policy_type", "config", "is_active" }
PUT /admin/routing/policies/:id
DELETE /admin/routing/policies/:id
```
Policy object:
```json
{
"id": "uuid",
"name": "Prefer Anthropic",
"scope": "global|team",
"team_id": "uuid|null",
"priority": 10,
"policy_type": "provider_prefer|team_route|cost_limit|model_alias|capability_match",
"config": {},
"is_active": true
}
```
| Policy type | Config | Behavior |
|------------|--------|----------|
| `provider_prefer` | `{ "providers": ["cfg-1", "cfg-2"] }` | Ordered fallback list |
| `team_route` | `{ "providers": ["cfg-1"] }` | Restrict team to specific providers |
| `cost_limit` | `{ "max_cost_per_request": 0.50 }` | Heuristic cost cap |
| `model_alias` | `{ "alias": "fast", "target_model": "...", "target_config": "..." }` | Alias → provider+model rewrite |
| `capability_match` | `{ "require": ["tool_calling"], "prefer": "cheapest" }` | Match cheapest model with required capabilities |
**Dry-run test:**
```
POST /admin/routing/test
```
```json
{
"model": "claude-sonnet-4-20250514",
"user_id": "uuid",
"team_id": "uuid|null"
}
```
Returns the ranked candidate list with health status for each.
### Provider Types
Registry of supported provider types with metadata.
```
GET /admin/provider-types
```
Returns `{ "types": [...] }` with name, display name, default endpoint,
profile schema, and supported features per type.
---

183
docs/ICD/surfaces.md
View File

@@ -1,183 +0,0 @@
# Surfaces
Dynamic surface lifecycle. Core surfaces are registered at startup.
Extension surfaces are uploaded by admins and served from the
`surface_registry` table.
## User Endpoints
### List Enabled Surfaces
```
GET /surfaces
```
Returns enabled surfaces with minimal nav info (no `source` or `enabled`
fields — these are admin concerns).
```json
{
"surfaces": [
{ "id": "chat", "title": "Chat", "route": "/" },
{ "id": "editor", "title": "Editor", "route": "/editor" },
{ "id": "hello-dashboard", "title": "Hello Dashboard", "route": "/s/hello-dashboard" }
]
}
```
**Auth:** Authenticated.
## Admin Endpoints
### List All Surfaces
```
GET /admin/surfaces
```
Returns all surfaces including disabled ones. Each entry is a full
[Surface Registry Object](#surface-registry-object).
```json
{
"surfaces": [
{ "id": "chat", "title": "Chat", "manifest": {...}, "enabled": true, "source": "core", "installed_at": "...", "updated_at": "..." },
{ "id": "hello-dashboard", "title": "Hello Dashboard", "manifest": {...}, "enabled": false, "source": "extension", "installed_at": "...", "updated_at": "..." }
]
}
```
**Auth:** Platform admin.
### Get Surface
```
GET /admin/surfaces/:id
```
Returns full surface details including manifest.
**Response:** A single [Surface Registry Object](#surface-registry-object).
**Errors:**
- `404` — surface not found (or store error).
**Auth:** Platform admin.
### Install Surface
```
POST /admin/surfaces/install
Content-Type: multipart/form-data
```
Field: `file` — a `.surface` or `.zip` archive (zip containing
`manifest.json` + static assets). The manifest declares `id`, `title`,
`route`, required data loaders, scripts, and CSS.
**Size limit:** 50 MB.
**Manifest validation:** `id` and `title` are required.
**Errors:**
- `400` — no file, wrong extension, too large, invalid zip, missing or
invalid manifest, missing `id`/`title`.
- `409` — surface ID conflicts with an existing core surface.
**Auth:** Platform admin.
### Enable / Disable
```
PUT /admin/surfaces/:id/enable
PUT /admin/surfaces/:id/disable
```
Disabled surfaces redirect to `/` and hide from navigation.
**Undisableable surfaces:** `chat` and `admin` cannot be disabled
(returns `400`).
**Errors:**
- `400` — attempting to disable `chat` or `admin`.
- `404` — surface not found.
**Auth:** Platform admin.
### Delete Surface
```
DELETE /admin/surfaces/:id
```
Removes the surface, its DB row, and its extracted static assets.
Core surfaces cannot be deleted.
**Errors:**
- `400` — attempting to delete a core surface.
- `404` — surface not found.
**Auth:** Platform admin.
**Defense-in-depth:** The store layer also enforces
`WHERE source = 'extension'` on delete, so even if the handler check
were bypassed, core surfaces are protected.
## Surface Registry Object
```json
{
"id": "hello-dashboard",
"title": "Hello Dashboard",
"manifest": { "route": "/s/hello-dashboard", "scripts": [...], "css": [...] },
"enabled": true,
"source": "core|extension",
"installed_at": "...",
"updated_at": "..."
}
```
## Internal Store Methods
The `SurfaceRegistryStore` interface exposes methods beyond what the
HTTP API uses directly:
- `Seed(ctx, id, title, source, manifest)` — upserts a surface at
startup. Does **not** overwrite the `enabled` flag (admin toggle
survives restarts). Called by the page engine's `SeedSurfaces()`.
- `ListEnabled(ctx) → []string` — returns IDs of all enabled
surfaces. Used by the page engine for nav rendering
(`EnabledSurfaceIDs()`), not exposed via HTTP.
## Page Routes
Extension surfaces are served at `/s/:slug` via `RenderExtensionSurface()`.
The page engine does a runtime DB lookup — no server restart needed after install.
Static assets are served from `/surfaces/:id/` (nginx location block in
both unified and split deployment). In unified mode, the Go server
handles this route with path-traversal protection. In split deployment,
nginx serves directly from `/data/surfaces/`.
## Surface Archive Format
A `.surface` file is a **zip** archive containing:
```
manifest.json — required (must have "id" and "title")
js/ — optional (extracted to /data/surfaces/{id}/js/)
css/ — optional (extracted to /data/surfaces/{id}/css/)
assets/ — optional (icons, images, etc.)
script.star — optional (Starlark entry point, v0.38.0)
star/ — optional (Starlark submodules, v0.38.0)
```
The archive may have a single top-level directory prefix (e.g.,
`my-surface/manifest.json`) — the installer strips it when locating
`manifest.json` and when extracting `js/`, `css/`, `assets/`, `star/`
directories and bare `.star` files. Files outside these directories
(e.g., `script.js` at root) are not extracted.
See [EXTENSION-SURFACES.md](../EXTENSION-SURFACES.md) for the full
authoring guide including manifest schema, platform API, and CSS
custom properties.

414
docs/ICD/tasks.md
View File

@@ -1,414 +0,0 @@
# Tasks
Autonomous agent scheduling. A task is a cron-driven, one-shot, or
webhook-triggered execution in a headless service channel — no human
participant in the loop.
## Configuration
Global config keys (set via `PUT /admin/settings/:key`):
| Key | Default | Description |
|-----|---------|-------------|
| `tasks.enabled` | `true` | Master kill switch for task scheduler |
| `tasks.allow_personal` | `true` | Allow non-admin users to create tasks |
| `tasks.max_concurrent` | `5` | Max tasks running simultaneously |
| `tasks.personal_require_byok` | `false` | Personal tasks must use BYOK provider |
Default budget ceilings (overridable per task):
| Key | Default |
|-----|---------|
| `tasks.default_max_tokens` | `4096` |
| `tasks.default_max_tool_calls` | `10` |
| `tasks.default_max_wall_clock` | `300` (seconds) |
## Personal Tasks
### List My Tasks
```
GET /tasks
```
Returns tasks owned by the current user.
**Auth:** Authenticated.
### Create Task
```
POST /tasks
```
```json
{
"name": "Morning News Digest",
"description": "Summarize top tech news",
"task_type": "prompt",
"persona_id": "uuid|null",
"model_id": "claude-sonnet-4-20250514",
"system_prompt": "You are a news summarizer.",
"user_prompt": "Summarize the top 5 tech news stories from today.",
"schedule": "@daily",
"timezone": "America/New_York",
"max_tokens": 4096,
"max_tool_calls": 10,
"max_wall_clock": 300,
"output_mode": "channel",
"webhook_url": "https://...",
"provider_config_id": "uuid|null",
"notify_on_complete": false,
"notify_on_failure": true,
"tool_grants": ["web_search", "url_fetch"]
}
```
`task_type`: `prompt` (direct LLM execution), `workflow` (deferred —
not yet implemented; creation is rejected at the API), `action`
(no LLM — relay/webhook only; requires `task.action` permission),
or `system` (v0.28.6 — built-in Go function, admin-only).
System tasks require `system_function` — a registered function name
from the platform's Go function registry. No LLM, no provider
resolution, no channel needed. The function receives the store set
and returns a result string. Designed for core platform ops
(`session_cleanup`, `staleness_check`, `retention_sweep`, `health_prune`)
that must not break from bad user code. Permanent track — not replaced
by Starlark in v0.29.0.
`schedule`: cron expression, `once`, or `webhook`. Supported cron
patterns: `@hourly`, `@daily`, `@weekly`, `*/N * * * *` (every N
minutes), full 5-field cron via `robfig/cron/v3`. When `schedule` is
`webhook`, the task fires only on inbound POST to its trigger URL.
`output_mode`: `channel` (default — output stays in service channel),
`note` (creates a note from the output), `webhook` (POST result to URL).
**Auth:** `tasks.create` permission required. Additionally, `task.action`
permission is required for `task_type: "action"`.
### Get Task
```
GET /tasks/:id
```
**Auth:** Owner or admin.
### Update Task
```
PUT /tasks/:id
```
Partial update. Accepted fields: `name`, `description`, `persona_id`,
`model_id`, `system_prompt`, `user_prompt`, `workflow_id`, `tool_grants`,
`schedule`, `timezone`, `is_active`, `max_tokens`, `max_tool_calls`,
`max_wall_clock`, `output_mode`, `output_channel_id`, `webhook_url`,
`provider_config_id`, `notify_on_complete`, `notify_on_failure`.
**Auth:** `tasks.create` permission, must be owner.
### Delete Task
```
DELETE /tasks/:id
```
**Auth:** `tasks.create` permission, must be owner.
### Run Now (Manual Trigger)
```
POST /tasks/:id/run
```
Immediately executes the task regardless of schedule. Creates a new
task run. Returns 409 if a run is already active or queued.
**Auth:** `tasks.create` permission, must be owner.
### Kill Active Run
```
POST /tasks/:id/kill
```
Cancels the active run. Sets status to `cancelled`.
**Auth:** `tasks.create` permission, must be owner.
### List Runs
```
GET /tasks/:id/runs
```
Returns run history for the task, most recent first.
**Auth:** Owner or admin.
## Webhook Trigger (Inbound)
External systems fire a task by POSTing to its trigger URL.
```
POST /api/v1/hooks/t/:token
```
The `:token` is a per-task secret generated at creation time (returned
in the `trigger_token` field). The request body is stored as
`trigger_payload` on the run record and forwarded to the executor:
- **Prompt tasks:** payload is prepended to the user prompt as context.
- **Action tasks:** payload is relayed to the outbound webhook as-is.
- **Workflow tasks:** (deferred) payload becomes initial stage data.
Returns `202 Accepted`:
```json
{
"triggered": true,
"run_id": "uuid",
"task_id": "uuid"
}
```
Returns `409 Conflict` if a run is already active or queued.
Returns `410 Gone` if the task is inactive.
**Auth:** None (token-based). The trigger token *is* the credential.
**Rate limiting:** Governed by global request rate limits. No per-task
rate limiting in v0.28 — defer to a future version if needed.
### Task-to-Task Chaining (v0.28.6)
Tasks can trigger other tasks by setting the `webhook_url` of Task A
to the trigger URL of Task B:
1. Create **Task B** with `schedule: "webhook"`. Note the trigger URL
from the response (`/api/v1/hooks/t/{trigger_token}`).
2. Create **Task A** with `output_mode: "webhook"` and set `webhook_url`
to Task B's trigger URL.
3. When Task A completes, its output is POSTed to Task B's trigger URL.
Task B starts with Task A's output as its `trigger_payload`.
This enables multi-step pipelines without external orchestration:
- Task A: "Summarize today's news" (prompt, cron: 6am)
- Task B: "Format and post to Slack" (action, webhook-triggered)
The `webhook_secret` on Task A signs outbound payloads with HMAC-SHA256.
Task B's trigger endpoint does not currently verify signatures (the
trigger token itself is the auth mechanism).
## Team Tasks
Team-scoped tasks visible to all team members, manageable by team admins.
### List Team Tasks (Member)
```
GET /teams/:teamId/tasks
```
Returns tasks scoped to the team. Read-only for members.
**Auth:** Team member.
### List Team Task Runs (Member)
```
GET /teams/:teamId/tasks/:id/runs
```
**Auth:** Team member.
### Create Team Task (Admin)
```
POST /teams/:teamId/tasks
```
Same shape as personal task creation. Team scope is injected automatically.
**Auth:** `tasks.create` permission, team admin.
### Update/Delete/Run/Kill Team Task
```
PUT /teams/:teamId/tasks/:id
DELETE /teams/:teamId/tasks/:id
POST /teams/:teamId/tasks/:id/run
POST /teams/:teamId/tasks/:id/kill
```
**Auth:** `tasks.create` permission, team admin.
## Admin Tasks
Platform-wide task management.
### List All Tasks
```
GET /admin/tasks
```
Returns all tasks across all users and teams.
**Auth:** Platform admin.
### Admin Run/Kill/Delete
```
POST /admin/tasks/:id/run
POST /admin/tasks/:id/kill
DELETE /admin/tasks/:id
```
**Auth:** Platform admin.
### System Functions (v0.28.6)
```
GET /admin/system-functions → { "data": [SystemFuncInfo, ...] }
```
Returns all registered built-in system function names and descriptions.
Used by the admin UI to populate the function dropdown when creating
system tasks.
**SystemFuncInfo:**
```json
{
"name": "session_cleanup",
"description": "Remove expired anonymous visitor sessions"
}
```
Built-in functions (v0.28.6): `session_cleanup`, `staleness_check`,
`retention_sweep`, `health_prune`.
**Auth:** Platform admin.
## Task Object
```json
{
"id": "uuid",
"owner_id": "uuid",
"team_id": "uuid|null",
"name": "Morning News Digest",
"description": "...",
"scope": "personal|team|global",
"task_type": "prompt|workflow|action|system",
"system_function": "session_cleanup",
"persona_id": "uuid|null",
"model_id": "claude-sonnet-4-20250514",
"system_prompt": "...",
"user_prompt": "...",
"workflow_id": "uuid|null",
"tool_grants": ["web_search", "url_fetch"],
"schedule": "@daily|once|webhook",
"timezone": "America/New_York",
"is_active": true,
"trigger_token": "hex-string|null",
"max_tokens": 4096,
"max_tool_calls": 10,
"max_wall_clock": 300,
"output_mode": "channel|note|webhook",
"output_channel_id": "uuid|null",
"webhook_url": "https://...",
"webhook_secret": "...",
"provider_config_id": "uuid|null",
"notify_on_complete": false,
"notify_on_failure": true,
"last_run_at": "...|null",
"next_run_at": "...|null",
"run_count": 42,
"created_at": "...",
"updated_at": "..."
}
```
## Task Run Object
```json
{
"id": "uuid",
"task_id": "uuid",
"channel_id": "uuid|null",
"status": "queued|running|completed|failed|budget_exceeded|cancelled",
"trigger_payload": "...|null",
"started_at": "...",
"completed_at": "...|null",
"tokens_used": 1234,
"tool_calls": 3,
"wall_clock": 45,
"error": "...|null"
}
```
## Execution
The `TaskScheduler` is a background goroutine polling every 30 seconds.
1. Finds tasks where `next_run_at <= now` and `is_active = true`
2. Skips if an active run exists (`GetActiveRun` returns non-nil)
3. Adopts a queued run if one exists (from webhook trigger), otherwise
creates a new run record
4. Marks `last_run_at` (records when execution started, not when it finished)
5. Creates or reuses a `service` channel (`output_channel_id`)
6. Persists the `user_prompt` as a message in the service channel
(with trigger payload prepended if present)
7. For **prompt** tasks: runs `CoreToolLoop` (headless completion)
8. For **action** tasks: skips LLM, relays trigger payload to webhook
9. Enforces budgets: `max_tokens`, `max_tool_calls`, `max_wall_clock`
10. On budget breach: status = `budget_exceeded`, owner notified
11. On completion: calculates `next_run_at`
**Workflow tasks:** Not yet implemented. The API rejects
`task_type: "workflow"` at creation time. Reserved for a future version.
**Provider resolution:** BYOK → team provider → global provider → routing
policy. If `tasks.personal_require_byok` is true, personal tasks that
don't have a BYOK provider fail at step 7.
**Action tasks** skip steps 6–9. They create the service channel, fire
the outbound webhook with the trigger payload, and complete.
## Webhooks (Outbound)
Tasks with `webhook_url` fire a POST on completion:
```json
{
"task_id": "uuid",
"run_id": "uuid",
"task_name": "Morning News Digest",
"channel_id": "uuid",
"status": "completed|failed|budget_exceeded",
"completed_at": "2025-03-11T12:00:00Z",
"output": "...",
"tokens_used": 1234,
"error": "...|null"
}
```
Signed with HMAC-SHA256 using `webhook_secret` in the
`X-Switchboard-Signature` header. Retry: 3 attempts, exponential
backoff (1s, 5s, 25s).
## Task Chaining
Task A's outbound `webhook_url` can be Task B's trigger URL:
```
Task A (webhook_url) → POST /api/v1/hooks/t/<task-B-token> → Task B runs
```
This enables multi-step pipelines: CI failure → summarize with LLM →
post to Slack. Each link is a separate task with its own type, budget,
and notification preferences.

258
docs/ICD/teams.md
View File

@@ -1,258 +0,0 @@
# Teams & Access Control
### My Teams
```
GET /teams/mine → { "data": [...] }
```
**Auth:** Authenticated user.
Returns teams the current user is a member of (active teams only).
Each team object:
```json
{
"id": "uuid",
"name": "Engineering",
"description": "...",
"is_active": true,
"settings": "{}",
"my_role": "admin|member",
"member_count": 5
}
```
### Team Administration
**Team CRUD (platform admin):**
```
GET /admin/teams → { "data": [...], "total": N, "page": N, "per_page": N }
POST /admin/teams ← { "name", "description" } → { "id", "name" }
GET /admin/teams/:id → team object (unwrapped)
PUT /admin/teams/:id ← { "name"?, "description"?, "is_active"?, "settings"? }
DELETE /admin/teams/:id
```
**Auth:** `RequireAdmin` (platform admin).
`POST` returns `201`. `PUT`/`DELETE` return `{"ok": true}`. `PUT` with
no fields returns `400`. Duplicate name returns `409`.
**Members (admin routes):**
```
GET /admin/teams/:id/members → { "data": [...] }
POST /admin/teams/:id/members ← { "user_id", "role" } → { "id" }
PUT /admin/teams/:id/members/:memberId ← { "role" }
DELETE /admin/teams/:id/members/:memberId
```
**Auth:** `RequireAdmin`.
`role` must be `admin` or `member`. Duplicate membership returns `409`.
**Team-scoped routes** (team admin self-service):
```
GET /teams/:teamId/members
POST /teams/:teamId/members ← { "user_id", "role" }
PUT /teams/:teamId/members/:memberId
DELETE /teams/:teamId/members/:memberId
```
**Auth:** `RequireTeamAdmin`.
Same handlers as admin routes; the `getTeamID()` helper reads either
`:id` or `:teamId`.
**Team Providers:**
```
GET /teams/:teamId/providers → { "data": [...], "allow_team_providers": bool }
POST /teams/:teamId/providers ← { "name", "provider", "endpoint", "api_key", ... }
PUT /teams/:teamId/providers/:id
DELETE /teams/:teamId/providers/:id
GET /teams/:teamId/providers/:id/models → { "models": [...], "provider": "name" }
```
**Auth:** `RequireTeamAdmin`.
`GET` returns provider configs scoped to the team. The `models` endpoint
performs a live query to the upstream provider.
**Team Models:**
```
GET /teams/:teamId/models → { "models": [...] }
```
**Auth:** `RequireTeamAdmin`.
Returns all models available to this team: global admin models with
`visibility IN ('enabled', 'team')` plus live-queried team provider
models with `source` field (`"global"` or `"team"`).
**Team Groups:**
```
GET /teams/:teamId/groups → { "data": [...] }
```
**Auth:** `RequireTeamAdmin`.
Returns groups scoped to this team.
**Team Personas:** See personas.md §Team Personas.
**Team Roles:**
```
GET /teams/:teamId/roles → { "data": {...} }
PUT /teams/:teamId/roles/:role ← RoleConfig object
DELETE /teams/:teamId/roles/:role
```
**Auth:** `RequireTeamAdmin`.
`GET` returns the team's `model_roles` settings map (composite object,
not an array). Initially empty `{}`. Each key is a role name, value is
a `RoleConfig`. `PUT` validates the role name. `DELETE` removes the
override, falling back to the global role definition.
**Team Audit:**
```
GET /teams/:teamId/audit → { "data": [...], "total": N, "page": N, "per_page": N }
GET /teams/:teamId/audit/actions → { "actions": [...] }
```
**Auth:** `RequireTeamAdmin`.
Audit log is scoped to actions performed by team members. Supports
query filters: `?action=`, `?actor_id=`, `?resource_type=`.
**Team Usage:**
```
GET /teams/:teamId/usage → { "totals": {...}, "results": [...] }
```
**Auth:** `RequireTeamAdmin`.
Returns usage against team-owned providers. Supports query params for
date range filtering.
### Groups & Resource Grants
Groups are ACL containers that decouple access from team membership.
A resource grant controls who can see a Persona or KB beyond its base
scope.
**My Groups:**
```
GET /groups/mine → { "data": [...] }
```
**Auth:** Authenticated user.
**Admin Group CRUD:**
```
GET /admin/groups → { "data": [...] }
POST /admin/groups ← { "name", "scope", "team_id"? }
GET /admin/groups/:id → group object (unwrapped)
PUT /admin/groups/:id ← { "name"?, "description"?, "permissions"?, ... }
DELETE /admin/groups/:id
```
**Auth:** `RequireAdmin`.
`POST` returns `201`. Duplicate name returns `409`. Deleting a
system-sourced group (e.g. Everyone) returns `400`.
**Group Members:**
```
GET /admin/groups/:id/members → { "data": [...] }
POST /admin/groups/:id/members ← { "user_id" }
DELETE /admin/groups/:id/members/:userId
```
**Auth:** `RequireAdmin`.
**Resource Grants:**
```
GET /admin/grants/:type/:id → grant object (unwrapped)
PUT /admin/grants/:type/:id ← { "grant_scope", "granted_groups"?: [...] }
DELETE /admin/grants/:type/:id
```
**Auth:** `RequireAdmin`.
`:type` is `persona` or `kb`. `:id` is the resource ID.
Grant scopes:
| `grant_scope` | Visibility |
|---------------|-----------|
| `team_only` | Only the owning team |
| `global` | All authenticated users |
| `groups` | Members of specified groups |
Setting `grant_scope: "groups"` without `granted_groups` returns `400`.
### Permissions
Groups carry fine-grained permissions. Permission resolution:
union of all group permissions + Everyone group permissions.
**List all permissions:**
```
GET /admin/permissions → { "permissions": ["model.use", "kb.read", ...] }
```
**Auth:** `RequireAdmin`.
**Get user's effective permissions:**
```
GET /admin/users/:id/permissions → { "permissions": [...], "user_id": "...", "groups": [...] }
```
**Auth:** `RequireAdmin`.
Returns the resolved permission set and contributing group IDs
(always includes the Everyone group).
**Group permission fields** (set via `PUT /admin/groups/:id`):
```json
{
"permissions": ["model.use", "kb.read", "channel.create"],
"token_budget_daily": 100000,
"token_budget_monthly": 3000000,
"allowed_models": ["claude-sonnet-4-20250514", "gpt-4o"]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `permissions` | string[] | Permission constants granted to members |
| `token_budget_daily` | int/null | Daily token ceiling (NULL = unlimited) |
| `token_budget_monthly` | int/null | Monthly token ceiling (NULL = unlimited) |
| `allowed_models` | string[]/null | Model allowlist (NULL = unrestricted) |
**Everyone group:** System-seeded group (ID `00000000-0000-0000-0000-000000000001`)
whose permissions apply to all authenticated users implicitly (no membership
row needed). Editable by admins, cannot be deleted (`source=system`).
**RequirePermission middleware:** Routes gated by permission check user's
resolved permission set. Returns 403 if the required permission is not present.
See [enums.md](enums.md) for the full permission constant list.

146
docs/ICD/utility.md
View File

@@ -1,146 +0,0 @@
# Export & Utility
### Health Check
```
GET /health → { "status": "ok" }
```
Public, no auth. Also available at `/api/v1/health`.
### Export
Convert markdown to PDF or DOCX via pandoc:
```
POST /export
```
```json
{
"content": "# My Document\n\nBody text...",
"format": "pdf|docx",
"filename": "my-document"
}
```
Returns the binary file with appropriate Content-Type. Requires
`pandoc` in the container (available in the unified and backend
Docker images).
---
## 17b. Files & Storage
All binary content in Chat Switchboard flows through a single
**ObjectStore** abstraction backed by PVC (filesystem) or S3. All
file metadata lives in one `files` table. The old `attachments`
table is dropped.
### Storage Backend
| Operation | Description |
|-----------|-------------|
| `Put(key, reader, size, contentType)` | Store a blob |
| `Get(key)` → reader, size, contentType | Retrieve a blob |
| `Delete(key)` | Remove a blob |
| `DeletePrefix(prefix)` | Bulk remove (channel deletion) |
| `Exists(key)` | Check without reading |
| `Healthy()` | Backend health check |
| `Stats()` | Aggregate metrics |
Backends: `pvc` (`STORAGE_PATH=/data/storage`) or `s3`
(`S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`).
### Key Namespacing
| Prefix | Pattern | What |
|--------|---------|------|
| `attachments/` | `attachments/{channel_id}/{file_id}_{filename}` | User uploads + AI outputs (legacy prefix retained for existing blobs) |
| `kb/` | `kb/{kb_id}/{doc_id}_{filename}` | Knowledge base documents |
| `exports/` | `exports/{user_id}/{export_id}_{filename}` | Chat/data exports |
| `workspace/` | `workspace/{workspace_id}/{path}` | Workspace files |
### File Object
```sql
CREATE TABLE files (
id UUID PRIMARY KEY,
channel_id UUID REFERENCES channels(id) ON DELETE CASCADE,
message_id UUID REFERENCES messages(id) ON DELETE SET NULL,
user_id UUID REFERENCES users(id) ON DELETE SET NULL,
project_id UUID REFERENCES projects(id) ON DELETE SET NULL,
origin TEXT NOT NULL DEFAULT 'user_upload'
CHECK (origin IN ('user_upload','tool_output','system')),
filename TEXT NOT NULL,
content_type TEXT NOT NULL DEFAULT 'application/octet-stream',
size_bytes BIGINT NOT NULL DEFAULT 0,
storage_key TEXT NOT NULL,
display_hint TEXT NOT NULL DEFAULT 'download'
CHECK (display_hint IN ('inline','download','thumbnail')),
extracted_text TEXT,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
```
**Origin:**
| Value | Who creates | Examples |
|-------|-------------|---------|
| `user_upload` | User via upload UI | PDF, image, doc attached to message |
| `tool_output` | Tool execution during completion | DALL-E image, video gen, code artifact |
| `system` | Platform | Export, avatar stored as file |
**Display hint:**
| Value | Frontend behavior |
|-------|-------------------|
| `inline` | Render in message stream (images, video, audio, HTML) |
| `download` | Filename + size + download button |
| `thumbnail` | Thumbnail preview, click to expand |
### Tool Output Flow
1. Tool executes (image gen, video render, etc.)
2. Tool writes blob via `ObjectStore.Put`
3. Tool creates file record: `POST /files` with `origin: "tool_output"`, `message_id` = current assistant message
4. Tool result includes `file_id` reference
5. WebSocket emits `file.created` event:
```json
{
"event": "file.created",
"channel_id": "uuid",
"message_id": "uuid",
"file": { ...file object... }
}
```
6. Frontend renders inline based on `display_hint` + `content_type`
The event fires during streaming so the user sees the artifact
before the completion finishes.
### Content Type → Display Hint Defaults
| Content Type | Default Hint | Rendering |
|-------------|-------------|-----------|
| `image/*` | `inline` | `<img>` in message, lightbox on click |
| `video/*` | `inline` | `<video>` player |
| `audio/*` | `inline` | `<audio>` player |
| `text/html` | `inline` | Sandboxed iframe |
| `application/pdf` | `thumbnail` | Thumb + PDF viewer on click |
| `text/*`, `application/json` | `inline` | Syntax-highlighted code block |
| Everything else | `download` | Filename + download button |
Tools can override the default hint explicitly.
### File Manager (Future Surface)
A user-scoped file browser backed by `GET /files?page=1&per_page=50`.
Displays all files owned by the user across all channels, grouped by
channel or date. Supports download and delete. Deleting a file that
is referenced by a message replaces the inline render with a
"file deleted" placeholder.

507
docs/ICD/websocket.md
View File

@@ -1,507 +0,0 @@
# WebSocket Protocol
Real-time event delivery. Single connection per client. All event
delivery uses targeted `SendToUser` — events are pushed to every
WebSocket connection belonging to the recipient user.
### Connection
**Preferred (v0.28.8+): Ticket exchange**
```
POST /api/v1/ws/ticket (authenticated via JWT Bearer header)
→ 200 { "ticket": "<opaque-hex-string>" }
```
Then connect with the opaque ticket:
```
ws://{host}{BASE_PATH}/ws?ticket={ticket}
```
Tickets are single-use (deleted on validation), 30-second TTL, and
stored in server memory. This avoids exposing the JWT in server logs,
proxy logs, and browser history.
**Legacy (deprecated): JWT query parameter**
```
ws://{host}{BASE_PATH}/ws?token={access_token}
```
Still accepted for backward compatibility with pre-v0.28.8 clients.
Server logs a deprecation warning on each legacy connection. Will be
removed in a future version.
**Non-browser clients** may also use the `Authorization: Bearer <jwt>`
header on the upgrade request.
**Origin validation:** The WebSocket upgrader validates the `Origin`
header against `CORS_ALLOWED_ORIGINS`. Connections from unlisted
origins are rejected at upgrade time. In development (no
`CORS_ALLOWED_ORIGINS` set), all origins are accepted.
**Lifecycle:**
- Server sends WebSocket-level `ping` frames every 54 seconds
- Client must respond with `pong` within 60 seconds or disconnection
- Max inbound message size: 4 KB
- Write deadline: 10 seconds per frame
- Connection ID: `{user_id}-{timestamp}` (internal, not exposed to client)
### Event Envelope
All messages are JSON. The field name is `event`, not `type`:
```json
{
"event": "message.created",
"payload": { ... },
"ts": 1710000000000
}
```
| Field | Type | Description |
|-------|------|-------------|
| `event` | string | Dot-delimited event label |
| `room` | string | Optional — room scope (currently unused client-side, see § Room Model) |
| `payload` | object | Event-specific data |
| `ts` | int64 | Unix milliseconds |
### Client → Server Events
The client can send JSON events to the server. Only events with
`DirFromClient` or `DirBoth` routing are accepted; all others are
silently dropped.
**Application-level ping:**
```json
{ "event": "ping" }
```
Server responds with `{ "event": "pong", "ts": ... }`. This is
separate from the WebSocket-level ping/pong frames.
**Typing (human → other participants):**
Client sends a `chat.typing.{channelID}` event. The server tags it
with `SenderID` and `ConnID`, then rebroadcasts to other participants
via `SendToUser` per channel membership.
### Room Model (Planned)
`JoinRoom` and `LeaveRoom` methods exist on the connection struct but
are **not yet wired** — no client-side subscribe/unsubscribe mechanism
is implemented. The `subscribeToBus` handler has room filtering logic
(`if e.Room != "" && !c.rooms[e.Room]`) but since connections never
join rooms, all room-scoped `Bus.Publish` calls are effectively
no-ops for WebSocket delivery.
Current workaround: all user-facing events use `Hub.SendToUser()`
which bypasses room filtering. Room-scoped `Bus.Publish()` calls
(e.g. `workflow.claimed` with `Room: assignmentID`) exist as
forward-looking plumbing but do not reach WebSocket clients today.
### Event Routing Table
Direction is determined by `events.routeTable` in `server/events/types.go`.
Prefix matching: longest matching prefix wins; unmatched labels default
to `DirLocal` (server-only, never crosses the wire).
| Label | Direction | Delivery | Description |
|-------|-----------|----------|-------------|
| `message.created` | → client | `SendToUser` per channel | New message in channel |
| `message.updated` | → client | `SendToUser` | Message content changed |
| `message.deleted` | → client | `SendToUser` | Message removed |
| `channel.created` | → client | `SendToUser` | New channel created |
| `channel.updated` | → client | `SendToUser` | Channel metadata changed |
| `channel.deleted` | → client | `SendToUser` | Channel removed |
| `channel.member.*` | → client | `SendToUser` | Participant added/removed/role changed |
| `typing.start` | → client | `SendToUser` | AI persona started generating |
| `typing.stop` | → client | `SendToUser` | AI persona finished generating |
| `typing.user` | → client | `SendToUser` per channel | Human typing indicator |
| `chat.typing.*` | ↔ both | Bus (prefix match) | Legacy typing relay |
| `user.presence` | → client | `SendToUser` | Online/offline status change |
| `user.mentioned` | → client | `SendToUser` | User @mentioned in a channel |
| `notification.new` | → client | `SendToUser` | New persisted notification |
| `notification.read` | → client | `SendToUser` | Badge sync across tabs |
| `role.fallback` | → client | Bus (Room: `admin`) | Role fallback activated (admin-targeted) |
| `workflow.assigned` | → client | `SendToUser` per team | New assignment for team members |
| `workflow.claimed` | → client | `SendToUser` per channel | Assignment claimed |
| `workflow.advanced` | → client | `SendToUser` per channel | Stage advanced |
| `workflow.completed` | → client | `SendToUser` per channel | Workflow finished |
| `workspace.file.*` | → client | Bus | Workspace file changed |
| `tool.call.*` | → client | `SendToUser` | Browser tool invocation |
| `tool.result.*` | ← client | Bus | Browser tool result |
| `system.notify` | → client | Bus | System broadcast |
| `ping` | ← client | Direct | Application-level keepalive |
| `pong` | → client | Direct | Response to application ping |
---
## Event Payload Shapes
### `message.created`
User message (mention_only / DM relay):
```json
{
"id": "uuid",
"channel_id": "uuid",
"role": "user",
"content": "message text",
"user_id": "uuid"
}
```
Assistant message (completion chain):
```json
{
"id": "uuid",
"channel_id": "uuid",
"role": "assistant",
"content": "response text",
"model": "model-id",
"participant_type": "persona",
"participant_id": "uuid",
"display_name": "Persona Name",
"avatar": "/path/to/avatar",
"tokens_used": 1234,
"chain_depth": 1
}
```
### `typing.start` / `typing.stop`
AI typing indicator (targeted via `SendToUser`):
```json
{
"channel_id": "uuid",
"participant_id": "uuid",
"participant_type": "persona",
"display_name": "Persona Name"
}
```
### `typing.user`
Human typing indicator (relayed to other channel participants):
```json
{
"channel_id": "uuid",
"user_id": "uuid",
"display_name": "Jane Doe"
}
```
### `user.presence`
Emitted on WebSocket connect/disconnect:
```json
{
"user_id": "uuid",
"status": "online|offline"
}
```
Note: the DB CHECK constraint allows `online`, `away`, `offline` but
the WebSocket hub currently only emits `online` (on connect) and
`offline` (on last connection closed). `away` is reserved for future
idle detection.
### `user.mentioned`
Targeted to the mentioned user:
```json
{
"channel_id": "uuid",
"from_user": "uuid",
"content": "truncated to 120 chars..."
}
```
### `notification.new`
Full notification object (same shape as REST `GET /notifications`):
```json
{
"id": "uuid",
"user_id": "uuid",
"type": "kb.ready",
"title": "Knowledge base ready",
"body": "optional detail",
"resource_type": "knowledge_base",
"resource_id": "uuid",
"is_read": false,
"created_at": "..."
}
```
### `notification.read`
Single notification marked read:
```json
{ "id": "uuid" }
```
Mark-all-read:
```json
{ "action": "mark_all_read" }
```
### `workflow.assigned`
Targeted via `SendToUser` to each team member:
```json
{
"channel_id": "uuid",
"team_id": "uuid",
"stage_name": "Review",
"assigned_to": "uuid|empty"
}
```
### `workflow.claimed`
Targeted via `SendToUser` to all user participants in the channel:
```json
{
"assignment_id": "uuid",
"claimed_by": "uuid",
"channel_id": "uuid"
}
```
### `workflow.advanced`
Targeted via `SendToUser` to all user participants in the channel:
```json
{
"channel_id": "uuid",
"workflow_id": "uuid",
"stage": "stage_key",
"stage_name": "Stage Name"
}
```
### `workflow.completed`
Targeted via `SendToUser` to all user participants in the channel:
```json
{
"channel_id": "uuid",
"workflow_id": "uuid",
"stage": "final_stage_key"
}
```
### `role.fallback`
Published to bus with `Room: "admin"`:
```json
{
"role": "primary",
"operation": "chat|embedding",
"primary_model": "model-a",
"fallback_model": "model-b",
"message": "Role \"primary\" primary (model-a) failed — using fallback (model-b)"
}
```
### `tool.call.*`
Browser tool invocation (targeted via `SendToUser`):
```json
{
"tool_name": "...",
"call_id": "...",
"input": { ... }
}
```
Client responds with `tool.result.*` containing the execution result.
---
## Appendix: Enums
### Channel Types
`direct` (single-user), `dm` (human-to-human, AI mention-only),
`group` (multi-user/multi-model), `channel` (named persistent),
`workflow` (staged), `service` (autonomous task execution)
### Channel AI Modes
`auto` (default), `mention_only` (default for `dm`), `off`
### Participant Types
`user`, `persona`, `session`
### Participant Roles
`owner`, `member`, `observer`, `visitor`
### Presence Statuses
`online`, `away`, `offline`
DB CHECK constraint: `online`, `away`, `offline`. Runtime: only
`online` and `offline` are emitted by the WebSocket hub. `away` is
reserved for future idle-timeout detection.
### Message Roles
`user`, `assistant`, `system`, `tool`
### Scopes
`personal`, `team`, `global`
### Visibility (model catalog)
`enabled`, `disabled`, `team`
### User Roles
`admin`, `user`
### Team Member Roles
`admin`, `member`
### Provider Types
`openai`, `anthropic`, `openrouter`, `venice` (extensible via registry)
### Model Types
`chat`, `embedding`, `image`
### Memory Scopes
`user`, `persona`, `persona_user`
### Memory Statuses
`active`, `pending_review`, `archived`
### Workspace Owner Types
`user`, `project`
### Workspace Statuses
`active`, `archived`, `deleting`
### Index Statuses
`pending`, `indexing`, `ready`, `error`, `skipped`
### KB Document Statuses
`pending`, `chunking`, `embedding`, `ready`, `error`
### Provider Health Statuses
`healthy`, `degraded`, `down`
### Routing Policy Types
`provider_prefer`, `team_route`, `cost_limit`, `model_alias`
### Extension Tiers
`browser` (implemented), `starlark` (future), `sidecar` (future)
### Grant Types
`team_only`, `global`, `groups`
### Resource Grant Scopes
`persona`, `kb`
### Notification Types
`role.fallback`, `kb.ready`, `kb.error`, `grant.changed`,
`memory.extracted`, `user.mentioned`, `workflow.assigned`,
`workflow.claimed`, `task.completed`, `task.failed`,
`task.budget_exceeded`
See [enums.md](enums.md#notification-types) for full descriptions.
### Workflow Assignment Statuses
`unassigned`, `claimed`, `completed`
### Task Run Statuses
`queued`, `running`, `completed`, `failed`, `budget_exceeded`, `cancelled`
### Workflow Instance Statuses
`active`, `completed`, `stale`, `cancelled`
### Git Auth Types
`https_pat`, `https_basic`, `ssh_key`
### Policies
| Key | Default | Description |
|-----|---------|-------------|
| `allow_registration` | `"true"` | Allow new user self-registration |
| `allow_user_byok` | `"true"` | Allow users to add personal API keys |
| `allow_user_personas` | `"true"` | Allow users to create personal Personas |
| `allow_team_providers` | `"true"` | Allow team admins to configure team providers |
| `kb_direct_access` | `"true"` | Show KB picker in channel (false = strict enterprise mode) |
| `require_email` | `"false"` | Require email on registration |
| `default_system_prompt` | `""` | Injected into all conversations (admin override) |
---
## Page Routes (Non-API)
Server-rendered Go template surfaces. Not REST endpoints — these
return HTML pages.
| Route | Surface | Description |
|-------|---------|-------------|
| `/login` | Login | Standalone login/register page |
| `/` | Chat | Main chat interface |
| `/chat/:chatID` | Chat | Chat with specific channel loaded |
| `/editor` | Editor | Workspace file editor |
| `/editor/:wsId` | Editor | Editor with specific workspace |
| `/notes` | Notes | Notes interface |
| `/notes/:noteId` | Notes | Notes with specific note loaded |
| `/admin` | Admin | Platform administration |
| `/admin/:section` | Admin | Admin with specific section |
| `/settings` | Settings | User settings |
| `/settings/:section` | Settings | Settings with specific section |
All page routes (except `/login`) require authentication via
`AuthOrRedirect` middleware (reads `sb_token` cookie). Admin routes
additionally require `RequireAdminPage()`.
**Dynamic surface routing** for admin-installed extension surfaces is
not yet implemented. The current five surfaces are hardcoded in Go
route registration. Future: extension manifests declare surface routes,
the page engine dynamically registers them with appropriate data
loaders. See EXTENSIONS.md for the design direction.

815
docs/ICD/workflows.md
View File

@@ -1,815 +0,0 @@
# Workflows
Team-owned, stage-based process execution. The channel is the runtime —
personas drive each stage, and the existing tool/notes infrastructure
handles structured data collection.
## Definitions
### List Workflows
```
GET /workflows?team_id=...
```
Returns `{ "data": [...] }`. If `team_id` is provided, scoped to that team.
Otherwise returns global (team_id IS NULL) workflows.
**Auth:** Authenticated.
### Create Workflow
```
POST /workflows
```
```json
{
"name": "Customer Intake",
"slug": "customer-intake",
"description": "Collect customer info and route to support",
"entry_mode": "public_link",
"team_id": "uuid|null",
"branding": { "accent_color": "#2563eb", "tagline": "Welcome" },
"retention": { "mode": "archive", "delete_after_days": 90 }
}
```
`slug` auto-generated from `name` if omitted. Must be 2-64 chars,
lowercase alphanumeric and hyphens. Unique within scope (team or global).
`entry_mode`: `public_link` (visitors can start via URL) or `team_only`
(only team members can start instances).
**Auth:** `workflow.create` permission required.
**Response:** 201 with workflow object.
### Get Workflow
```
GET /workflows/:id
```
Returns workflow with its stages included in the `stages` array.
**Auth:** Authenticated.
### Update Workflow
```
PATCH /workflows/:id
```
Partial update. Accepted fields: `name`, `description`, `branding`,
`entry_mode`, `is_active`, `on_complete`, `retention`, `webhook_url`.
Any edit increments the `version` number.
**Auth:** `workflow.create` permission required.
### Delete Workflow
```
DELETE /workflows/:id
```
Cascades: deletes all stages, versions, and active instances.
**Auth:** `workflow.create` permission required.
### Workflow Object
```json
{
"id": "uuid",
"team_id": "uuid|null",
"name": "Customer Intake",
"slug": "customer-intake",
"description": "...",
"branding": { "accent_color": "#2563eb", "logo_url": "...", "tagline": "..." },
"entry_mode": "public_link|team_only",
"is_active": true,
"version": 2,
"on_complete": { "action": "start_workflow", "target_slug": "follow-up", "data_map": {"name": "customer_name"} },
"retention": { "mode": "archive|delete", "delete_after_days": 90 },
"webhook_url": "https://...",
"webhook_secret": "...",
"created_by": "uuid",
"created_at": "...",
"updated_at": "...",
"stages": [...]
}
```
## Stages
Ordered steps within a workflow. Each stage has a driving persona,
optional human assignment team, and a **stage mode** that controls
how data is collected.
### Stage Modes (v0.29.3)
| Mode | Description |
|------|-------------|
| `chat_only` | Default. Persona drives the conversation. |
| `form_only` | UI-rendered form, no LLM. Visitor fills out fields. |
| `form_chat` | Both form and chat available. Persona assists alongside the form. |
| `review` | Human review queue. Shows structured stage_data with approve/reject. |
### List Stages
```
GET /workflows/:id/stages
```
Returns `{ "data": [...] }` ordered by `ordinal`.
**Auth:** Authenticated.
### Create Stage
```
POST /workflows/:id/stages
```
```json
{
"name": "Collect Info",
"ordinal": 0,
"persona_id": "uuid|null",
"assignment_team_id": "uuid|null",
"stage_mode": "chat_only|form_only|form_chat|review",
"form_template": { "fields": [...] },
"history_mode": "full|summary|fresh",
"auto_transition": false,
"transition_rules": { "auto_assign": "round_robin", "conditions": [...] },
"sla_seconds": 3600
}
```
`stage_mode`: controls data collection method. Default: `chat_only`.
`form_only` and `form_chat` require a typed `form_template` with fields.
`history_mode`: what chat history the next stage sees. `full` = complete,
`summary` = utility-role summary, `fresh` = clean slate.
`form_template`: when `stage_mode` is `form_only` or `form_chat`, the
template uses a typed schema (see below). For `chat_only` stages, legacy
freeform templates are still supported as guidance for the LLM.
`transition_rules.auto_assign`: `round_robin` assigns to team members
in rotation. Null = unassigned (manual claim).
`transition_rules.conditions`: array of routing conditions (v0.35.0).
See [Conditional Routing](#conditional-routing-v035).
`sla_seconds`: optional SLA budget in seconds for this stage. Used by
the monitoring dashboard for deadline tracking. Null = no SLA.
**Auth:** `workflow.create` permission required.
### Typed Form Template (v0.29.3)
```json
{
"fields": [
{ "key": "name", "type": "text", "label": "Full Name", "required": true, "validation": { "min_length": 2 } },
{ "key": "email", "type": "email", "label": "Email", "required": true },
{ "key": "dept", "type": "select", "label": "Department", "options": [
{ "value": "eng", "label": "Engineering" },
{ "value": "sales", "label": "Sales" }
]},
{ "key": "notes", "type": "textarea", "label": "Additional Notes",
"condition": { "when": "dept", "op": "eq", "value": "eng" } }
],
"hooks": {
"package_id": "uuid",
"validate": "on_validate",
"on_submit": "on_form_submit"
}
}
```
**Field types:** `text`, `email`, `number`, `date`, `textarea`, `select`, `checkbox`, `file`.
**Validation rules** (in `validation` object per field):
- `min_length`, `max_length` — string length bounds
- `pattern` — regex pattern for text/email fields
- `min`, `max` — numeric bounds for number fields
- `min_date`, `max_date` — date range bounds (YYYY-MM-DD format)
**Conditional fields (v0.35.0):** A field can have a `condition` object
that controls visibility. When the condition is not met, the field is
hidden on the client and skipped during server-side validation.
```json
{ "when": "field_key", "op": "eq|neq|in|exists", "value": "..." }
```
- `eq` (default): field is shown when the referenced field equals `value`
- `neq`: shown when not equal
- `in`: shown when the referenced field's value is in the `value` array
- `exists`: shown when the referenced field has any non-empty value
**Progressive forms / fieldsets (v0.35.0):** For multi-step forms within
a single stage, use `fieldsets` instead of (or alongside) `fields`:
```json
{
"fieldsets": [
{ "label": "Contact Info", "fields": [
{ "key": "name", "type": "text", "label": "Name", "required": true },
{ "key": "email", "type": "email", "label": "Email", "required": true }
]},
{ "label": "Details", "fields": [
{ "key": "description", "type": "textarea", "label": "Description" }
]}
]
}
```
When `fieldsets` is present, the visitor sees one fieldset at a time with
next/back navigation. All fields from all fieldsets are submitted together.
On the server, `ParseTypedFormTemplate` flattens fieldsets into the
`fields` array for validation — backward compatible with existing logic.
**Hooks:** optional Starlark hooks via extension packages.
- `validate`: called before form submission is accepted; can return field errors
- `on_submit`: fire-and-forget hook after successful submission
Requires `forms.validate` extension permission.
### Update Stage
```
PUT /workflows/:id/stages/:sid
```
Full replacement of stage fields.
**Auth:** `workflow.create` permission required.
### Delete Stage
```
DELETE /workflows/:id/stages/:sid
```
**Auth:** `workflow.create` permission required.
### Reorder Stages
```
PATCH /workflows/:id/stages/reorder
```
```json
{ "ordered_ids": ["stage-uuid-2", "stage-uuid-1", "stage-uuid-3"] }
```
Sets ordinals based on array position.
**Auth:** `workflow.create` permission required.
### Stage Object
```json
{
"id": "uuid",
"workflow_id": "uuid",
"ordinal": 0,
"name": "Collect Info",
"stage_mode": "chat_only",
"persona_id": "uuid|null",
"assignment_team_id": "uuid|null",
"form_template": {},
"history_mode": "full",
"auto_transition": false,
"transition_rules": {},
"surface_pkg_id": "uuid|null",
"sla_seconds": 3600,
"created_at": "..."
}
```
## Versions (Immutable Snapshots)
### Publish
```
POST /workflows/:id/publish
```
Creates an immutable snapshot of the current workflow definition, stages,
and persona tool grants. Running instances are pinned to their version.
Returns 201 with the version object. Returns 409 if the current version
number has already been published (edit the workflow to increment first).
**Auth:** `workflow.create` permission required.
### Get Version
```
GET /workflows/:id/versions/:version
```
Returns the version snapshot.
**Auth:** Authenticated.
### Version Object
```json
{
"id": "uuid",
"workflow_id": "uuid",
"version_number": 2,
"snapshot": { "workflow": {...}, "stages": [...] },
"created_at": "..."
}
```
## Instances
A workflow **instance** is a channel with `type=workflow`. The channel's
`workflow_id`, `workflow_version`, `current_stage`, `stage_data`, and
`workflow_status` columns track instance state.
### Start Instance
```
POST /workflows/:id/start
```
Creates a new workflow channel, sets `current_stage=0`, pins the latest
published version. Returns the channel object.
**Auth:** Authenticated (for team members). Visitors use the entry API.
### Get Instance Status
```
GET /channels/:id/workflow/status
```
```json
{
"workflow_id": "uuid",
"workflow_version": 2,
"current_stage": 1,
"stage_data": { "name": "Jane", "email": "jane@co.com" },
"status": "active|completed|stale",
"last_activity_at": "...",
"stage_entered_at": "..."
}
```
**Auth:** Authenticated.
### Advance Stage
```
POST /channels/:id/workflow/advance
```
```json
{ "data": { "name": "Jane", "email": "jane@co.com" } }
```
Merges `data` into the channel's accumulated stage data. The next stage
is determined by the **conditional routing engine** (v0.35.0):
`transition_rules.conditions` are evaluated against the merged stage
data. First matching condition selects the target stage; if no conditions
match or none are defined, advances to `ordinal + 1`. If the target
is beyond the last stage, marks the workflow as `completed` and sets
`ai_mode=off`.
After a successful transition, the `on_advance` hook fires if configured
(see below). On completion, triggers `on_complete` chaining and webhook
delivery if configured.
**Auth:** Authenticated.
### Reject (Go Back)
```
POST /channels/:id/workflow/reject
```
```json
{ "reason": "Missing required field: email" }
```
Moves back one stage. Cannot go below stage 0. `reason` is required —
persisted as a system message in the channel history.
**Auth:** Authenticated.
### Workflow Advance Tool
The `workflow_advance` tool is available to personas during workflow
completions. When the LLM determines enough data has been collected,
it calls this tool to trigger the stage transition programmatically.
The tool has a `RequireWorkflow` predicate — it only appears in
workflow channels.
## Assignments
Human review queue for workflow stages that have an `assignment_team_id`.
### List My Assignments
```
GET /workflow-assignments/mine
```
Returns assignments where `assigned_to` is the current user or
`status=unassigned` for the user's teams.
**Auth:** Authenticated.
### Claim Assignment
```
POST /workflow-assignments/:id/claim
```
Sets `assigned_to` to the current user, status to `claimed`.
Returns 409 if the assignment is already claimed or not found.
**Auth:** Authenticated.
### Complete Assignment
```
POST /workflow-assignments/:id/complete
```
Marks assignment as `completed`. Does NOT auto-advance the stage —
the human reviews and manually advances or the persona tool does it.
Returns 409 if the assignment is not in `claimed` state.
**Auth:** Authenticated.
### List Team Assignments
```
GET /teams/:teamId/assignments?status=unassigned
```
Assignments for the team, filtered by `status` (default: `unassigned`).
Valid values: `unassigned`, `claimed`, `completed`.
**Auth:** Authenticated (team member).
### Get Assignment (v0.35.0)
```
GET /workflow-assignments/:id
```
Returns the full assignment including review comments.
**Auth:** Authenticated.
### Add Review Comment (v0.35.0)
```
POST /workflow-assignments/:id/comment
```
```json
{ "text": "Looks good, approved with minor note about address." }
```
Appends a comment to the assignment's `review_comments` array.
Comments are visible to the next stage when `history_mode=full`.
**Auth:** Authenticated.
### Assignment Object
```json
{
"id": "uuid",
"channel_id": "uuid",
"stage": 1,
"team_id": "uuid",
"assigned_to": "uuid|null",
"status": "unassigned|claimed|completed",
"review_comments": [
{ "text": "...", "user_id": "uuid", "created_at": "..." }
],
"created_at": "...",
"claimed_at": "...|null",
"completed_at": "...|null"
}
```
## Visitor Entry
Public-facing workflow entry for unauthenticated visitors.
### Landing Page
```
GET /w/:id
GET /w/:id/:slug
```
Server-rendered branded page. Shows workflow name, tagline, branding
colors, and a "Start" button. If the visitor has an existing session
cookie for this workflow, resumes the conversation.
Not an API endpoint — returns HTML.
### Start Visitor Session
```
POST /api/v1/workflow-entry/:scope/:slug
```
No request body. Display name is auto-generated (`Visitor`,
`Visitor 2`, etc.).
`:scope` is a team ID or `global`. Creates a session participant,
creates a workflow channel, sets `allow_anonymous=true`. Returns:
```json
{
"channel_id": "uuid",
"session_id": "uuid",
"redirect_to": "/w/channel-uuid"
}
```
The session token is set as a cookie (`sb_session`) for subsequent
requests. `redirect_to` is the workflow chat surface URL.
### Visitor Message/Completion
Visitors interact through the workflow channel using session-scoped routes:
```
POST /api/v1/w/:id/messages
GET /api/v1/w/:id/messages
POST /api/v1/w/:id/completions
```
These use `AuthOrSession` middleware — the session cookie identifies
the visitor without requiring a JWT.
## Form Submission (v0.29.3)
For stages with `stage_mode` of `form_only` or `form_chat`, visitors
submit structured form data through dedicated endpoints.
### Get Form Template
```
GET /api/v1/w/:id/form
```
Returns the current stage's typed form template and submission status.
```json
{
"stage_mode": "form_only",
"stage_name": "Contact Info",
"form_template": { "fields": [...] },
"submitted": false
}
```
**Auth:** `AuthOrSession` (JWT or session cookie).
### Submit Form
```
POST /api/v1/w/:id/form-submit
```
```json
{
"name": "Jane Doe",
"email": "jane@example.com",
"dept": "eng"
}
```
Validates data against the typed form template. Returns field-level
errors on validation failure:
```json
{
"error": "validation failed",
"errors": [
{ "key": "email", "message": "invalid email format" },
{ "key": "name", "message": "required field missing" }
]
}
```
On success, merges data into `stage_data`. For `form_only` stages with
`auto_transition=true`, auto-advances to the next stage.
Emits `workflow.form_submitted` WebSocket event on success.
**Auth:** `AuthOrSession` (JWT or session cookie).
## Staleness Sweep
Background goroutine. Checks active workflow instances for staleness
(no activity within `WORKFLOW_STALE_HOURS`, default 72). Stale instances
are marked `workflow_status=stale`. Retention policy then applies:
`archive` keeps the channel, `delete` removes it after `delete_after_days`.
## On-Complete Chaining
When a workflow completes and has `on_complete` configured:
```json
{
"on_complete": {
"action": "start_workflow",
"target_slug": "follow-up-survey",
"data_map": { "name": "customer_name", "case_id": "case_id" }
}
}
```
`action` must be `start_workflow`. `target_slug` identifies the workflow
to chain into (looked up in the same team scope). `data_map` is a flat
key remapping from source stage data keys to target initial data keys.
If `data_map` is empty/omitted, all stage data is passed through as-is.
The system starts a new instance of the target workflow, passing mapped
stage data as initial context. Webhook delivery (if configured) fires
before chaining.
## Conditional Routing (v0.35)
The routing engine evaluates `transition_rules.conditions` on the
departing stage against accumulated `stage_data`. Conditions are
evaluated in order; first match wins.
### Condition Object
```json
{
"conditions": [
{ "field": "category", "op": "eq", "value": "billing", "target_stage": "Billing Review" },
{ "field": "priority", "op": "gte", "value": 5, "target_stage": "Urgent" }
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `field` | string | Key in `stage_data` to evaluate |
| `op` | string | Operator (see below) |
| `value` | any | Value to compare against |
| `target_stage` | string | Stage name or ordinal to route to |
**Operators:** `eq`, `neq`, `gt`, `lt`, `gte`, `lte`, `in`, `contains`, `exists`, `not_exists`.
`target_stage` resolves by name (case-insensitive) first, then as
a numeric ordinal. Backward jumps are allowed (loop stages).
If no condition matches, falls back to `ordinal + 1`.
### workflow_route Tool (v0.35.0)
Persona-callable tool for AI-triggered routing. Available only in
workflow channels (`RequireWorkflow` predicate).
```json
{
"name": "workflow_route",
"parameters": {
"target_stage": "Stage name to route to",
"reason": "Explanation for the routing decision"
}
}
```
Resolves `target_stage` by name. Records the routing decision in
`stage_data._route_history_latest` with `from`, `to`, `reason`, `ts`.
### Starlark workflow.route()
```python
workflow.route(channel_id, target_stage, reason)
```
Extension-callable routing for programmatic escalation patterns.
Requires `workflow.access` permission.
## on_advance Hook (v0.35.0)
Starlark entry point that fires synchronously after a stage transition.
Configured per-stage in `transition_rules`:
```json
{
"on_advance": {
"package_id": "uuid",
"entry_point": "on_advance"
}
}
```
The hook receives a context dict:
```python
def on_advance(ctx):
# ctx["channel_id"] - workflow channel ID
# ctx["previous_stage"] - ordinal of the departed stage
# ctx["current_stage"] - ordinal of the new stage
# ctx["stage_data"] - accumulated stage data dict
return {"stage_data": ctx["stage_data"]} # enriched
# return None — no change
# return {"error": "msg"} — reject (logged, not rolled back)
```
Use cases: external API enrichment via `http.fetch`, cross-field
validation, data transformation between stages.
## Monitoring Dashboard (v0.35.0)
Admin endpoints for tracking active workflow instances and SLA status.
### List Active Instances
```
GET /admin/workflows/monitor/instances
GET /teams/:teamId/workflows/monitor/instances
```
Returns all active (`workflow_status=active`) instances with SLA info.
```json
{
"data": [{
"channel_id": "uuid",
"channel_title": "...",
"workflow_id": "uuid",
"workflow_name": "...",
"current_stage": 1,
"stage_name": "Review",
"age_seconds": 3600,
"stage_age_seconds": 1200,
"sla_seconds": 7200,
"sla_remaining_seconds": 6000,
"sla_breached": false,
"last_activity_at": "..."
}]
}
```
SLA is computed as `stage_entered_at + sla_seconds` vs current time.
**Auth:** Admin (global) or team member (team-scoped).
### Stage Funnel
```
GET /admin/workflows/monitor/funnel/:id
```
Returns per-stage instance counts for bottleneck detection.
```json
{
"data": [
{ "stage_ordinal": 0, "stage_name": "Intake", "count": 5 },
{ "stage_ordinal": 1, "stage_name": "Review", "count": 12 },
{ "stage_ordinal": 2, "stage_name": "Complete", "count": 2 }
]
}
```
**Auth:** Admin.
### Stale Instances
```
GET /admin/workflows/monitor/stale?threshold_hours=48
```
Returns instances where `last_activity_at` exceeds the threshold.
Default threshold: 48 hours.
**Auth:** Admin.
## WebSocket Events
| Event | When |
|-------|------|
| `workflow.advanced` | Stage transition (includes new stage info) |
| `workflow.completed` | Workflow finished |
| `workflow.assigned` | New assignment created |
| `workflow.claimed` | Assignment claimed by user |
| `workflow.form_submitted` | Form data submitted for a form stage |

449
docs/ICD/workspaces.md
View File

@@ -1,449 +0,0 @@
# Workspaces & Git
The **Workspace** is a virtual filesystem for the editor surface.
Each workspace has a root directory, file operations, optional Git
integration, and full-text indexing for code search.
**All endpoints require:** `Auth()` middleware (JWT bearer token).
Workspace-scoped endpoints additionally verify ownership via
`loadAndAuthorize` (user owns workspace, or is member of the
owning team/project/channel).
---
### Workspace CRUD
```
GET /workspaces → { "data": [...] }
POST /workspaces ← { "name", "owner_type", "owner_id" }
GET /workspaces/default → workspace object (v0.37.18)
GET /workspaces/:id → workspace object
PATCH /workspaces/:id ← partial update
DELETE /workspaces/:id
```
**Auth:** Authenticated user. `GET /workspaces` returns user-owned
workspaces plus those owned by the user's teams. All `:id` endpoints
verify the caller can access the workspace's owner entity.
#### `GET /workspaces/default` (v0.37.18)
Returns the current user's personal workspace, creating it on first
access. The workspace is named "My Files" and owned by the current user.
Idempotent — safe to call on every page load.
**Response:** 200 with workspace object (same shape as `GET /workspaces/:id`).
Includes `file_count` and `total_bytes` stats.
Workspace object:
```json
{
"id": "uuid",
"name": "my-project",
"owner_type": "user",
"owner_id": "uuid",
"status": "active",
"max_bytes": 52428800,
"indexing_enabled": false,
"git_remote_url": "https://github.com/...",
"git_branch": "main",
"git_credential_id": "uuid",
"git_last_sync": "2025-06-15T14:30:00Z",
"file_count": 42,
"total_bytes": 1048576,
"created_at": "2025-06-15T14:30:00Z",
"updated_at": "2025-06-15T14:30:00Z"
}
```
| Field | Type | Notes |
|-------|------|-------|
| `id` | string | UUIDv4 |
| `name` | string | Max 200 chars |
| `owner_type` | `"user"` \| `"project"` \| `"channel"` \| `"team"` | Owner entity type |
| `owner_id` | string | UUID of owner entity |
| `status` | `"active"` \| `"archived"` \| `"deleting"` | Lifecycle state |
| `max_bytes` | int \| null | Quota ceiling. Omitted if unset. |
| `indexing_enabled` | bool | Full-text search indexing |
| `git_remote_url` | string \| null | Git remote. Omitted if unset. |
| `git_branch` | string \| null | Tracked branch. Omitted if unset. |
| `git_credential_id` | string \| null | FK to git credential. Omitted if unset. |
| `git_last_sync` | string \| null | ISO 8601. Omitted if never synced. |
| `file_count` | int | Computed. Enriched on `GET /:id` only. |
| `total_bytes` | int | Computed. Enriched on `GET /:id` only. |
| `created_at` | string | ISO 8601 |
| `updated_at` | string | ISO 8601 |
> **Note:** `root_path` is a server-internal filesystem path and is
> never exposed in API responses (json tag `"-"`).
**Create request:**
```json
{
"name": "my-project",
"owner_type": "user",
"owner_id": "uuid",
"max_bytes": 52428800
}
```
Returns `201` with the workspace object.
**PATCH request** (all fields optional):
```json
{
"name": "new-name",
"status": "archived",
"max_bytes": 104857600,
"indexing_enabled": true,
"git_remote_url": "https://...",
"git_branch": "develop",
"git_credential_id": "uuid"
}
```
Returns updated workspace object.
**DELETE** returns `{"ok": true}`. Deletion is async (status set to
`"deleting"`, filesystem cleanup runs in background).
---
### File Operations
All paths are relative to the workspace root. File path is passed
as `?path=` query parameter.
```
GET /workspaces/:id/files?path=/ → { "data": [entries] }
GET /workspaces/:id/files/read?path=/a.md → raw file content (binary stream)
PUT /workspaces/:id/files/write?path=/a.md ← raw body (Content-Length required)
DELETE /workspaces/:id/files/delete?path=/a.md
POST /workspaces/:id/files/mkdir?path=/src
```
**Auth:** Authenticated user with workspace access.
**`GET .../files`** accepts `?recursive=true` for deep listing.
Returns `{"data": [...]}` (never bare array). Nil slice guarded.
File entry (from `WorkspaceFile` model):
```json
{
"id": "uuid",
"workspace_id": "uuid",
"path": "/src/main.go",
"is_directory": false,
"content_type": "text/x-go",
"size_bytes": 4096,
"sha256": "abc123...",
"index_status": "ready",
"chunk_count": 3,
"created_at": "2025-06-15T14:30:00Z",
"updated_at": "2025-06-15T14:30:00Z"
}
```
**`GET .../files/read`** returns raw file content with appropriate
`Content-Type` header and `Content-Length`. This is a binary stream,
not JSON.
**`PUT .../files/write`** reads the request body as raw content.
Returns `{"ok": true, "path": "/a.md"}`. Returns `413` if the
write would exceed the workspace quota.
**`DELETE .../files/delete`** accepts `?recursive=true` for
directory removal. Returns `{"ok": true}`.
**`POST .../files/mkdir`** returns `201 {"ok": true, "path": "/src"}`.
---
### Archive
**Download** (full workspace as archive):
```
GET /workspaces/:id/archive/download?format=zip
```
**Auth:** Authenticated user with workspace access.
`format` query param: `zip` (default), `tar.gz`, `tgz`.
Returns binary stream with `Content-Disposition: attachment`.
**Upload** (restore from archive):
```
POST /workspaces/:id/archive/upload
Content-Type: multipart/form-data (field: "file")
```
**Auth:** Authenticated user with workspace access.
Accepts `.zip`, `.tar.gz`, `.tgz` archives. Format detected from
filename extension.
Returns:
```json
{
"ok": true,
"files_extracted": 42
}
```
---
### Reconcile, Stats, Index Status
**Reconcile** (sync filesystem state with DB metadata):
```
POST /workspaces/:id/reconcile
```
**Auth:** Authenticated user with workspace access.
Returns:
```json
{
"added": 5,
"removed": 2,
"updated": 3
}
```
**Stats:**
```
GET /workspaces/:id/stats
```
**Auth:** Authenticated user with workspace access.
Returns a `WorkspaceStats` object:
```json
{
"workspace_id": "uuid",
"file_count": 42,
"dir_count": 8,
"total_bytes": 1048576,
"max_bytes": 52428800
}
```
**Index status:**
```
GET /workspaces/:id/index-status
```
**Auth:** Authenticated user with workspace access.
Returns indexing progress for full-text search:
```json
{
"workspace_id": "uuid",
"indexing_enabled": true,
"status_counts": {
"pending": 3,
"indexing": 1,
"ready": 38,
"error": 0,
"skipped": 2
},
"total_chunks": 156
}
```
---
### Git Integration
All Git operations are workspace-scoped. Require the workspace to
have a Git remote configured.
```
POST /workspaces/:id/git/clone ← { "url", "branch", "credential_id" }
POST /workspaces/:id/git/pull
POST /workspaces/:id/git/push
GET /workspaces/:id/git/status → GitStatus object
GET /workspaces/:id/git/diff → { "diff": "..." }
POST /workspaces/:id/git/commit ← { "message", "paths": [...] }
GET /workspaces/:id/git/log → { "data": [GitLogEntry] }
GET /workspaces/:id/git/branches → { "branches": [...], "current": "main" }
POST /workspaces/:id/git/checkout ← { "branch": "feature-x" }
```
**Auth:** Authenticated user with workspace access.
**`POST .../git/clone`** request:
```json
{
"url": "https://github.com/user/repo.git",
"branch": "main",
"credential_id": "uuid"
}
```
`branch` and `credential_id` are optional. Returns `{"status": "cloned"}`.
**`POST .../git/pull`** returns `{"status": "pulled"}`.
**`POST .../git/push`** returns `{"status": "pushed"}`.
**`GET .../git/status`** returns the full `GitStatus` object:
```json
{
"branch": "main",
"clean": false,
"ahead": 2,
"behind": 0,
"staged": [{ "path": "file.go", "status": "M" }],
"modified": [{ "path": "other.go", "status": "M" }],
"untracked": ["new-file.txt"]
}
```
**`GET .../git/diff`** accepts optional `?path=file.go` for single-file diff.
Returns `{"diff": "..."}` (unified diff string).
**`POST .../git/commit`** request:
```json
{
"message": "fix: resolve null pointer",
"paths": ["/src/main.go", "/src/util.go"]
}
```
`paths` is optional (commits all staged if omitted). Returns
`{"status": "committed"}`.
**`GET .../git/log`** accepts optional `?n=20` (default 20, max 100).
Returns `{"data": [GitLogEntry]}`:
```json
{
"data": [
{
"hash": "abc123...",
"short_hash": "abc123",
"author": "Jane Doe",
"date": "2025-06-15T14:30:00Z",
"message": "Initial commit"
}
]
}
```
**`GET .../git/branches`** returns:
```json
{
"current": "main",
"branches": ["main", "develop", "feature-x"]
}
```
**`POST .../git/checkout`** returns
`{"status": "checked out", "branch": "feature-x"}`.
---
### Git Credentials
User-owned, encrypted credential storage for Git operations.
Encrypted with the user's UEK via the platform vault (same pattern
as BYOK keys — admins cannot recover).
Independent of workspace scope — credentials are user-level.
```
POST /git-credentials ← { "name", "auth_type", ... }
POST /git-credentials/generate ← { "name", "persona_id?" }
GET /git-credentials → { "data": [GitCredentialSummary] }
GET /git-credentials/:id/public-key → { "public_key", "fingerprint" }
DELETE /git-credentials/:id
```
**Auth:** Authenticated user. Credentials are scoped to the requesting user.
**Create request (manual):**
`auth_type` determines which fields are required:
| `auth_type` | Required fields |
|-------------|----------------|
| `https_pat` | `token` |
| `https_basic` | `username`, `password` |
| `ssh_key` | `private_key` (+ optional `passphrase`) |
Returns `201` with the credential summary (never exposes encrypted data).
**Generate request (v0.28.6 — server-side SSH keygen):**
```json
{
"name": "GitHub - main",
"persona_id": "uuid"
}
```
Generates an ED25519 SSH keypair server-side. The private key is
vault-encrypted before storage and never leaves the server. Returns
the credential summary including the public key (authorized_keys
format) and SHA256 fingerprint. The user adds the public key to their
git host (GitHub, Gitea, etc.).
Optional `persona_id` binds the key to a persona for commit
attribution — git operations using this credential will set the
commit author to the persona's name and handle.
Returns `201`.
**Get Public Key:**
```
GET /git-credentials/:id/public-key
```
Returns `{ "public_key": "ssh-ed25519 AAAA...", "fingerprint": "SHA256:..." }`.
Owner-only — returns `404` for other users or credentials without
a public key (e.g. PAT credentials).
**List response:**
```json
{
"data": [
{
"id": "uuid",
"name": "GitHub PAT",
"auth_type": "https_pat",
"public_key": "",
"fingerprint": "",
"persona_id": null,
"created_at": "2025-06-15T14:30:00Z"
}
]
}
```
`public_key` and `fingerprint` are populated for `ssh_key` type
credentials created via `/generate`. Empty for PAT/basic auth.
**DELETE** returns `{"deleted": true}`. Returns `404` if credential
not found or not owned by the requesting user.
---

275
docs/PACKAGES.md
View File

@@ -1,275 +0,0 @@
# Package Format & Manifest Reference
> v0.30.2
A `.pkg` file is a ZIP archive containing a `manifest.json` and optional
assets. Packages extend Chat Switchboard with surfaces (routable UIs),
extensions (server-side hooks/tools), or workflows.
## Archive Structure
```
manifest.json required — package metadata and configuration
js/ optional — JavaScript assets (served at /surfaces/:id/js/)
main.js entry point (loaded by surface template)
css/ optional — stylesheets
assets/ optional — images, icons, etc.
script.star optional — Starlark script (extension/sidecar tiers)
migrations/ optional — Starlark migration scripts (keyed by version)
1.star
2.star
```
## Building & Installing
```bash
# Build from packages/ directory
bash packages/build.sh my-package # one package
bash packages/build.sh # all packages
# Install via API
curl -X POST http://localhost:3000/api/v1/admin/packages/install \
-H "Authorization: Bearer $TOKEN" \
-F file=@dist/my-package.pkg
# Install via admin UI
# Admin → System → Packages → Upload
```
## Manifest Schema
### Required Fields
| Field | Type | Description |
|-------|------|-------------|
| `id` | `string` | Unique package identifier (kebab-case). Used as directory name and DB key. |
| `title` | `string` | Human-readable display name. |
### Optional Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `type` | `string` | `"surface"` | Package type (see below). |
| `version` | `string` | `""` | Semver version string. |
| `description` | `string` | `""` | Short description shown in admin UI. |
| `author` | `string` | `""` | Author name or org. |
| `tier` | `string` | `"browser"` | Execution tier: `browser`, `starlark`, `sidecar`. |
| `route` | `string` | — | URL path for surface routing (e.g. `/s/my-surface`). |
| `auth` | `string` | `"authenticated"` | Auth requirement: `authenticated` or `public`. |
| `layout` | `string` | `"single"` | Page layout template. |
| `permissions` | `string[]` | `[]` | Required permissions (see Permissions below). |
| `tools` | `object[]` | — | Server-side tool declarations for AI tool-use. |
| `hooks` | `string[]` | — | Lifecycle hooks: `surface`, `on_install`, `on_uninstall`. |
| `settings` | `object` | — | Settings schema for admin-configurable options. |
| `schema_version` | `integer` | `0` | Current data schema version (for migrations). |
| `migrations` | `object` | — | Starlark migration scripts keyed by target version. |
| `network_access` | `string[]` | — | Allowed external hostnames (sidecar tier). |
| `api_routes` | `object[]` | — | Custom HTTP routes handled by Starlark `on_request`. |
| `db_tables` | `object[]` | — | Extension-owned database tables. |
| `pipes` | `object` | — | Filter pipeline registrations. |
| `components` | `string[]` | — | UI component identifiers. |
| `workflow_definition` | `object` | — | Embedded workflow definition (type `"workflow"` only). |
## Package Types
### `surface` (default)
A routable page served at the `route` path. The browser loads
`js/main.js` and renders into `#extension-mount`.
```json
{
"id": "my-dashboard",
"title": "My Dashboard",
"type": "surface",
"route": "/s/my-dashboard",
"auth": "authenticated",
"layout": "single",
"version": "1.0.0"
}
```
### `extension`
Server-side logic without a routable UI. Runs Starlark code with
granted permissions.
```json
{
"id": "auto-tagger",
"title": "Auto Tagger",
"type": "extension",
"tier": "starlark",
"version": "1.0.0",
"permissions": ["filters.pre_completion"],
"tools": [
{
"name": "tag_message",
"description": "Auto-tag messages based on content"
}
]
}
```
### `full`
Both a surface and an extension. Has a routable page and server-side
hooks/tools.
```json
{
"id": "analytics",
"title": "Analytics Suite",
"type": "full",
"tier": "starlark",
"route": "/s/analytics",
"version": "1.0.0",
"permissions": ["db.read", "db.write"],
"db_tables": [
{ "name": "events", "columns": { "ts": "text", "event": "text", "data": "text" } }
]
}
```
### `workflow`
A packaged workflow definition with optional surfaces and handlers.
See [WORKFLOW-PACKAGES.md](WORKFLOW-PACKAGES.md) for details.
```json
{
"id": "onboarding-flow",
"title": "Employee Onboarding",
"type": "workflow",
"version": "1.0.0",
"workflow_definition": {
"name": "Employee Onboarding",
"slug": "employee-onboarding",
"entry_mode": "public_link",
"stages": [...]
}
}
```
## Permissions
Declared in `manifest.permissions`. Admins grant/revoke per-package in
the admin UI.
| Permission | Description |
|-----------|-------------|
| `secrets.read` | Read secrets from the vault |
| `notifications.send` | Send push notifications |
| `filters.pre_completion` | Intercept messages before AI completion |
| `db.read` | Read from extension-owned tables |
| `db.write` | Write to extension-owned tables |
| `api.http` | Make outbound HTTP requests (sidecar tier) |
| `provider.complete` | Call LLM completion APIs |
| `forms.validate` | Validate workflow form submissions |
| `workflow.access` | Access workflow definitions and stage data |
## Settings Schema
Packages can declare admin-configurable settings. The schema drives a
form in the admin UI under System → Packages → Settings.
```json
{
"settings": {
"api_key": {
"type": "string",
"label": "API Key",
"description": "External service API key",
"required": true
},
"max_results": {
"type": "number",
"label": "Max Results",
"default": 10
},
"enabled": {
"type": "boolean",
"label": "Enable Feature",
"default": true
}
}
}
```
Settings are stored in the `package_settings` column and accessible from
Starlark via `settings.get("key")`.
## Data Migrations
Packages with `db_tables` can version their schema using Starlark
migration scripts. The engine runs migrations sequentially on
install/upgrade and rejects downgrades.
```json
{
"schema_version": 2,
"migrations": {
"1": "def migrate(db):\n db.query('CREATE TABLE ...')\n",
"2": "def migrate(db):\n db.query('ALTER TABLE ...')\n"
}
}
```
Or reference files in the archive:
```
migrations/
1.star # def migrate(db): ...
2.star # def migrate(db): ...
```
## Extension Database Tables
Tables are namespaced to `ext_{pkg_id}_{table}` to prevent collisions.
Platform views provide read-only access to core data:
| View | Columns |
|------|---------|
| `ext_view_users` | `id`, `display_name`, `email` |
| `ext_view_channels` | `id`, `title`, `type`, `team_id` |
The `ext_data_tables` catalog tracks all extension-owned tables for
install/uninstall lifecycle management.
## Server-Side Tools
Extensions can declare tools for AI tool-use. The `on_tool_call` entry
point in the Starlark script handles dispatching.
```json
{
"tools": [
{
"name": "lookup_customer",
"description": "Look up customer by email",
"parameters": {
"email": { "type": "string", "required": true }
}
}
]
}
```
```python
# script.star
def on_tool_call(tool_name, params):
if tool_name == "lookup_customer":
result = http.get("https://api.example.com/customers?email=" + params["email"])
return {"name": result["name"], "plan": result["plan"]}
```
## User-Installable Packages
Packages can be scoped to individual users or teams (v0.30.0). Only
`browser`-tier packages support user installation. Scope options:
| Scope | Visibility |
|-------|-----------|
| `global` | All users (admin-installed only) |
| `team` | Team members only |
| `personal` | Installing user only |

433
docs/ROADMAP.md
View File

@@ -1,433 +0,0 @@
# Roadmap — Chat Switchboard
**See also:**
- [ARCHITECTURE.md](ARCHITECTURE.md) — Core services design, store layer, scope model
- [EXTENSIONS.md](EXTENSIONS.md) — Extension system spec (Browser/Starlark/Sidecar tiers,
manifests, browser tool bridge, surfaces/modes, model roles)
- [EXTENSION-SURFACES.md](EXTENSION-SURFACES.md) — Extension surface authoring guide
(manifest format, platform API, CSS properties, install workflow)
- [CHANGELOG.md](../CHANGELOG.md) — Detailed release notes for all completed versions
**Versioning (pre-1.0):** `0.<major>.<minor>` — hotfixes use quad: `0.x.y.z`
No compatibility guarantees before 1.0.
---
## Dependency Graph
Three parallel implementation tracks converge at v0.50.0 (MVP). The
extension track (v0.38.x) builds the package ecosystem that dynamic
workflows depend on. The workflow track (v0.39.x) builds both simple
and dynamic workflow capabilities on that foundation. The operations
track builds production readiness. A polish series (v0.40.x) follows
to stabilize for MVP.
```
v0.9.x–v0.28.7 Foundation through Platform Polish ✅
(see CHANGELOG.md for full history)
│
v0.28.8 ICD Green Board ✅
│
┌───────────────┼───────────────┐
│ │ │
Extension Track UI Rewrite Operations Track
│ Track │
v0.29.0 ✅ │ v0.32.0 ✅
v0.29.1 ✅ │ v0.33.0 ✅
v0.29.2 ✅ │ v0.34.0 ✅
v0.29.3 ✅ │ │
v0.30.0 ✅ │ │
v0.30.1 ✅ │ │
v0.30.2 ✅ │ │
v0.31.0 ✅ │ │
v0.31.1 ✅ │ │
v0.31.2 ✅ │ │
│ │ v0.35.0 Workflow Product
│ │ │
│ │ v0.36.0 Full OpenAPI ✅
│ │ │
│ v0.37.1 Perm Audit ✅ │
│ v0.37.2 Primitives ✅ │
│ v0.37.3 SDK ✅ │
│ v0.37.4 Shell ✅ │
│ v0.37.5–16 Surfaces ✅ │
│ v0.37.17–18 Surfaces ✅ │
│ v0.37.19 Tag ✅ │
│ │ │
v0.38.0–.5 ✅ │ │
Extension │ │
Continuation │ │
│ │ │
├────────────┤ │
│ │ │
│ v0.39.x Workflow │
│ Product Maturity │
│ (simple + dynamic) │
│ │ │
│ v0.40.x Polish │
│ (bug hunt, dead │
│ code, stabilize) │
│ │ │
══════╪════════════╪═════════════════╪══════
│ MVP v0.50.0 │
│ (all tracks complete │
│ + mobile + deploy docs) │
══════╪════════════╪═════════════════╪══════
│
v0.50.0+ Rich Media + Beyond
(image gen, code sandbox,
STT/TTS, desktop app)
```
## Completed Versions (see CHANGELOG.md for details)
| Version | Summary |
|---------|---------|
| v0.28.0–v0.28.8 | Platform polish, ICD green board, security, infrastructure |
| v0.29.0–v0.29.3 | Extension track: Starlark sandbox, API extensions, DB extensions, workflow forms |
| v0.30.0–v0.30.2 | Package lifecycle, SDK adoption, workflow packages |
| v0.31.0–v0.31.2 | Editor package, SDK composability, team workflow self-service |
| v0.32.0 | Multi-replica HA (PG SKIP LOCKED, cross-pod WS, shared tickets/rate limits) |
| v0.33.0 | Observability (slog, Prometheus, Grafana, OpenAPI, admin dashboard) |
| v0.34.0 | Data portability (export/import, GDPR, backup/restore) |
| v0.35.0 | Workflow product (forms, data pipeline, conditional routing, structured review) |
| v0.36.0 | Full OpenAPI spec (20 ICD domains, auth docs, WebSocket schemas) |
---
## UI Rewrite Track ✅
Scorched earth rebuild of the frontend on Preact+htm. Three-layer
architecture (Primitives → SDK → Shell) replacing the imperative
DOM manipulation codebase. See `UI redesign.md` for full design doc.
**Complete as of v0.37.19.**
Depends on: v0.36.0 (OpenAPI spec), v0.31.2 (SDK composability).
### v0.37.1 — Permission Audit & Enforcement ✅
Backend prerequisite for frontend RBAC. See `CHANGESET.md`.
- [x] `RequirePermission` middleware on 8 unprotected routes
- [x] `GET /profile/permissions` — self-service resolved permission set
- [x] 12 permission enforcement integration tests (deny → grant → allow)
- [x] Mirror route fixes in integration test harness
### v0.37.2 — Layer 0: UI Primitives + Shell Layout ✅
Preact+htm component library and application frame.
- [x] Vendor: preact.module.js, hooks.module.js, htm.module.js (~5KB total)
- [x] 14 primitive components: Button, Spinner, Avatar, FormField,
Tooltip, Banner, Toast, Dialog, Confirm, Prompt, Menu, Drawer,
Tabs, Dropdown
- [x] AppShell layout: fixed top/bottom banners (single config),
dismissible message bar, footer, surface viewport
- [x] `sw-primitives.css` + `sw-shell.css` — `sw-` prefixed, no
conflicts with existing styles
- [x] `dev.html` — interactive primitives gallery + shell controls
- [x] ES module loading with `window.preact/hooks/html` globals
### v0.37.3 — Layer 1: SDK ✅
Namespaced REST client, auth module, RBAC gate, event bus, pipe/filter
pipeline, theme control.
Depends on: v0.37.2.
- [x] `sw.api.{domain}.*` — 18 namespaced domain clients (auth,
channels, personas, knowledge, notes, projects, workspaces,
memory, models, providers, notifications, extensions, profile,
teams, workflows, tasks, surfaces, admin)
- [x] `sw.auth.*` — login, logout, refresh, session lifecycle,
permissions/teams/policies cache
- [x] `sw.can(perm)` — synchronous RBAC gate from cached permission set
- [x] `sw.on/off/emit` — event bus with wildcard patterns, WebSocket
connection with ticket-first auth
- [x] `sw.pipe.*` — carry forward pre-send/stream/render pipeline
- [x] `sw.theme.*` — theme control with FOUC prevention
### v0.37.4 — Layer 2: Shell ✅
Application frame with user menu, surface viewport, banners, toast/dialog
stacks. Temp auth bypass for visual validation.
Depends on: v0.37.3.
- [x] Root `<App>` component tree (AppShell + SurfaceViewport +
ToastContainer + DialogStack)
- [x] `UserMenu` — avatar + flyout (Settings, Admin, Team Admin,
Debug, Sign Out) with RBAC gating
- [x] `SurfaceViewport` — mount point for active surface
- [x] `ToastContainer` + `DialogStack` — global overlay management
- [x] `sw.userMenu(container, opts)` — imperative mount helper for
non-Preact surfaces
### v0.37.5 — Login + Settings Surfaces ✅
First surface rebuilds. Login page and Settings surface converted from
Go templates + imperative DOM JS to Preact component trees.
Depends on: v0.37.4.
- [x] Login surface — 5 components (Hero, LoginForm, RegisterForm,
SSOPanel, LoginSurface root)
- [x] Settings surface — framework with nav + section routing
- [x] 7 Preact sections: General, Appearance, Profile, Models,
Teams, Providers (stub), Personas (stub)
- [x] Bridge pattern for deferred sections (Tasks, Git Keys,
Workflows, Data & Privacy)
- [x] `sw.toast()` SDK wiring, nav gating (BYOK + Personas)
- [x] `sw-login.css` extracted from inline styles
**Known issues (tracked in CHANGELOG, resolve before v0.37.# tag):**
- BYOK sections not tested with policy enabled
- ~~Bridge sections~~ Resolved in v0.37.7 (all settings sections native Preact)
- ~~`ui-settings.js` + `settings-handlers.js`~~ Confirmed deleted, stale `sw.js`
references cleaned in v0.37.12
### v0.37.6–v0.37.18 — Remaining Surface Rebuilds
Each surface rebuilt as Preact component tree. Old JS deleted per surface.
| Version | Surface | Notes |
|---------|---------|-------|
| v0.37.6 | Admin | Largest section count, no real-time |
| v0.37.7 | Team Admin | Inherits from Admin patterns |
| v0.37.8 | ChatPane ✅ | Composable kit (9 files), standalone stripped from old chat-pane.js |
| v0.37.9 | NotesPane ✅ | Obsidian-grade kit (10 files), old notes code scorched (~1,630 lines) |
| v0.37.10 | Chat surface ✅ | Scorched earth: 21 files deleted (−12,841 net), Preact surface + ChatPane kit upgrades |
| v0.37.11 | Notes surface ✅ | Preact surface (7 JS + 1 CSS), sidebar folders/tags, UserMenu, template rewrite, Menu primitive fix |
| v0.37.12 | Scorched Earth II ✅ | 23 files deleted (~2,600 lines): 9 JS, 6 CSS, 8 admin templates; sw.js + base.html cleaned |
| v0.37.13 | Scorched Earth III ✅ | 5 JS deleted (−2,269 lines): ui-primitives, code-editor, file-tree, user-menu, app-state; SDK slimmed, Theme→Preact |
| v0.37.14 | SE IV + Pane audit ✅ | 4 JS deleted (~−1,672 lines); double-unwrap cleanup (93 sites); API envelope normalization (18 Go handlers); thinking tags persistence; deleteMessage endpoint; extension surface user menu fix; 7 bug fixes |
| v0.37.15 | Workflow surfaces ✅ | Workflow ownership/lifecycle, stage CRUD, team-admin tabs, assignments queue |
| v0.37.16 | Projects surface ✅ | Card grid + detail, inline ChatPane, context menu, sidebar pickers, deep-link |
| v0.37.17 | Workspaces ✅ | Workspace-first project file storage, tree browser UI, archive upload/download |
| v0.37.18 | Debug / model surface ✅ | Debug modal Preact rebuild (CR P2-5), default workspace, tool_output auto-save, save-to-workspace bridge, model/provider polish |
| v0.37.19 | Tag ✅ | `sw.can()` RBAC gates (CR P2-1), `__USER__`/`__PAGE_DATA__` removal (CR P2-4), `_getScale` → `sw.shell.getScale()` (CR P3-3), light mode CSS audit, dead code sweep |
**v0.37.14 — Scorched Earth IV + Pane Audit ✅** (10 sessions, v0.37.14.0–.22):
Scorched Earth IV (4 JS deleted, ~−1,672 lines), unified sidebar with nested
folders + DnD, notification bell, @mentions, typing indicators, member panel,
double-unwrap cleanup (93 sites), API envelope normalization (18 Go handlers),
thinking tags persistence, deleteMessage endpoint, 6 bug fixes.
Cumulative scorched earth: 53 files, ~−19,382 lines.
**v0.37.15 — Workflow Ownership & Lifecycle:**
Team-admin workflow management as first-class path. See
[DESIGN-0.37.15](Workflow/DESIGN-0.37.15.md) for full design.
Instance cancel/unclaim/reassign, stage CRUD FE wiring, team-admin
workflow queue + stage editor + instance monitor surfaces.
**v0.37.16 — Projects Surface ✅:**
Card-grid list + two-column detail (inline ChatPane, context menu, sidebar
pickers, star toggle, deep-linking). UserMenu in header. Backend was already
complete (16+2 endpoints, 18 SDK methods). File upload UI wired but backend
storage deferred to v0.37.17.
**v0.37.17 — Workspaces (File Storage):**
Workspace-first file management for projects. Architecture: `files` table +
objStore = ephemeral message attachments; Workspace FS = persistent file
collections (tree, dirs, archive, quota, indexing). Bridge ("Save to workspace")
deferred to v0.37.18.
Schema and model FK already exist (`workspaces` table, `Project.WorkspaceID`).
Workspace backend fully implemented (FS layer, handlers, store, SDK). This
version bridges projects to workspaces and builds the file browser UI.
Git integration (clone, pull, branch tracking) is optional/post-MVP.
- [x] Auto-create workspace for project on first file upload
- [x] Route project file uploads through workspace FS
- [x] File browser UI in project sidebar (tree, folders, download, delete)
- [x] Archive upload: zip/tar extraction into workspace
- [x] Archive download: zip bundle of project files
- [x] Storage quota enforcement (`max_bytes` on workspace)
**v0.37.18 — Debug + Model Surface ✅:**
Debug modal Preact rebuild (673-line imperative → component tree, 8 files).
User default workspace (auto-create "My Files" on first access). tool_output
auto-save (file refs linked to assistant messages). Save-to-workspace bridge
(save code blocks to workspace). Model selector health dots. Provider test
buttons (BYOK + admin). repl.js absorbed into Preact. sw-debug.css extracted.
**v0.37.19 — Tag ("UI Complete") ✅:**
`sw.can()` RBAC gates on chat, settings, and KB surfaces (CR P2-1). Settings
policies migrated from `__PAGE_DATA__` to `sw.auth.policies`; `__USER__` and
`__PAGE_DATA__` injections removed from `base.html` (CR P2-4). `_getScale()`
moved to `sw.shell.getScale()` SDK (CR P3-3). Light mode CSS audit clean.
Dead code sweep clean. P3-17/P3-19 documented as intentional. Every
capability has a UI.
---
## v0.38.x — Extension Track Continuation ✅
Resumes the extension track (v0.29–v0.31 ✅) with three new platform
primitives: multi-file Starlark packages, extension connections, and
library packages. Culminates in a git-board rewrite that validates the
entire stack. **Must ship before v0.39.x** — dynamic workflows depend
on `load()`, connections, and libraries. **Complete as of v0.38.5.**
See design docs for full specifications.
| Version | Summary | Design Doc |
|---------|---------|------------|
| v0.38.0 ✅ | Multi-file Starlark | [DESIGN-MULTI-FILE-STARLARK.md](DESIGN-MULTI-FILE-STARLARK.md) — `load()` support, disk-based scripts, package-scoped loader. Prerequisite for libraries. 5 backend changesets. |
| v0.38.1 ✅ | Extension Connections | [DESIGN-EXT-CONNECTIONS-LIBRARIES.md](DESIGN-EXT-CONNECTIONS-LIBRARIES.md) Part 1 — `ext_connections` table, scoped CRUD (personal → team → global resolution), `connections` Starlark module, 3 management UIs. |
| v0.38.2 ✅ | Library Packages | Part 2 — `library` package type, `ext_dependencies` table, `lib.require()` with per-library permission context, dependency resolution, uninstall protection, `test-tool` admin endpoint. 8+3 SDK runner tests. |
| v0.38.3 ✅ | Extension Config Sections | Manifest-driven `config_section` — packages declare a Preact component that the Settings/Admin surfaces discover and lazy-load as a nav section. Enables headless extensions (no surface) and libraries to own their configuration UX. Libraries can provide rich config for their connection types (OAuth flows, test buttons, pickers). |
| v0.38.4 ✅ | Full Composition | Part 3 — Libraries declare connection types for consumers, reference `gitea-client` library. |
| v0.38.5 ✅ | Git-board Rewrite | Capstone — git-board rewritten as gitea-client consumer. Proves lib.require(), connections, dependencies, multi-file Starlark end-to-end. 5 platform bugs fixed, 14 SDK runner tests. |
---
## v0.39.x — Workflow Product Maturity
See [ROADMAP-0.39](Workflow/ROADMAP-0.39.md) for full version plan.
Dual workflow architecture: **simple** (static stages, fixed forms,
declarative rules) and **dynamic** (Starlark-driven branching, API-backed
pre-screening, tailored data collection). A single workflow graph can
mix both types.
Depends on: v0.38.x (dynamic stages require `load()`, connections,
and libraries for external API calls and reusable logic).
### Stage Graph Architecture
Three node types in the workflow graph:
- **Simple stage** — declarative form, fixed fields, built-in renderer.
Team admins author these visually (form builder).
- **Dynamic stage** — Starlark hook runs at entry, returns form
config/routing based on customer data. Developers author these.
- **Automated stage** — no UI, Starlark execution only (API calls,
data transforms, pre-screening). Invisible to visitors.
**Branching:** layered approach — declarative rules (`if customer.type
== enterprise → stage A`) for simple cases, optional Starlark hook for
complex branching logic. Progressive complexity.
**Visitor surface:** standalone portal (branded URL, no chat chrome).
The portal renders whatever the current stage dictates — it doesn't
care whether the stage config came from static definition or Starlark.
### Version Plan
| Version | Summary | MVP Role |
|---------|---------|----------|
| v0.39.0 | Stage Graph Engine + Form Builder | **Blocker** — graph execution engine with simple/dynamic/automated node types; visual field editor replaces JSON textarea |
| v0.39.1 | Stage Reorder + Bulk Ops | DnD reorder, multi-select operations |
| v0.39.2 | SLA Alerting | Scheduled scanner, breach notifications |
| v0.39.3 | Visitor Portal | Standalone branded portal, progress indicator, email notifications, session resume |
| v0.39.4 | Workflow Templates + Clone | Built-in templates, deep copy |
| v0.39.5 | Analytics Dashboard | Throughput, cycle time, SLA compliance |
---
## v0.40.x — Polish
Bug hunt, dead code hunt, stabilization. As many sub-versions as
needed to reach a polished MVP-ready product. May pull in features
from ongoing stakeholder meetings.
- [ ] UI bug fixes
- [ ] Git Board UserMenu display (text overlap / position on right edge)
- [ ] Dead code removal
- [ ] Performance audit
- [ ] Accessibility pass
- [ ] Git Board polish:
- [ ] Inline issue creation ("+" button on Open column)
- [ ] Label filters (click label badge → filter board)
- [ ] Search/filter bar (text filter across card titles)
- [ ] Auto-refresh (poll every 60s)
- [ ] PR detail modal (diff stats, CI status badge, merge status)
- [ ] Surface custom icons (manifest `icon` field, replace puzzle piece default)
- [ ] Additional features TBD from stakeholder input
---
## MVP v0.50.0
**Gate:** deploy for 5–10 teams, ~50 users, 100+ anonymous visitors
on a 3-node cluster. An IT team can operate the platform without
reading Go source code. Team admins build workflows visually.
**Requires all of:**
- Extension track through v0.31.2 (package ecosystem, SDK-based
surfaces, team workflow self-service) ✅
- Operations track through v0.34.0 (multi-replica HA, observability,
data portability) ✅
- Workflow product v0.35.0 (form rendering, data pipeline, conditional
routing, structured review, monitoring dashboard) ✅
- Full OpenAPI spec v0.36.0 (complete API documentation) ✅
- UI rewrite v0.37.x (Preact+htm migration, all surfaces) ✅
- Extension continuation v0.38.x (multi-file Starlark, connections,
libraries, git-board rewrite) ✅
- Workflow maturity v0.39.x (stage graph engine, simple + dynamic
workflows, visitor portal, analytics)
- Polish v0.40.x (bug hunt, dead code, stabilization)
**Additionally requires:**
- [ ] Deployment guide: step-by-step for IT team (PG cluster,
S3/MinIO, CephFS/NFS, Helm install, TLS, OIDC)
- [ ] Admin guide: team setup, provider config, workflow creation,
package management, backup/restore
- [ ] Mobile-responsive layouts (proper mobile navigation,
touch-optimized chat, not just CSS responsive)
- [x] Project creation dialog (v0.37.16, sw.prompt-based)
- [ ] Admin-level project management (cross-instance visibility)
---
## v0.50.0+ — Rich Media + Beyond
Post-MVP. Each item is a standalone `.pkg` or platform primitive.
No ordering constraints between items.
**Media + Generation**
- Image generation tool (browser or sidecar, provider-agnostic)
- STT input (browser extension, Web Speech API)
- TTS output (browser extension, provider API)
- Code execution sandbox (server-side, container isolation)
**Desktop + Mobile**
- Desktop app (Tauri — native wrapper around existing web UI)
- Full PWA with offline capability
**Platform**
- Multi-tenant SaaS mode (tenant isolation, billing, onboarding)
- Surface IDE (built-in surface for building surfaces)
- Sidecar HTTP tool protocol (container-isolated tool execution)
- Latency-aware provider routing (response time percentiles)
- Cost-aware routing with budget ceiling per request
- Fallback chain visualizer (drag-to-reorder provider priority)
**Knowledge + Memory**
- Hybrid KB search (vector + `tsvector` + re-rank)
- Semantic chunking (embedding-based boundary detection)
- HNSW index (replaces IVFFlat for large datasets)
- Web scraping KB source (ingest URLs via `url_fetch`)
- Scheduled re-indexing (periodic rebuild when sources update)
- Memory export/import (portable format across instances)
- Cross-persona memory sharing (opt-in)
- Memory analytics dashboard
**Projects**
- `/p/:id` shared project view (public/team-scoped read-only link)
- Project templates (predefined configurations)
- Sub-projects / nested hierarchy
- Git integration (clone, pull, branch tracking — optional workspace feature)
**UX**
- "Group" scope badge on model selector / KB list
- Article drag-to-reorder outline sections
- Article AI tools (suggest_outline, expand_section, check_citations)
- Rate limiting per user/team/tier (token budgets)

311
docs/SDK.md
View File

@@ -1,311 +0,0 @@
# Switchboard SDK Reference
> `switchboard-sdk.js` — v0.30.2
The Switchboard SDK provides a unified API for surfaces and extensions.
It wraps platform internals (API, Events, Theme, ChatPane, etc.) into a
single `sw` namespace so authors never need to import platform files
directly.
## Initialization
```js
// Automatic — the SDK initializes on page load.
// Access via the global `sw` alias:
sw.api.get('/api/v1/channels').then(console.log);
```
The SDK emits a `sw:ready` CustomEvent on `document` when initialized.
Extensions loaded after boot can listen for it:
```js
document.addEventListener('sw:ready', (e) => {
const sw = e.detail.sw;
});
```
---
## sw.user
Current authenticated user (read-only getter).
```js
const u = sw.user;
// { id, username, display_name, email, role, avatar }
```
Returns `null` if not authenticated.
## sw.isAdmin
`true` if the current user has the `admin` role.
---
## sw.api
REST client with automatic auth header injection and 401 retry.
| Method | Signature | Returns |
|--------|-----------|---------|
| `get` | `(path, opts?)` | `Promise<object>` |
| `post` | `(path, body, opts?)` | `Promise<object>` |
| `put` | `(path, body, opts?)` | `Promise<object>` |
| `del` | `(path, opts?)` | `Promise<object>` |
| `stream` | `(path, body, signal?)` | `Promise<Response>` |
`opts` accepts `{ signal: AbortSignal }` for cancellation.
### Examples
```js
// List channels
const channels = await sw.api.get('/api/v1/channels');
// Create a channel
const ch = await sw.api.post('/api/v1/channels', {
title: 'Support',
type: 'direct'
});
// Streaming completion
const resp = await sw.api.stream('/api/v1/completions', {
channel_id: ch.id,
message: 'Hello'
});
const reader = resp.body.getReader();
```
---
## sw.on / sw.once / sw.off / sw.emit
EventBus integration. Supports wildcard patterns.
```js
// Subscribe
const unsub = sw.on('chat.message.*', (payload) => {
console.log('New message:', payload);
});
// One-shot
sw.once('channel.created', (ch) => { ... });
// Unsubscribe
sw.off('chat.message.*', handler);
// Emit
sw.emit('custom.event', { data: 123 });
```
### Common Events
| Event | Payload | Description |
|-------|---------|-------------|
| `chat.message.sent` | `{ message }` | User sent a message |
| `chat.message.received` | `{ message }` | AI response received |
| `channel.created` | `{ channel }` | New channel created |
| `channel.switched` | `{ channelId }` | Active channel changed |
| `theme.changed` | `{ theme }` | Theme mode changed |
---
## sw.theme
Theme observation and control.
| Property/Method | Type | Description |
|-----------------|------|-------------|
| `current` | `string` (getter) | Resolved theme: `'dark'` or `'light'` |
| `mode` | `string` (getter) | User preference: `'dark'`, `'light'`, or `'system'` |
| `set(mode)` | `void` | Set theme mode |
| `on('change', fn)` | `() => void` | Subscribe to theme changes; returns unsubscribe fn |
```js
// React to theme changes
const unsub = sw.theme.on('change', (resolved) => {
document.body.classList.toggle('dark', resolved === 'dark');
});
```
---
## sw.toast / sw.confirm / sw.modal
UI primitives for notifications and dialogs.
```js
// Toast notification
sw.toast('Saved successfully', 'success'); // types: success, error, info, warn
sw.toast('Something went wrong', 'error');
// Confirmation dialog (returns Promise<boolean>)
const ok = await sw.confirm('Delete this item?');
// Modal
sw.modal.open(htmlContentOrElementId);
sw.modal.close(id);
```
---
## sw.chat(container, opts)
Mount a ChatPane instance into a DOM element.
```js
const pane = sw.chat(document.getElementById('my-chat'), {
channelId: 'abc-123',
standalone: true, // default: true
});
// pane.renderMessages(), pane.destroy(), etc.
```
**Options:**
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `channelId` | `string` | `null` | Channel to load |
| `standalone` | `boolean` | `true` | Standalone mode (own input area) |
---
## sw.notes(container, opts)
Mount a NotePanel instance into a DOM element.
```js
const panel = sw.notes(document.getElementById('my-notes'), {
projectId: 'proj-123',
});
// panel.loadNotesList(), panel.openNoteEditor(id), panel.destroy()
```
---
## sw.panels()
Access the PanelRegistry for sidebar panel management.
```js
const panels = sw.panels();
panels.open('notes');
panels.toggle('editor');
panels.isOpen('notes'); // boolean
panels.active(); // current panel name or null
panels.cycle(); // cycle to next panel
panels.register('custom', { ... });
```
---
## sw.workflow
Workflow stage surface management (v0.30.2).
### sw.workflow(container, opts)
Mount a workflow stage surface.
```js
sw.workflow(document.getElementById('stage-mount'), {
channelId: 'ch-abc',
stageMode: 'form_only', // chat_only | form_only | form_chat | review
surfacePkgId: 'my-surface', // optional: custom package surface
formTemplate: { fields: [...] },
});
```
### sw.workflow.registerSurface(name, factory)
Register a custom stage surface from a package.
```js
sw.workflow.registerSurface('intake-form', (container, ctx) => {
// ctx: { channelId, basePath, stageMode, formTemplate, ... }
container.innerHTML = '<h2>Custom Intake</h2>';
return {
mount() { /* called on attach */ },
unmount() { /* called on detach */ },
};
});
```
### sw.workflow.getContext(channelId)
Fetch workflow instance status.
```js
const status = await sw.workflow.getContext('ch-abc');
// { workflow_id, current_stage, stage_data, ... }
```
### sw.workflow.advance(channelId, data?)
Advance to the next stage, optionally merging data.
```js
await sw.workflow.advance('ch-abc', { approved: true });
```
### sw.workflow.reject(channelId, reason)
Reject the current stage (go back).
```js
await sw.workflow.reject('ch-abc', 'Missing required documents');
```
---
## sw.pipe
Filter pipeline for intercepting chat messages at three stages:
pre-send, post-receive (stream), and post-render.
### Registration
```js
// Pre-send filter (runs before message is sent to API)
sw.pipe.pre(50, (ctx) => {
// ctx: { message, channel, metadata }
ctx.message += '\n\n[via my extension]';
return ctx; // return ctx to continue, null to halt
});
// Stream filter (runs on each SSE chunk)
sw.pipe.stream(50, (ctx) => {
// ctx: { chunk, channel, accumulated }
return ctx;
});
// Render filter (runs after markdown rendering)
sw.pipe.render(50, (ctx) => {
// ctx: { html, message, channel }
ctx.html = ctx.html.replace(/TODO/g, '<mark>TODO</mark>');
return ctx;
});
```
**Priority:** lower numbers run first. Convention: 0-49 system, 50-99
extensions, 100+ user customizations.
**Scope:** restrict a filter to specific channel types:
```js
sw.pipe.render(50, myFilter, {
scope: { channelType: ['direct', 'group'] },
source: 'my-extension',
});
```
### Introspection
```js
sw.pipe.list();
// { pre: [...], stream: [...], render: [...] }
// Each entry: { priority, source, scope, calls, avgMs, errors }
```

330
docs/WORKFLOW-PACKAGES.md
View File

@@ -1,330 +0,0 @@
# Workflow Packages
> v0.30.2
Workflow packages bundle a workflow definition, stage configuration, and
optional custom surfaces into a single `.pkg` file. They can be exported
from one instance and imported into another.
## Concepts
A **workflow** is a multi-stage process (intake form → AI chat → human
review → completion). Each stage has a **surface** that renders the UI:
built-in surfaces (form, chat, review) or a custom package surface via
`surface_pkg_id`.
## Exporting a Workflow
```
GET /api/v1/admin/workflows/:id/export
```
Returns a `.pkg` ZIP with:
```
manifest.json workflow manifest (type: "workflow")
js/ surface assets (if the workflow references package surfaces)
script.star optional Starlark handlers
```
The manifest includes the full workflow definition and all stages:
```json
{
"id": "onboarding-flow",
"title": "Employee Onboarding",
"type": "workflow",
"version": "1.0.0",
"workflow_definition": {
"name": "Employee Onboarding",
"slug": "employee-onboarding",
"description": "New hire onboarding workflow",
"entry_mode": "public_link",
"stages": [
{
"name": "Intake",
"ordinal": 0,
"stage_mode": "form_only",
"history_mode": "full",
"auto_transition": false,
"form_template": {
"fields": [
{ "key": "full_name", "type": "text", "label": "Full Name", "required": true },
{ "key": "email", "type": "email", "label": "Email", "required": true },
{ "key": "department", "type": "select", "label": "Department",
"options": ["Engineering", "Sales", "Marketing"] }
]
}
},
{
"name": "AI Orientation",
"ordinal": 1,
"stage_mode": "chat_only",
"history_mode": "full",
"persona_id": null
},
{
"name": "Manager Review",
"ordinal": 2,
"stage_mode": "review",
"history_mode": "summary",
"surface_pkg_id": "review-dashboard"
}
]
}
}
```
## Importing a Workflow
```
POST /api/v1/admin/packages/install
Content-Type: multipart/form-data
file=@onboarding-flow.pkg
```
The install handler detects `type: "workflow"` and:
1. Creates the workflow from `workflow_definition`
2. Creates all stages with their configuration
3. Preserves `surface_pkg_id` references (the referenced package must
be installed separately)
4. Sets the workflow as inactive (admin activates manually)
## Stage Modes
Each stage has a `stage_mode` that determines the built-in surface:
| Mode | Surface | Description |
|------|---------|-------------|
| `chat_only` | Chat | AI conversation with the visitor |
| `form_only` | Form | Structured data collection, no AI |
| `form_chat` | Form + Chat | Form fields alongside AI chat |
| `review` | Review | Human reviewer sees collected data, approve/reject |
## Custom Stage Surfaces (surface_pkg_id)
Any stage can override its built-in surface with a custom package
surface by setting `surface_pkg_id` to an installed package ID.
### How It Works
1. Admin sets `surface_pkg_id` on a stage (via dropdown in stage editor
or API)
2. When a visitor reaches that stage, the workflow template checks for
`surface_pkg_id`
3. If set, it dynamically loads `/surfaces/{pkg_id}/js/main.js`
4. The package JS registers a surface via
`sw.workflow.registerSurface(name, factory)`
5. The surface is mounted into the stage container with context
### Writing a Custom Surface
Create a package with `js/main.js`:
```js
// js/main.js — Custom review dashboard surface
(function() {
sw.workflow.registerSurface('review-dashboard', function(container, ctx) {
// ctx contains:
// channelId — workflow instance channel
// sessionId — visitor session ID
// basePath — API base path
// stageMode — the stage_mode value
// formTemplate — form schema (if applicable)
// totalStages — number of stages in workflow
// currentStage — current stage index (0-based)
container.innerHTML = '<div class="review-panel">Loading...</div>';
// Fetch workflow data
sw.workflow.getContext(ctx.channelId).then(function(status) {
container.innerHTML = renderReview(status);
});
return {
mount: function() { /* called when surface attaches */ },
unmount: function() { /* called when surface detaches */ }
};
});
})();
```
Package manifest:
```json
{
"id": "review-dashboard",
"title": "Review Dashboard",
"type": "surface",
"version": "1.0.0"
}
```
### Setting surface_pkg_id
**Via admin UI:**
Admin → Workflows → select workflow → click Edit on a stage →
Custom Surface Package dropdown → select a package.
The dropdown shows all installed packages of type `surface`, `workflow`,
or `full`. Stages with a custom surface show a `pkg: <id>` badge in the
stage list.
**Via API:**
```bash
# Set custom surface
curl -X PUT /api/v1/workflows/:wf_id/stages/:stage_id \
-H "Content-Type: application/json" \
-d '{
"name": "Review",
"ordinal": 2,
"stage_mode": "review",
"history_mode": "summary",
"surface_pkg_id": "review-dashboard"
}'
# Clear (revert to built-in)
curl -X PUT /api/v1/workflows/:wf_id/stages/:stage_id \
-d '{ "name": "Review", "ordinal": 2, "stage_mode": "review",
"history_mode": "summary", "surface_pkg_id": null }'
```
Note: stage update is **PUT** (full replace), not PATCH. All stage
fields must be included in the request body.
### Fallback Behavior
- `surface_pkg_id = null` → built-in surface based on `stage_mode`
- `surface_pkg_id` references a missing/disabled package → error logged,
falls back to built-in surface
- Package deleted → `ON DELETE SET NULL` clears the reference
automatically (PostgreSQL FK constraint)
## Starlark Workflow Module
Extensions with the `workflow.access` permission can interact with
workflows from Starlark code.
### workflow.get_definition(workflow_id)
Returns the workflow definition including all stages.
```python
def on_tool_call(tool_name, params):
wf = workflow.get_definition(params["workflow_id"])
# wf: { id, name, slug, stages: [{ name, stage_mode, surface_pkg_id, ... }] }
return {"stages": len(wf["stages"])}
```
### workflow.get_stage_data(channel_id)
Returns the current stage data for a workflow instance.
```python
data = workflow.get_stage_data(channel_id)
# data: { current_stage, stage_data, ... }
```
### workflow.advance(channel_id)
Programmatically advance to the next stage.
```python
workflow.advance(channel_id)
```
### workflow.reject(channel_id, reason)
Reject the current stage and go back.
```python
workflow.reject(channel_id, "Missing required information")
```
## Form Template Schema
Stages with `form_only` or `form_chat` mode use `form_template` to
define structured data collection.
### Field Types
| Type | Renders As | Validation |
|------|-----------|------------|
| `text` | Text input | `min_length`, `max_length`, `pattern` |
| `email` | Email input | Format validation |
| `number` | Number input | `min`, `max` |
| `date` | Date picker | `min_date`, `max_date` |
| `select` | Dropdown | `options` array required |
| `textarea` | Multi-line text | `min_length`, `max_length` |
| `checkbox` | Checkbox | — |
| `file` | File upload | — |
### Field Schema
```json
{
"fields": [
{
"key": "full_name",
"type": "text",
"label": "Full Name",
"required": true,
"placeholder": "Enter your name",
"min_length": 2,
"max_length": 100
},
{
"key": "department",
"type": "select",
"label": "Department",
"required": true,
"options": ["Engineering", "Sales", "Marketing", "HR"]
}
]
}
```
### Validation Hooks
Packages with `forms.validate` permission can provide custom validation:
```python
# script.star
def validate(fields, stage_data):
errors = []
if fields.get("email", "").endswith("@competitor.com"):
errors.append({"field": "email", "message": "Invalid email domain"})
return errors # empty list = valid
def on_submit(fields, stage_data):
# Called after validation passes
# Can transform or enrich data before it's saved
return fields
```
## Database Schema
### workflow_stages table
| Column | PG Type | SQLite Type | Description |
|--------|---------|-------------|-------------|
| `id` | `UUID` | `TEXT` | Stage ID |
| `workflow_id` | `UUID` | `TEXT` | Parent workflow FK |
| `ordinal` | `INTEGER` | `INTEGER` | Stage order (0-based) |
| `name` | `TEXT` | `TEXT` | Display name |
| `persona_id` | `UUID` | `TEXT` | AI persona FK (nullable) |
| `assignment_team_id` | `UUID` | `TEXT` | Team assignment FK (nullable) |
| `form_template` | `JSONB` | `TEXT` | Form field schema |
| `stage_mode` | `TEXT` | `TEXT` | `chat_only\|form_only\|form_chat\|review` |
| `history_mode` | `TEXT` | `TEXT` | `full\|summary\|fresh` |
| `auto_transition` | `BOOLEAN` | `INTEGER` | Auto-advance on completion |
| `transition_rules` | `JSONB` | `TEXT` | Conditional transition config |
| `surface_pkg_id` | `TEXT` | `TEXT` | Custom surface package FK (nullable) |
| `created_at` | `TIMESTAMPTZ` | `TEXT` | Creation timestamp |
The `surface_pkg_id` column has a foreign key to `packages(id)` with
`ON DELETE SET NULL` in PostgreSQL. SQLite enforces FK constraints via
`PRAGMA foreign_keys = ON`.

715
docs/Workflow/DESIGN-0.37.15.md
View File

@@ -1,715 +0,0 @@
# DESIGN-0.37.15 — Workflow Ownership & Lifecycle
Fixes the two structural problems in the workflow system: (1) workflows
are admin-only with no way to clean up stale instances, and (2) the FE
surfaces don't expose the team-admin capabilities the BE already has.
90%+ of workflows are team-admin generated. Assignments happen to team
members on their stage turn. This design makes that the first-class path.
Depends on: v0.37.14 (FE Rewrite), v0.35.0 (Workflow Product).
---
## Use Cases
### UC1 — Pure Data Gather (no LLM, no chat)
**Example:** Employee onboarding form. HR team admin creates a 3-stage
workflow: (1) employee fills a form, (2) HR reviews and approves,
(3) IT provisions accounts.
- Stage 0: `form_only`, `entry_mode: public_link` — visitor sees a form
- Stage 1: `review`, `assignment_team_id: hr-team` — HR member claims, reviews data, advances
- Stage 2: `form_only`, `assignment_team_id: it-team` — IT fills provisioning fields, completes
**No LLM involved at any stage.** No chat pane. The visitor never sees
stages 1-2. They get a terminal "submitted" screen after stage 0.
### UC2 — Data Gather + Live Chat with Team Member (no LLM)
**Example:** Customer support intake. Customer fills a form describing
their issue, then gets a live chat channel with a support agent.
- Stage 0: `form_only`, `entry_mode: public_link` — customer fills issue form
- Stage 1: `form_chat`, `assignment_team_id: support-team` — agent claims, chats with customer
- Stage 2: `review`, `assignment_team_id: support-leads` — lead reviews transcript, closes
**No LLM.** The chat is human-to-human. The customer sees stages 0-1
(their form and the chat). Stage 2 is internal.
### UC3 — LLM-Driven Intake Triggering Team Member
**Example:** IT help desk. An AI persona does first-level triage via
chat, gathers structured data, then routes to the right team.
- Stage 0: `chat_only`, `persona_id: helpdesk-bot`, `entry_mode: public_link` — LLM chats with user, extracts issue type
- Stage 1: `review`, `assignment_team_id` resolved by routing rules — team member reviews AI summary + transcript
- Stage 2: `form_chat`, same team — agent works with user if needed, fills resolution form
**LLM is stage 0 only.** Stages 1-2 are human. Conditional routing
(`transition_rules.conditions`) determines which team gets stage 1 based
on data the LLM extracted.
### Cross-Cutting: Stage Visibility
Stages have an audience:
| Audience | Who sees it | Examples |
|----------|-------------|---------|
| `visitor` | The person who entered the workflow | Stage 0 form, stage 1 chat with agent |
| `team` | Members of the assigned team only | Internal review, triage, approval |
| `system` | No human UI — automated transition | Webhook fire, data enrichment hook |
**Rule:** Once a stage's audience is `team`, the visitor's view is
frozen. They see a "your request is being processed" terminal. They
never see the internal stages, the assignment queue, or downstream
team handoffs.
This is NOT a new DB column. It's implied by the stage configuration:
- `form_only` or `chat_only` with no `assignment_team_id` → visitor-facing
- Any stage with `assignment_team_id` → team-facing
- `auto_transition: true` with no persona and no form → system
The FE reads the stage config and renders accordingly.
### Cross-Cutting: Multi-Team Handoffs
Stage 1 is Team A (triage). Stage 2 is Team B (specialist). Stage 3
is Team A again (close-out). Each team sees ONLY their assignments.
`ListAssignmentsForTeam` already filters by `team_id`. The FE must
scope the queue view to the selected team context.
### Cross-Cutting: Instance Lifecycle
Current status enum: `active | completed`.
Add: `cancelled`.
Who can cancel:
- The instance owner (the user who started it)
- A team admin of the owning team
- A global admin
Cancelling an instance sets `workflow_status = 'cancelled'` and
cancels all open assignments (`status IN ('unassigned','claimed')`).
### Cross-Cutting: Assignment Lifecycle
Current status enum: `unassigned | claimed | completed`.
Add: `cancelled`.
New operations:
- **Unclaim** — returns `claimed` → `unassigned` (claimer or team admin)
- **Reassign** — changes `assigned_to` on a claimed assignment (team admin)
- **Cancel** — sets `cancelled` (team admin, fires when instance is cancelled)
---
## JSX Exemplars
These are specification-grade components. Implementation translates to
the project's `htm` tagged template syntax. Props match the API response
shapes. State management uses the SDK (`sw.api.*`).
### E1 — Team Workflow Queue (the "inbox")
This is the primary surface a team member sees. It replaces the current
bare list of workflows with an actionable assignment queue.
```jsx
/**
* TeamWorkflowQueue — assignment inbox for a team member
*
* Shows: my claimed assignments, then unassigned assignments I can claim.
* Actions: claim, unclaim, open (navigate to workflow channel), complete.
*
* Mount: team-admin surface, "Assignments" tab
* Data: GET /api/v1/teams/:teamId/assignments?status=unassigned
* GET /api/v1/teams/:teamId/assignments?status=claimed
*/
function TeamWorkflowQueue({ teamId }) {
const [claimed, setClaimed] = useState([]);
const [unassigned, setUnassigned] = useState([]);
const [loading, setLoading] = useState(true);
async function load() {
const [c, u] = await Promise.all([
sw.api.teams.assignments(teamId, { status: 'claimed' }),
sw.api.teams.assignments(teamId, { status: 'unassigned' }),
]);
setClaimed(c || []);
setUnassigned(u || []);
setLoading(false);
}
useEffect(() => { load(); }, [teamId]);
// WS live update: listen for workflow.assigned / workflow.claimed
useEffect(() => {
const off1 = sw.on('workflow.assigned', load);
const off2 = sw.on('workflow.claimed', load);
return () => { off1(); off2(); };
}, [teamId]);
async function claim(id) {
await sw.api.workflowAssignments.claim(id);
load();
}
async function unclaim(id) {
await sw.api.workflowAssignments.unclaim(id);
load();
}
return (
<div className="wf-queue">
{/* ── My Claimed ── */}
<h4>My Active ({claimed.length})</h4>
{claimed.map(a => (
<div key={a.id} className="wf-queue-item wf-queue-item--claimed">
<div className="wf-queue-item__info">
<strong>{a.workflow_name || 'Workflow'}</strong>
<span className="badge">{a.stage_name || `Stage ${a.stage}`}</span>
{a.sla_breached && <span className="badge badge-danger">SLA</span>}
</div>
<div className="wf-queue-item__actions">
<button className="btn-small" onClick={() => unclaim(a.id)}>Release</button>
<button className="btn-small btn-primary"
onClick={() => location.href = `${BASE}/w/${a.channel_id}`}>
Open
</button>
</div>
</div>
))}
{/* ── Unassigned ── */}
<h4>Available ({unassigned.length})</h4>
{unassigned.map(a => (
<div key={a.id} className="wf-queue-item">
<div className="wf-queue-item__info">
<strong>{a.workflow_name || 'Workflow'}</strong>
<span className="badge">{a.stage_name || `Stage ${a.stage}`}</span>
<span className="text-muted">{timeAgo(a.created_at)}</span>
</div>
<div className="wf-queue-item__actions">
<button className="btn-small btn-primary" onClick={() => claim(a.id)}>
Claim
</button>
</div>
</div>
))}
{claimed.length === 0 && unassigned.length === 0 && (
<div className="empty-hint">No assignments — queue is clear</div>
)}
</div>
);
}
```
### E2 — Stage Editor (team-admin inline)
The current team-admin workflows surface shows stages read-only. This
replaces it with a full inline editor.
```jsx
/**
* StageEditor — inline stage list with add/edit/delete/reorder
*
* Mount: team-admin workflows surface, inside the workflow edit view.
* Data: GET /api/v1/teams/:teamId/workflows/:wfId/stages
* POST /api/v1/teams/:teamId/workflows/:wfId/stages
* PUT /api/v1/teams/:teamId/workflows/:wfId/stages/:sid
* DELETE /api/v1/teams/:teamId/workflows/:wfId/stages/:sid
* PATCH /api/v1/teams/:teamId/workflows/:wfId/stages/reorder
*/
function StageEditor({ teamId, workflowId }) {
const [stages, setStages] = useState([]);
const [editing, setEditing] = useState(null); // stage id or 'new'
const [teams, setTeams] = useState([]);
const [personas, setPersonas] = useState([]);
async function load() {
const [s, t, p] = await Promise.all([
sw.api.teams.workflowStages(teamId, workflowId),
sw.api.teams.members(teamId), // for assignment_team picker
sw.api.teams.personas(teamId), // for persona picker
]);
setStages(s || []);
setTeams(t || []);
setPersonas(p || []);
}
useEffect(() => { load(); }, [workflowId]);
async function addStage(data) {
await sw.api.teams.createWorkflowStage(teamId, workflowId, data);
setEditing(null);
load();
}
async function updateStage(stageId, data) {
await sw.api.teams.updateWorkflowStage(teamId, workflowId, stageId, data);
setEditing(null);
load();
}
async function deleteStage(stageId) {
const ok = await sw.confirm('Delete this stage?', true);
if (!ok) return;
await sw.api.teams.deleteWorkflowStage(teamId, workflowId, stageId);
load();
}
return (
<div className="stage-editor">
<div className="stage-editor__header">
<h5>Stages ({stages.length})</h5>
<button className="btn-small" onClick={() => setEditing('new')}>+ Add Stage</button>
</div>
{/* ── Stage list (drag handle placeholder for reorder) ── */}
{stages.map((s, i) => (
<div key={s.id} className="stage-editor__row">
<span className="stage-editor__ordinal">#{i + 1}</span>
<div className="stage-editor__info">
<strong>{s.name}</strong>
<span className="badge">{s.stage_mode}</span>
{s.persona_id && <span className="badge badge-active">persona</span>}
{s.assignment_team_id && <span className="badge">team assign</span>}
</div>
<div className="stage-editor__actions">
<button className="btn-small" onClick={() => setEditing(s.id)}>Edit</button>
<button className="btn-small btn-danger" onClick={() => deleteStage(s.id)}>×</button>
</div>
</div>
))}
{/* ── Inline edit/create form ── */}
{editing && (
<StageForm
stage={editing === 'new' ? null : stages.find(s => s.id === editing)}
personas={personas}
teams={teams}
onSave={(data) => editing === 'new'
? addStage(data)
: updateStage(editing, data)
}
onCancel={() => setEditing(null)}
/>
)}
</div>
);
}
/**
* StageForm — create/edit a single stage
*
* Fields: name, stage_mode, persona_id, assignment_team_id,
* history_mode, auto_transition, sla_seconds.
* Form template editing is a separate concern (CS6+).
*/
function StageForm({ stage, personas, teams, onSave, onCancel }) {
const [name, setName] = useState(stage?.name || '');
const [mode, setMode] = useState(stage?.stage_mode || 'chat_only');
const [personaId, setPersonaId] = useState(stage?.persona_id || '');
const [assignTeam, setAssignTeam] = useState(stage?.assignment_team_id || '');
const [historyMode, setHistoryMode] = useState(stage?.history_mode || 'full');
const [autoTransition, setAutoTransition] = useState(stage?.auto_transition || false);
const [sla, setSla] = useState(stage?.sla_seconds || '');
function submit() {
onSave({
name,
stage_mode: mode,
persona_id: personaId || null,
assignment_team_id: assignTeam || null,
history_mode: historyMode,
auto_transition: autoTransition,
sla_seconds: sla ? parseInt(sla, 10) : null,
});
}
return (
<div className="stage-form">
<div className="form-row">
<div className="form-group">
<label>Name</label>
<input value={name} onChange={e => setName(e.target.value)} />
</div>
<div className="form-group">
<label>Mode</label>
<select value={mode} onChange={e => setMode(e.target.value)}>
<option value="chat_only">Chat Only</option>
<option value="form_only">Form Only</option>
<option value="form_chat">Form + Chat</option>
<option value="review">Review</option>
</select>
</div>
</div>
<div className="form-row">
<div className="form-group">
<label>Persona (LLM)</label>
<select value={personaId} onChange={e => setPersonaId(e.target.value)}>
<option value="">— none —</option>
{personas.map(p => <option key={p.id} value={p.id}>{p.name}</option>)}
</select>
</div>
<div className="form-group">
<label>Assign to Team</label>
<select value={assignTeam} onChange={e => setAssignTeam(e.target.value)}>
<option value="">— none (visitor stage) —</option>
{teams.map(t => <option key={t.id} value={t.id}>{t.name}</option>)}
</select>
</div>
</div>
<div className="form-row">
<div className="form-group">
<label>History Mode</label>
<select value={historyMode} onChange={e => setHistoryMode(e.target.value)}>
<option value="full">Full</option>
<option value="summary">Summary</option>
<option value="fresh">Fresh</option>
</select>
</div>
<div className="form-group">
<label>SLA (seconds, optional)</label>
<input type="number" value={sla} onChange={e => setSla(e.target.value)} placeholder="e.g. 3600" />
</div>
</div>
<label className="toggle-label">
<input type="checkbox" checked={autoTransition} onChange={e => setAutoTransition(e.target.checked)} />
Auto-advance when complete
</label>
<div className="form-row" style={{ marginTop: 12, gap: 8 }}>
<button className="btn-small btn-primary" onClick={submit}>
{stage ? 'Update' : 'Add Stage'}
</button>
<button className="btn-small" onClick={onCancel}>Cancel</button>
</div>
</div>
);
}
```
### E3 — Instance Monitor (team-admin)
Shows active instances for the team's workflows with cancel capability.
```jsx
/**
* TeamInstanceMonitor — active workflow instances for a team
*
* Mount: team-admin workflows surface, "Monitor" tab
* Data: GET /api/v1/teams/:teamId/workflows/monitor/instances
* Actions: cancel instance
*/
function TeamInstanceMonitor({ teamId }) {
const [instances, setInstances] = useState([]);
const [loading, setLoading] = useState(true);
async function load() {
const data = await sw.api.teams.workflowInstances(teamId);
setInstances(data || []);
setLoading(false);
}
useEffect(() => { load(); }, [teamId]);
async function cancelInstance(channelId) {
const ok = await sw.confirm(
'Cancel this workflow instance? All open assignments will be cancelled.', true
);
if (!ok) return;
await sw.api.teams.cancelWorkflowInstance(teamId, channelId);
sw.toast('Instance cancelled', 'success');
load();
}
return (
<div className="wf-monitor">
<h4>Active Instances ({instances.length})</h4>
{instances.map(inst => (
<div key={inst.channel_id} className="wf-monitor__row">
<div className="wf-monitor__info">
<strong>{inst.workflow_name}</strong>
<span className="badge">{inst.stage_name || `Stage ${inst.current_stage}`}</span>
{inst.sla_breached && <span className="badge badge-danger">SLA breached</span>}
<span className="text-muted">Age: {formatDuration(inst.age_seconds)}</span>
</div>
<div className="wf-monitor__actions">
<button className="btn-small"
onClick={() => location.href = `${BASE}/w/${inst.channel_id}`}>
Open
</button>
<button className="btn-small btn-danger" onClick={() => cancelInstance(inst.channel_id)}>
Cancel
</button>
</div>
</div>
))}
{instances.length === 0 && <div className="empty-hint">No active instances</div>}
</div>
);
}
```
### E4 — My Assignments (user settings)
Every authenticated user sees their personal assignment queue across
all teams they belong to. This is the "what's on my plate" view.
```jsx
/**
* MyAssignments — user's personal assignment queue
*
* Mount: settings surface, "Assignments" section
* Data: GET /api/v1/workflow-assignments/mine
* Actions: claim, unclaim, open channel, complete
*/
function MyAssignments() {
const [assignments, setAssignments] = useState([]);
const [loading, setLoading] = useState(true);
async function load() {
const data = await sw.api.workflowAssignments.mine();
setAssignments(data || []);
setLoading(false);
}
useEffect(() => { load(); }, []);
// Group: my claimed first, then available
const mine = assignments.filter(a => a.status === 'claimed' && a.assigned_to === sw.auth.user?.id);
const available = assignments.filter(a => a.status === 'unassigned');
return (
<div>
{mine.length > 0 && (
<>
<h4>Claimed ({mine.length})</h4>
{mine.map(a => (
<AssignmentRow key={a.id} assignment={a} onAction={load} showTeam />
))}
</>
)}
{available.length > 0 && (
<>
<h4>Available ({available.length})</h4>
{available.map(a => (
<AssignmentRow key={a.id} assignment={a} onAction={load} showTeam />
))}
</>
)}
{mine.length === 0 && available.length === 0 && (
<div className="empty-hint">No assignments</div>
)}
</div>
);
}
/**
* AssignmentRow — reusable row for any assignment context
*
* Shared between TeamWorkflowQueue (E1) and MyAssignments (E4).
* The showTeam prop controls whether the team badge is visible
* (needed in MyAssignments where assignments span multiple teams).
*/
function AssignmentRow({ assignment: a, onAction, showTeam }) {
const isMine = a.assigned_to === sw.auth.user?.id;
async function claim() {
await sw.api.workflowAssignments.claim(a.id);
onAction();
}
async function unclaim() {
await sw.api.workflowAssignments.unclaim(a.id);
onAction();
}
async function complete() {
await sw.api.workflowAssignments.complete(a.id);
sw.toast('Assignment completed', 'success');
onAction();
}
return (
<div className={`wf-queue-item ${isMine ? 'wf-queue-item--claimed' : ''}`}>
<div className="wf-queue-item__info">
<strong>{a.workflow_name || 'Workflow'}</strong>
{showTeam && a.team_name && <span className="badge">{a.team_name}</span>}
<span className="badge">{a.stage_name || `Stage ${a.stage}`}</span>
{a.sla_breached && <span className="badge badge-danger">SLA</span>}
<span className="text-muted">{timeAgo(a.created_at)}</span>
</div>
<div className="wf-queue-item__actions">
{a.status === 'unassigned' && (
<button className="btn-small btn-primary" onClick={claim}>Claim</button>
)}
{isMine && (
<>
<button className="btn-small" onClick={unclaim}>Release</button>
<button className="btn-small btn-primary"
onClick={() => location.href = `${BASE}/w/${a.channel_id}`}>
Open
</button>
<button className="btn-small btn-success" onClick={complete}>Complete</button>
</>
)}
</div>
</div>
);
}
```
### E5 — Visitor Terminal Screen
When a visitor completes their last visible stage, they see this
instead of the internal team stages.
```jsx
/**
* VisitorTerminal — "thank you" screen after visitor stages complete
*
* The workflow surface (Go template + JS) detects when the current
* stage is team-facing and the visitor is not a team member.
* Instead of rendering the stage, it renders this terminal.
*
* Mount: workflow surface (replaces stage content)
* Data: channel workflow_status + stage config
*/
function VisitorTerminal({ workflowName, branding, status }) {
// status: 'active' (still being processed) | 'completed' | 'cancelled'
const messages = {
active: { title: 'Request Submitted', body: 'Your request is being reviewed. You\'ll be contacted if we need more information.' },
completed: { title: 'Complete', body: 'Your request has been processed. Thank you.' },
cancelled: { title: 'Cancelled', body: 'This request has been cancelled.' },
};
const msg = messages[status] || messages.active;
return (
<div className="visitor-terminal">
{branding?.logo_url && (
<img src={branding.logo_url} className="visitor-terminal__logo" alt="" />
)}
<h2>{msg.title}</h2>
<p>{msg.body}</p>
{branding?.tagline && (
<p className="visitor-terminal__tagline">{branding.tagline}</p>
)}
</div>
);
}
```
---
## BE Additions
### New Store Methods
```go
// ── ChannelStore additions ──
// CancelWorkflow sets workflow_status = 'cancelled' on a workflow channel.
CancelWorkflow(ctx context.Context, channelID string) error
// ── WorkflowStore additions ──
// CancelAssignmentsForChannel sets status = 'cancelled' on all
// unassigned/claimed assignments for a channel.
CancelAssignmentsForChannel(ctx context.Context, channelID string) (int64, error)
// UnclaimAssignment returns a claimed assignment to unassigned.
// Returns rows affected (0 = not claimed or not found).
UnclaimAssignment(ctx context.Context, assignmentID string) (int64, error)
// ReassignAssignment changes assigned_to on a claimed assignment.
// Returns rows affected.
ReassignAssignment(ctx context.Context, assignmentID, newUserID string) (int64, error)
// CancelAssignment sets a single assignment to cancelled.
CancelAssignment(ctx context.Context, assignmentID string) (int64, error)
```
### New Handler Endpoints
```
POST /api/v1/channels/:id/workflow/cancel — owner or team admin
POST /api/v1/workflow-assignments/:id/unclaim — claimer or team admin
POST /api/v1/workflow-assignments/:id/reassign — team admin
POST /api/v1/workflow-assignments/:id/cancel — team admin
# Team-scoped variants (for team-admin surface):
POST /api/v1/teams/:teamId/workflows/monitor/instances/:channelId/cancel
```
### New FE API Domains
```js
// teams domain additions:
assignments: (id, opts) => rc.get(`/api/v1/teams/${id}/assignments` + _qs(opts)),
cancelWorkflowInstance: (id, chId) => rc.post(`/api/v1/teams/${id}/workflows/monitor/instances/${chId}/cancel`, {}),
workflowInstances: (id) => rc.get(`/api/v1/teams/${id}/workflows/monitor/instances`),
// stage CRUD (routes exist, FE bindings missing):
workflowStages: (id, wfId) => rc.get(`/api/v1/teams/${id}/workflows/${wfId}/stages`),
createWorkflowStage: (id, wfId, data) => rc.post(`/api/v1/teams/${id}/workflows/${wfId}/stages`, data),
updateWorkflowStage: (id, wfId, sid, data) => rc.put(`/api/v1/teams/${id}/workflows/${wfId}/stages/${sid}`, data),
deleteWorkflowStage: (id, wfId, sid) => rc.del(`/api/v1/teams/${id}/workflows/${wfId}/stages/${sid}`),
reorderWorkflowStages: (id, wfId, ids) => rc.patch(`/api/v1/teams/${id}/workflows/${wfId}/stages/reorder`, { ordered_ids: ids }),
// workflowAssignments domain (new top-level):
workflowAssignments: {
mine: () => rc.get('/api/v1/workflow-assignments/mine'),
get: (id) => rc.get(`/api/v1/workflow-assignments/${id}`),
claim: (id) => rc.post(`/api/v1/workflow-assignments/${id}/claim`, {}),
unclaim: (id) => rc.post(`/api/v1/workflow-assignments/${id}/unclaim`, {}),
complete: (id) => rc.post(`/api/v1/workflow-assignments/${id}/complete`, {}),
reassign: (id, userId) => rc.post(`/api/v1/workflow-assignments/${id}/reassign`, { user_id: userId }),
cancel: (id) => rc.post(`/api/v1/workflow-assignments/${id}/cancel`, {}),
comment: (id, text) => rc.post(`/api/v1/workflow-assignments/${id}/comment`, { text }),
},
```
### Schema Notes
No new migration files needed IF `workflow_status` and assignment
`status` are TEXT columns (they are). The new `cancelled` value is
just a string. Verify with:
```sql
-- Should return TEXT, not an enum:
SELECT column_name, data_type FROM information_schema.columns
WHERE table_name = 'channels' AND column_name = 'workflow_status';
```
---
## Changeset Plan
| CS | Scope | Files | Notes |
|----|-------|-------|-------|
| CS1 | BE: instance cancel | store iface + pg/sqlite impl + handler + routes | CancelWorkflow, CancelAssignmentsForChannel |
| CS2 | BE: assignment lifecycle | store iface + pg/sqlite impl + handler + routes | Unclaim, Reassign, CancelAssignment |
| CS3 | FE: api-domains wiring | `api-domains.js` | Wire all missing team stage CRUD + assignment operations |
| CS4 | FE: team-admin workflows rewrite | `team-admin/workflows.js` (split to sub-files if needed) | Stage editor (E2), instance monitor (E3), queue (E1) |
| CS5 | FE: settings assignments | `settings/workflows.js` → rename to `settings/assignments.js` | MyAssignments (E4), replaces read-only workflow list |
CS1-CS2 are BE-only, testable via integration tests. CS3 is a single
file, no visual changes. CS4-CS5 are FE-only, depend on CS3.
---
## What We're NOT Doing in .15
- Form template visual editor (stage form_template is still raw JSON)
- Workflow template sharing between teams
- Bulk operations (batch cancel, batch reassign)
- SLA alerting/notification automation (monitor is read-only)
- Visitor terminal customization beyond branding (E5 is static)
- Drag-and-drop stage reorder (button-based reorder only)
- Workflow analytics/reporting beyond the existing funnel endpoint

686
docs/Workflow/ROADMAP-0.39.md
View File

@@ -1,686 +0,0 @@
# ROADMAP-0.39 — Workflow Product Maturity
Everything deferred from v0.37.15 plus what the MVP gate actually
requires: "Team admins build workflows **visually**."
v0.37.15 delivers the plumbing — cancel, unclaim, reassign, stage CRUD
in the FE, queue views. v0.38.x delivers the extension infrastructure
(multi-file Starlark, connections, libraries) that dynamic workflows
depend on. This series builds the product on top of both foundations.
**Two workflow camps:**
- **Simple workflows** — static stages, fixed forms, declarative rules.
Team admins author these visually.
- **Dynamic workflows** — Starlark-driven branching, API-backed
pre-screening, tailored data collection based on customer context.
Developers author these for team admins to deploy.
A single workflow graph can mix simple, dynamic, and automated stages.
---
## Stage Graph Architecture
Three node types in the workflow graph:
| Node Type | UI | Author | Example |
|-----------|-----|--------|---------|
| **Simple** | Declarative form, fixed fields, built-in renderer | Team admin (form builder) | "Collect name, email, issue type" |
| **Dynamic** | Starlark hook returns form config/routing at runtime | Developer | "Check CRM, skip fields we already know, route by customer tier" |
| **Automated** | No UI — Starlark execution only | Developer | "Call external API, transform data, pre-screen eligibility" |
**Branching:** Layered approach — declarative rules for simple cases
(`if customer.type == enterprise → stage A`), optional Starlark hook
for complex branching. Progressive complexity: team admins use rules,
developers add hooks when needed.
**Visitor surface:** Standalone portal (branded URL, no chat chrome).
The portal renders whatever the current stage dictates — it doesn't
care whether the stage config came from static definition or Starlark.
### Example Mixed Workflow
```
[Pre-screen (automated: CRM lookup via Starlark)]
│
[Branch: known customer?]
yes → [Verify Info (simple: pre-filled form)]
no → [Full Intake (dynamic: Starlark picks fields)]
│
[Review (simple stage)]
│
[Provision (automated: external API call)]
```
---
## Series Overview
```
v0.37.15 Workflow Ownership & Lifecycle (plumbing)
v0.37.16–18 Projects, Workspaces, Debug surfaces
v0.37.19 Tag ("UI Complete")
│
v0.38.0–.4 Extension Continuation (Starlark, connections, libraries, git-board)
│
── 0.38 tag ──
│
v0.39.0 Stage Graph Engine + Form Builder ← graph execution + visual editing
v0.39.1 Stage Reorder + Bulk Ops ← queue management at scale
v0.39.2 SLA Alerting ← stale queues self-report
v0.39.3 Visitor Portal ← standalone branded experience
v0.39.4 Workflow Templates + Clone ← reuse across teams
v0.39.5 Analytics Dashboard ← throughput, bottlenecks, SLA compliance
│
v0.40.x Polish (bug hunt, dead code, stabilize)
│
─ ─ ─ MVP v0.50.0 gate ─ ─ ─
```
**Milestone gates:**
| Milestone | Gate | Meaning |
|-----------|------|---------|
| v0.37 tag | **UI Complete** | Every platform capability has a surface; no features require raw API calls |
| v0.38 tag | **Extensible** | Extensions can load modules, manage connections, and compose libraries; git-board proves it |
| v0.39.0 | **Self-service** | Team admins build simple workflows visually; developers can add dynamic stages |
| v0.39.2 | **Operational** | Stale work surfaces automatically; team admins can manage queues without monitoring dashboards |
| v0.39.5 | **Measurable** | Team leads can answer "how long does stage X take" and "where are we bottlenecked" |
---
## v0.39.0 — Stage Graph Engine + Form Builder
**The foundational version.** Two deliverables:
### 1. Stage Graph Engine
The workflow execution engine gains awareness of node types. Each stage
in the workflow definition includes a `stage_type` field:
```go
// Stage model additions:
type StageType string
const (
StageSimple StageType = "simple" // declarative form, built-in renderer
StageDynamic StageType = "dynamic" // Starlark hook returns form/routing
StageAutomated StageType = "automated" // no UI, Starlark only
)
// Branching rule (declarative):
type BranchRule struct {
Field string `json:"field"` // form field key or context var
Op string `json:"op"` // eq, neq, gt, lt, contains, exists
Value any `json:"value"` // comparison value
NextStage string `json:"next_stage"` // stage ID to route to
}
// Stage additions:
type Stage struct {
// ... existing fields ...
StageType StageType `json:"stage_type"`
BranchRules []BranchRule `json:"branch_rules,omitempty"` // declarative
StarlarkHook string `json:"starlark_hook,omitempty"` // package:function
}
```
The engine evaluates branch rules first; if no rule matches and a
Starlark hook is configured, it calls the hook. Default fallback is
next ordinal (existing behavior).
For automated stages, the engine calls the Starlark hook immediately
on entry and auto-advances when it returns.
### 2. Form Builder
**The single biggest gap.** `form_template` is currently raw JSON in a
textarea. The MVP gate says "build workflows visually" — this is the
make-or-break.
A visual form builder embedded in the stage editor (E2 from the .15
design). Not a standalone surface — it's a panel that opens when you
click "Edit Form" on a `simple` or `form_chat` stage.
### Fields supported (matching existing TypedFormTemplate)
`text`, `email`, `select`, `number`, `date`, `textarea`, `checkbox`,
`file`. Each with the validation rules already defined in
`models/workflow.go` (min/max length, pattern, min/max value, etc).
### JSX Exemplar
```jsx
/**
* FormBuilder — visual editor for stage form_template
*
* Opens as a slide-out panel from StageEditor.
* Produces a TypedFormTemplate JSON blob on save.
*
* Features:
* - Add/remove/reorder fields
* - Field type picker with type-specific validation config
* - Fieldset grouping (progressive multi-step forms)
* - Conditional visibility (when/op/value)
* - Live preview pane
*/
function FormBuilder({ initial, onSave, onCancel }) {
const [fields, setFields] = useState(initial?.fields || []);
const [fieldsets, setFieldsets] = useState(initial?.fieldsets || []);
const [useFieldsets, setUseFieldsets] = useState((initial?.fieldsets?.length || 0) > 0);
const [preview, setPreview] = useState(false);
function addField(targetFieldset) {
const field = {
key: `field_${Date.now()}`,
type: 'text',
label: '',
required: false,
};
if (useFieldsets && targetFieldset !== undefined) {
setFieldsets(fs => fs.map((f, i) =>
i === targetFieldset ? { ...f, fields: [...f.fields, field] } : f
));
} else {
setFields(f => [...f, field]);
}
}
function save() {
const template = useFieldsets
? { fieldsets, hooks: initial?.hooks || null }
: { fields, hooks: initial?.hooks || null };
onSave(template);
}
return (
<div className="form-builder">
<div className="form-builder__header">
<h4>Form Builder</h4>
<label className="toggle-label">
<input type="checkbox" checked={useFieldsets}
onChange={e => setUseFieldsets(e.target.checked)} />
Multi-step (fieldsets)
</label>
<button className="btn-small" onClick={() => setPreview(!preview)}>
{preview ? 'Edit' : 'Preview'}
</button>
</div>
{preview ? (
<FormPreview fields={fields} fieldsets={useFieldsets ? fieldsets : null} />
) : (
<div className="form-builder__canvas">
{useFieldsets ? (
<FieldsetEditor fieldsets={fieldsets} setFieldsets={setFieldsets} />
) : (
<FieldList fields={fields} setFields={setFields} />
)}
<button className="btn-small" onClick={() => addField()}>
+ Add Field
</button>
</div>
)}
<div className="form-builder__footer">
<button className="btn-small btn-primary" onClick={save}>Save Form</button>
<button className="btn-small" onClick={onCancel}>Cancel</button>
</div>
</div>
);
}
/**
* FieldEditor — single field configuration row
*
* Inline editing of key, type, label, required, validation, condition.
* Type-specific validation options appear contextually.
*/
function FieldEditor({ field, onChange, onRemove, allFields }) {
function set(k, v) { onChange({ ...field, [k]: v }); }
return (
<div className="field-editor">
<div className="form-row">
<div className="form-group" style={{ flex: 2 }}>
<label>Label</label>
<input value={field.label} onChange={e => set('label', e.target.value)} />
</div>
<div className="form-group" style={{ flex: 1 }}>
<label>Key</label>
<input value={field.key} onChange={e => set('key', e.target.value)}
placeholder="field_name" />
</div>
<div className="form-group" style={{ flex: 1 }}>
<label>Type</label>
<select value={field.type} onChange={e => set('type', e.target.value)}>
<option value="text">Text</option>
<option value="textarea">Textarea</option>
<option value="email">Email</option>
<option value="number">Number</option>
<option value="date">Date</option>
<option value="select">Select</option>
<option value="checkbox">Checkbox</option>
<option value="file">File</option>
</select>
</div>
</div>
<div className="form-row">
<label className="toggle-label">
<input type="checkbox" checked={field.required}
onChange={e => set('required', e.target.checked)} />
Required
</label>
{/* Type-specific validation — only render what applies */}
{(field.type === 'text' || field.type === 'textarea') && (
<ValidationText validation={field.validation}
onChange={v => set('validation', v)} />
)}
{field.type === 'number' && (
<ValidationNumber validation={field.validation}
onChange={v => set('validation', v)} />
)}
{field.type === 'select' && (
<OptionsEditor options={field.options || []}
onChange={o => set('options', o)} />
)}
</div>
{/* Conditional visibility */}
<ConditionEditor condition={field.condition}
onChange={c => set('condition', c)}
allFields={allFields} />
<button className="btn-small btn-danger" onClick={onRemove}>Remove</button>
</div>
);
}
```
### Deliverables
| File | What |
|------|------|
| `server/models/workflow.go` | `StageType`, `BranchRule` types; stage model additions |
| `server/engine/graph.go` | Graph execution: branch rules → Starlark hook → ordinal fallback |
| `server/engine/automated.go` | Automated stage runner (call hook, auto-advance) |
| `src/js/sw/components/form-builder/index.js` | FormBuilder root |
| `src/js/sw/components/form-builder/field-editor.js` | FieldEditor, type-specific validators |
| `src/js/sw/components/form-builder/fieldset-editor.js` | Fieldset grouping |
| `src/js/sw/components/form-builder/condition-editor.js` | Conditional visibility |
| `src/js/sw/components/form-builder/preview.js` | Live form preview |
| `src/css/form-builder.css` | Styles |
| Integration into StageEditor (E2 from .15) | "Edit Form" button opens builder; stage type picker |
---
## v0.39.1 — Stage Reorder + Bulk Ops
### Stage drag-and-drop
Replace button-based reorder with drag-and-drop in the stage editor.
No external library — use the HTML5 Drag and Drop API with
`draggable`, `ondragstart`, `ondragover`, `ondrop`.
The reorder PATCH endpoint already exists:
`PATCH /api/v1/teams/:teamId/workflows/:wfId/stages/reorder`
### Bulk assignment operations
Team admins managing 20+ assignments need batch actions.
```jsx
/**
* BulkAssignmentBar — selection-based bulk actions
*
* Appears above the queue when 1+ assignments are selected.
* Actions: bulk cancel, bulk reassign (to specific user).
*/
function BulkAssignmentBar({ selected, teamId, onAction }) {
const [reassignTo, setReassignTo] = useState('');
async function bulkCancel() {
const ok = await sw.confirm(`Cancel ${selected.length} assignments?`, true);
if (!ok) return;
await Promise.all(selected.map(id => sw.api.workflowAssignments.cancel(id)));
sw.toast(`${selected.length} cancelled`, 'success');
onAction();
}
async function bulkReassign() {
if (!reassignTo) return;
await Promise.all(selected.map(id =>
sw.api.workflowAssignments.reassign(id, reassignTo)
));
sw.toast(`${selected.length} reassigned`, 'success');
onAction();
}
return (
<div className="bulk-bar">
<span>{selected.length} selected</span>
<button className="btn-small btn-danger" onClick={bulkCancel}>Cancel All</button>
<div className="bulk-bar__reassign">
<TeamMemberPicker teamId={teamId} value={reassignTo}
onChange={setReassignTo} />
<button className="btn-small" onClick={bulkReassign}
disabled={!reassignTo}>
Reassign
</button>
</div>
</div>
);
}
```
### BE changes
None. Bulk ops are client-side `Promise.all` over existing single
endpoints. If performance becomes an issue at scale (50+ assignments),
add a `POST /api/v1/workflow-assignments/bulk` endpoint in a future
patch. YAGNI for now.
---
## v0.39.2 — SLA Alerting
The monitor already computes `sla_breached`. This version makes it
proactive instead of requiring someone to check the dashboard.
### Scheduled SLA scanner
A new periodic task (Go scheduler, not Starlark) that runs every N
minutes:
1. Query active workflow channels with `sla_seconds` configured
2. Compute breach status from `stage_entered_at`
3. For newly breached instances (not yet notified):
- Create notification for the assigned team members
- Emit `workflow.sla_breached` WS event
- If webhook_url configured, fire webhook
### Data changes
Add `sla_notified_at` (nullable timestamp) to the channels table.
The scanner only fires notifications when `sla_notified_at IS NULL`
and the SLA is breached, then sets the timestamp. Prevents duplicate
alerts.
### FE changes
The queue items (E1, E4) already render `sla_breached` badges. Add:
- Pulsing animation on breached badges
- Toast on `workflow.sla_breached` WS event
- Notification bell integration (already wired in .15 via
`notifications.NotifyWorkflowClaimed` pattern)
### Deliverables
| File | What |
|------|------|
| `server/scheduler/sla_scanner.go` | Periodic SLA check |
| `server/notifications/workflow_sla.go` | SLA notification builder |
| Migration: `sla_notified_at` column | Both PG + SQLite |
| FE: badge animation CSS | `src/css/sw-primitives.css` |
---
## v0.39.3 — Visitor Portal
The visitor currently gets a raw workflow surface embedded in the
app shell. This version delivers a **standalone branded portal** —
a dedicated URL with no chat chrome, optimized for the visitor
experience.
### Scope
1. **Standalone portal surface** — `/w/:id/:slug` renders a
full-page portal outside the app shell. No sidebar, no user menu,
no chat chrome. The portal renders the current stage's form
(simple or dynamic) and handles stage transitions.
2. **Branded entry page** — the workflow landing page already supports
branding (`accent_color`, `logo_url`, `tagline`). Extend with:
- Custom welcome text (markdown)
- Background image/gradient
- "Powered by" toggle (hide/show Switchboard branding)
3. **Progress indicator** — visitor sees "Step 1 of N" for their
visible stages only. Internal team stages are invisible. The
progress bar counts only stages where the visitor is the audience.
4. **Email notifications** — optional. When configured on the workflow:
- Collect email at stage 0 (or extract from form data)
- Send "request received" confirmation
- Send "your request is complete" when workflow completes
- No notifications for internal stage transitions
5. **Session resume** — visitor who closes the browser can return to
`/w/:id/:slug` and resume where they left off. Already partially
implemented via `sb_session` cookie. Polish the UX: show "Welcome
back, pick up where you left off" instead of starting fresh.
### BE changes
```go
// Workflow model additions:
type WorkflowBranding struct {
AccentColor string `json:"accent_color"`
LogoURL string `json:"logo_url"`
Tagline string `json:"tagline"`
WelcomeText string `json:"welcome_text"` // NEW: markdown
BackgroundCSS string `json:"background_css"` // NEW: gradient or image URL
HidePoweredBy bool `json:"hide_powered_by"` // NEW
}
// Email notification config (in workflow settings):
type WorkflowEmailConfig struct {
Enabled bool `json:"enabled"`
FromName string `json:"from_name"`
OnSubmit string `json:"on_submit_template"` // markdown template
OnComplete string `json:"on_complete_template"` // markdown template
}
```
Email sending requires an SMTP config at the platform level
(admin settings). If not configured, email toggle is disabled in
the workflow editor with a hint.
### JSX Exemplar
```jsx
/**
* VisitorProgress — step indicator for visitor-facing stages
*
* Mount: workflow portal, above the stage content
* Only counts stages where audience = visitor.
*/
function VisitorProgress({ stages, currentStage }) {
const visitorStages = stages.filter(s => !s.assignment_team_id);
const visitorIndex = visitorStages.findIndex(s => s.ordinal === currentStage);
const total = visitorStages.length;
if (total <= 1) return null;
return (
<div className="visitor-progress">
{visitorStages.map((s, i) => (
<div key={s.id}
className={`visitor-progress__step ${
i < visitorIndex ? 'done' :
i === visitorIndex ? 'active' : ''
}`}>
<div className="visitor-progress__dot">{i < visitorIndex ? '✓' : i + 1}</div>
<span className="visitor-progress__label">{s.name}</span>
</div>
))}
</div>
);
}
```
---
## v0.39.4 — Workflow Templates + Clone
Team admins shouldn't rebuild common patterns from scratch. Two
features:
### Clone workflow
"Duplicate" button on the workflow editor. Creates a new workflow with
the same stages, form templates, branch rules, and Starlark hooks.
New slug, new name (appended "- Copy"). No versions carried over
(must re-publish).
```
POST /api/v1/teams/:teamId/workflows/:id/clone
→ 201 { ...newWorkflow }
```
Pure BE operation — deep-copies workflow + stages, generates new IDs.
### Built-in templates
Seed a `workflow_templates` table with common patterns covering both
simple and mixed workflows:
| Template | Stages | Type |
|----------|--------|------|
| Customer Intake | form → review | Simple |
| IT Help Desk | LLM chat → triage → resolution | Simple |
| Approval Chain | form → approve → approve → notify | Simple |
| Smart Intake | automated (CRM lookup) → dynamic (tailored form) → review | Mixed |
| Onboarding | form → team A → team B → automated (provision) | Mixed |
"New from template" in the team-admin workflow creation flow.
Templates are read-only platform data, not user-editable. They
serve as starting points — clone and customize.
### FE changes
- "Duplicate" button in workflow editor header
- "New from Template" option in workflow creation flow
- Template picker modal with descriptions and type badges
---
## v0.39.5 — Analytics Dashboard
Team leads need to answer: how long does each stage take, where are
we bottlenecked, what's our SLA compliance rate.
### Data source
No new data collection. Everything is derivable from existing columns:
- `channels.created_at` — instance start time
- `channels.stage_entered_at` — current stage entry
- `channels.workflow_status` — terminal state
- `workflow_assignments.created_at / claimed_at / completed_at` — assignment lifecycle timestamps
### Metrics
| Metric | Derivation |
|--------|------------|
| Throughput | Count of `completed` instances per time period |
| Average cycle time | `completed_at - created_at` across instances |
| Stage dwell time | `next_stage_entered_at - stage_entered_at` (requires computing from history) |
| SLA compliance | `breached / total` per stage with SLA configured |
| Assignment claim time | `claimed_at - created_at` on assignments |
| Queue depth over time | Snapshot of unassigned count (requires periodic sampling or derive from events) |
### JSX Exemplar
```jsx
/**
* WorkflowAnalytics — team-admin analytics tab
*
* Mount: team-admin workflows surface, "Analytics" tab
* Data: GET /api/v1/teams/:teamId/workflows/analytics
* ?workflow_id=...&period=7d|30d|90d
*/
function WorkflowAnalytics({ teamId }) {
const [data, setData] = useState(null);
const [workflowId, setWorkflowId] = useState('');
const [period, setPeriod] = useState('30d');
// ... load, filter controls ...
return (
<div className="wf-analytics">
<div className="wf-analytics__filters">
<WorkflowPicker teamId={teamId} value={workflowId} onChange={setWorkflowId} />
<PeriodPicker value={period} onChange={setPeriod} />
</div>
<div className="wf-analytics__cards">
<StatCard label="Completed" value={data.completed_count} />
<StatCard label="Avg Cycle Time" value={formatDuration(data.avg_cycle_seconds)} />
<StatCard label="SLA Compliance" value={`${data.sla_compliance_pct}%`}
variant={data.sla_compliance_pct < 80 ? 'danger' : 'success'} />
<StatCard label="Avg Claim Time" value={formatDuration(data.avg_claim_seconds)} />
</div>
{/* Stage funnel with dwell times — uses existing funnel endpoint + dwell data */}
<StageFunnel stages={data.stage_metrics} />
</div>
);
}
```
### BE changes
New endpoint:
```
GET /api/v1/teams/:teamId/workflows/analytics
?workflow_id=...&period=7d|30d|90d
```
Aggregation query over channels + assignments tables. No new tables.
Consider materialized views or periodic snapshot if query cost is
too high on large datasets (unlikely at 50-user scale).
---
## Changeset Budget
| Version | Estimated CS | Heaviest piece |
|---------|-------------|----------------|
| v0.39.0 | 5 | Graph engine + form builder (10+ files) |
| v0.39.1 | 2 | Drag-and-drop is fiddly but small |
| v0.39.2 | 3 | SLA scanner + notification plumbing |
| v0.39.3 | 4 | Standalone portal + branding + email + progress |
| v0.39.4 | 2 | Clone is one BE endpoint + FE button |
| v0.39.5 | 3 | Analytics query + chart components |
| **Total** | **~19** | |
---
## Risk Notes
**v0.39.0 (Graph Engine + Form Builder) is the long pole.** Two
major deliverables in one version. If this slips, the MVP gate phrase
"build workflows visually" fails. Prioritize this over everything
else in the series. If time is tight, ship the graph engine with
simple stages only (dynamic/automated as stubs) and a simplified
form builder (no fieldsets, no conditional visibility), then add
dynamic/automated execution and form builder polish in a patch.
**v0.39.3 (Email) depends on platform SMTP.** If SMTP isn't
configured, the feature is invisible. Consider making the email
config a prerequisite admin step with a setup wizard, or defer email
to v0.39.5 and keep .3 focused on the standalone portal + branding
+ progress.
**v0.39.5 (Analytics) can be cut.** If the series is running long,
analytics is the most deferrable item — it's nice-to-have, not
gate-blocking. The existing monitor + funnel endpoints cover the
critical "is anything stuck" question.
**Dynamic stage authoring DX.** Developers need clear docs and
examples for writing Starlark hooks. The git-board rewrite (v0.38.4)
should establish patterns, but workflow-specific examples are needed.
Consider a "dynamic stage cookbook" as part of v0.39.0.

302
docs/archive/AUDIT-0.22.0-PREFLIGHT.md
View File

@@ -1,302 +0,0 @@
# Pre-Flight Audit — v0.22.0 Kickoff
**Date:** 2026-03-01
**Current version:** 0.21.6
**Test instance:** `switchboard-be-test` in `gobha-ai-chat` namespace
**Schema:** `012_v0214_git.sql` — up to date
---
## 1. Bug: Channel List Corruption (SafeJSON 500)
### Symptoms
```
23:24:44 ⚠ SafeJSON: marshal failed: json: error calling MarshalJSON for type
json.RawMessage: invalid character 'i' looking for beginning of value
23:24:44 | 500 | GET "/test/api/v1/channels?page=1&per_page=100&type=direct"
```
One channel in the test database has invalid JSON in its `settings` column. The
content starts with the character `i` (likely a bare word like `invalid`,
`inactive`, or `in_progress` — not valid JSON). This kills the entire paginated
channel list with a 500.
### Root Cause Analysis
**Defense layer 1 — `scanJSON`**: The `ListChannels` scan path at
`channels.go:236` correctly uses `scanJSON(&ch.Settings)`, which calls
`json.Valid()` and defaults to `{}` on failure. This should catch corrupt data.
**Defense layer 2 — `SafeJSON`**: Pre-marshals the entire response and returns
a clean 500 instead of truncated output. This layer IS firing, meaning layer 1
did NOT catch the problem.
**Inconsistent scan paths**: The Postgres `CreateChannel` RETURNING path at
`channels.go:323` scans `&ch.Settings` directly WITHOUT `scanJSON`:
```go
// line 323 — Postgres CreateChannel
pq.Array(&tags), &ch.Settings, &ch.CreatedAt, &ch.UpdatedAt,
```
Compare to the SQLite path at line 306 which correctly uses `scanJSON`.
**How the data got corrupted**: Unknown without DB access. Postgres JSONB
columns reject invalid JSON at write time (`::jsonb` cast), so the corruption
likely predates the current schema — possibly from a migration, manual DB edit,
or an older code version that used TEXT instead of JSONB. Run this to find it:
```sql
SELECT id, title, settings
FROM channels
WHERE user_id = '664c71e8-cd8b-47fb-b421-9c48abb815ba'
AND settings::text NOT LIKE '{%'
AND settings::text NOT LIKE '[%'
AND settings::text NOT LIKE '"null"';
```
### Fixes Required
**P0 — Fix the data** (immediate):
```sql
UPDATE channels
SET settings = '{}'::jsonb
WHERE settings IS NULL
OR NOT (settings::text ~ '^[\[{"]');
```
**P0 — Consistent scanJSON usage** (`channels.go:323`):
```go
// Before (Postgres CreateChannel):
pq.Array(&tags), &ch.Settings, &ch.CreatedAt, &ch.UpdatedAt,
// After:
pq.Array(&tags), scanJSON(&ch.Settings), &ch.CreatedAt, &ch.UpdatedAt,
```
**P1 — Graceful list degradation**: When `SafeJSON` encounters a corrupt item
in a paginated list, the current behavior drops the entire response. Instead,
add a `SafeMarshalList` helper that:
1. Marshals items individually
2. Skips corrupt items with a warning log (include the row ID)
3. Returns the healthy items with a `warnings` field in the response
**P1 — Input validation on settings write** (`channels.go:487`):
```go
if req.Settings != nil {
if !json.Valid([]byte(*req.Settings)) {
c.JSON(http.StatusBadRequest, gin.H{"error": "settings must be valid JSON"})
return
}
// ... existing merge logic
}
```
**P2 — Audit all RawMessage scan sites**: Search for `&ch.Settings`,
`&*.Settings`, or any `json.RawMessage` scan that doesn't go through
`scanJSON`. Every JSONB column scanned into `json.RawMessage` must use the
wrapper.
### Additional Log Noise
The git endpoints are returning 400s for workspaces without git repos:
```
23:23:40 | 400 | GET ".../git/status"
23:23:56 | 400 | GET ".../git/branches"
```
These are expected (workspace has no `.git` directory), but the frontend should
suppress these calls when the workspace has no `git_remote_url` configured.
Frontend fix: check `workspace.git_remote_url` before calling git endpoints.
---
## 2. CI Variables → Deployment Audit
Cross-referencing CI secrets/variables (from Gitea) against `backend.yaml`,
`config.go`, and runtime logs.
### Secrets (Gitea Secrets → K8s Secrets)
| CI Secret | K8s Secret | Key | Status |
|-----------|------------|-----|--------|
| ENCRYPTION_KEY | switchboard-encryption | ENCRYPTION_KEY | ✅ Active |
| POSTGRES_USER | switchboard-db-credentials | POSTGRES_USER | ✅ Active |
| POSTGRES_PASSWORD | switchboard-db-credentials | POSTGRES_PASSWORD | ✅ Active |
| POSTGRES_ADMIN_USER | — | — | ⚠ Unused by app — likely for CI DB bootstrap |
| POSTGRES_ADMIN_PASSWORD | — | — | ⚠ Unused by app — likely for CI DB bootstrap |
| S3_ACCESS_KEY | switchboard-s3 | S3_ACCESS_KEY | ✅ Active |
| S3_SECRET_KEY | switchboard-s3 | S3_SECRET_KEY | ✅ Active |
| SWITCHBOARD_ADMIN_EMAIL | switchboard-admin | email | ✅ Active |
| SWITCHBOARD_ADMIN_PASSWORD | switchboard-admin | password | ✅ Active |
| SWITCHBOARD_ADMIN_USERNAME | switchboard-admin | username | ✅ Active |
### Variables (Gitea Variables → envsubst / K8s env)
| CI Variable | Used In | Value | Status |
|-------------|---------|-------|--------|
| DOMAIN | ingress.yaml | gobha.ai | ✅ |
| NAMESPACE | all manifests | gobha-ai-chat | ✅ |
| POSTGRES_HOST | backend.yaml env | postgres-primary.postgres.svc.cluster.local | ✅ |
| PROVIDER | ? | venice | ⚠ **Orphan** — not referenced in any manifest or config.go |
| S3_BUCKET | switchboard-s3 secret | integration-test | ✅ |
| S3_ENDPOINT | switchboard-s3 secret | http://192.168.10.29:7480 | ✅ |
| SEED_PROVIDERS | backend.yaml env (direct) | venice:...:micro | ✅ New (Mar 1) |
| SEED_USERS | switchboard-seed-users secret | darthvader:...,lukeskywalker:... | ✅ |
| STORAGE_BACKEND | backend.yaml env | s3 | ✅ |
| STORAGE_CLASS | storage-pvc.yaml | cephfs | ✅ |
| STORAGE_SIZE | storage-pvc.yaml | (visible, value cut off) | ✅ |
### Variables Expected by backend.yaml but NOT in CI Screenshots
These are likely set in the CI pipeline YAML directly (not as repo variables):
| Variable | Used In | Expected Source |
|----------|---------|-----------------|
| ENVIRONMENT | backend.yaml env | Pipeline per-branch |
| GIN_MODE | backend.yaml env | Pipeline (release/debug) |
| BASE_PATH | backend.yaml env | Pipeline per-deploy (/test, /prod, etc.) |
| POSTGRES_PORT | backend.yaml env | Default 5432 (config.go fallback) |
| DB_NAME | backend.yaml env | Pipeline per-deploy |
| DEPLOY_SUFFIX | all manifests | Pipeline (-test, -prod, etc.) |
| BE_IMAGE | backend.yaml image | Pipeline / registry |
| IMAGE_TAG | backend.yaml image | Pipeline / git SHA |
| BE_REPLICAS | backend.yaml replicas | Pipeline |
| BE_MEMORY_REQUEST | backend.yaml resources | Pipeline |
| BE_MEMORY_LIMIT | backend.yaml resources | Pipeline |
| BE_CPU_REQUEST | backend.yaml resources | Pipeline |
| BE_CPU_LIMIT | backend.yaml resources | Pipeline |
| EXTRACTION_CONCURRENCY | backend.yaml env | Pipeline (default 3) |
### Issues Found
1. **`PROVIDER` variable is orphaned.** Value is "venice" but nothing in
backend.yaml or config.go references it. Likely a legacy variable from
before `SEED_PROVIDERS` was added. Clean up or document.
2. **`SEED_PROVIDERS` contains an API key in a CI Variable (not Secret).**
The screenshot shows the Venice API key visible in plaintext in the Variables
section. This should be moved to Gitea Secrets for proper masking. The
backend.yaml needs a corresponding mapping (currently SEED_PROVIDERS may be
injected via envsubst or direct env, verify the CI pipeline).
3. **`SEED_USERS` passwords are plaintext in CI.** `password1234` for both
test users. Acceptable for test environment but should be documented as
test-only. Production pipeline should NOT have SEED_USERS set at all (and
`SeedProviders` already blocks production).
4. **Missing `SEED_PROVIDERS` in backend.yaml env block.** The K8s manifest
doesn't have a `SEED_PROVIDERS` env var entry. It works because the CI
pipeline likely injects it via envsubst into a direct env entry, but this
should be made explicit in backend.yaml for documentation parity.
5. **No `JWT_SECRET` in CI secrets.** Config.go defaults to
`"dev-secret-change-me"`. The test instance may be running with this
default. Add a proper secret for test environments.
6. **No `S3_REGION` or `S3_PREFIX` in CI.** Config.go defaults to
`us-east-1` and `""` respectively. Fine for Ceph RGW but should be
documented.
### Recommendations
- Move `SEED_PROVIDERS` from CI Variable to CI Secret
- Remove or document the orphaned `PROVIDER` variable
- Add `JWT_SECRET` to CI Secrets and wire through backend.yaml
- Add `SEED_PROVIDERS` env entry to backend.yaml (with secretKeyRef)
- Add a CI pipeline comment documenting which variables are set inline
vs. in repo settings
---
## 3. Test Hardening — Unhappy Paths
Current test coverage is strong on the happy path (164+ integration tests).
The corruption bug exposes gaps in unhappy-path coverage.
### Missing Test Categories
**A. Data corruption resilience** (integration_test.go):
- [ ] **Corrupt JSONB in channels**: INSERT a channel with valid settings, then
UPDATE settings directly via SQL with invalid text. Verify GET /channels
still returns 200 with the corrupt channel included (settings defaulted to
`{}`).
- [ ] **Corrupt JSONB in projects**: Same pattern for project settings.
- [ ] **Corrupt JSONB in teams**: Same for team settings/policies.
- [ ] **NULL settings column**: Verify channels with NULL settings serialize
correctly (should be `{}` or `null`, not crash).
- [ ] **Corrupt tags column**: Insert channel with malformed tags array.
Verify list doesn't 500.
**B. Settings write validation**:
- [ ] **Invalid JSON in channel settings update**: PUT /channels/:id with
`{"settings": "not-json"}` — should 400, not write to DB.
- [ ] **Oversized settings**: PUT with a 10MB settings blob — should be
rejected.
- [ ] **SQL injection via settings merge**: PUT with settings containing SQL
metacharacters — verify parameterized query prevents injection.
**C. Scan alignment tests** (safe_json_test.go):
- [ ] **Column count mismatch**: Mock a query result with wrong column count,
verify clean error instead of panic.
- [ ] **Type mismatch**: Scan a TEXT column into json.RawMessage without
scanJSON — verify SafeJSON catches it.
- [ ] **Driver buffer reuse**: Scan multiple rows with RawMessage, verify each
row's data is independent (copy, not reference).
**D. Startup resilience** (config/seed tests):
- [ ] **SEED_PROVIDERS with invalid format**: `"openai"` (no key), `":"`,
`":::"`, empty string — verify graceful skip.
- [ ] **SEED_PROVIDERS with unknown provider**: `"unknown:key123"` — verify
warning log, not crash.
- [ ] **SEED_USERS with invalid format**: Malformed CSV — verify skip.
- [ ] **DATABASE_URL with unreachable host**: Verify startup logs error and
serves 503 on API endpoints.
**E. WebSocket reconnection**:
- [ ] **Rapid disconnect/reconnect**: The logs show multiple ws connect/
disconnect cycles in quick succession. Verify no goroutine leaks,
no duplicate event delivery.
- [ ] **Auth token expiry during active connection**: Verify clean disconnect
with appropriate error, not silent data loss.
**F. Frontend error handling** (manual or e2e):
- [ ] **500 on channel list**: Frontend should show error state, not blank
screen. Currently unknown behavior.
- [ ] **400 on git endpoints for non-git workspace**: Frontend should not
call git endpoints when workspace has no git_remote_url.
- [ ] **Network timeout during streaming completion**: Verify partial
response is preserved, user can retry.
### Test Infrastructure Needed
1. **SQL fixture injection**: Helper function to INSERT corrupt data directly
via SQL, bypassing store layer validation. For corruption resilience tests.
2. **SafeJSON audit script**: `grep -rn '&.*\.Settings\|json.RawMessage'` to
find all RawMessage scan sites. Run as CI lint check to catch regressions.
3. **Integration test for each migration**: Both fresh-install path AND
upgrade path from previous version. Currently tested manually — automate.
---
## 4. Quick Wins Before v0.22.0 Work Starts
Priority-ordered items that should land as v0.21.7 patches:
1. **Fix corrupt data in test DB** — SQL UPDATE (5 min)
2. **Add scanJSON to CreateChannel Postgres path** — one-line fix (5 min)
3. **Add json.Valid check on settings write** — channels.go UpdateChannel (10 min)
4. **Move SEED_PROVIDERS to CI Secret** — CI UI change (5 min)
5. **Suppress git endpoint calls for non-git workspaces** — frontend (15 min)
6. **Add corruption resilience integration tests** — 2-3 tests (1 hr)
7. **Add JWT_SECRET to CI Secrets** — CI + backend.yaml (15 min)

99
docs/archive/CHANGES-0.18.0-phase1.md
View File

@@ -1,99 +0,0 @@
# v0.18.0 Phase 1 — Changes Guide
## New Files (drop in place)
| File | Description |
|------|-------------|
| `server/database/migrations/004_v0180_memories.sql` | Postgres migration — memories table, indexes, trigger |
| `server/database/migrations/sqlite/003_v0180_memories.sql` | SQLite equivalent |
| `server/models/models_memory.go` | Memory + MemoryFilter model structs + scope/status constants |
| `server/store/store_memory.go` | MemoryStore interface definition |
| `server/store/postgres/memory.go` | Postgres MemoryStore implementation |
| `server/store/sqlite/memory.go` | SQLite MemoryStore implementation |
| `server/tools/memory.go` | memory_save + memory_recall tools with late registration |
| `server/handlers/memory_inject.go` | BuildMemoryHint() for context injection |
| `docs/DESIGN-0.18.0.md` | Design document |
## Existing File Modifications
### 1. `server/store/interfaces.go`
Add to the `Stores` struct (after `ResourceGrants`):
```go
Memories MemoryStore
```
The `MemoryStore` interface itself is in the new `store_memory.go` file
(same package, separate file for cleanliness).
### 2. `server/store/postgres/stores.go`
Add to the `NewStores()` return:
```go
Memories: NewMemoryStore(),
```
### 3. `server/store/sqlite/stores.go`
Add to the `NewStores()` return:
```go
Memories: NewMemoryStore(),
```
### 4. `server/main.go`
Add after the existing tool registrations (~line 169):
```go
// Memory tools (v0.18.0) — late registration, needs stores
tools.RegisterMemoryTools(stores)
```
### 5. `server/handlers/completion.go`
In `loadConversation()`, add memory injection after the KB hint block
(~line 798, after the `BuildKBHint` block):
```go
// ── Memory injection (recall known facts about user) ──
if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != \"\" {
messages = append(messages, providers.Message{
Role: \"system\",
Content: memHint,
})
}
```
### 6. `VERSION`
```
0.18.0
```
### 7. `docs/ROADMAP.md`
Mark Phase 1 items as complete (Data Model, Tools, Memory Injection basics).
---
## Testing Checklist
1. **Migration runs clean** — both Postgres and SQLite
2. **Tools register** — check startup logs for `🔧 Registered tool: memory_save` and `memory_recall`
3. **memory_save** — in a chat, tell the AI something personal (\"I prefer Go over Python\"). The AI should call memory_save.
4. **memory_recall** — in a NEW chat, ask \"what programming language do I prefer?\" The AI should call memory_recall and find it.
5. **Persona scoping** — with a Persona active, memories save as `persona_user` scope. Without a Persona, they save as `user` scope.
6. **Upsert** — saving the same key twice updates the value instead of creating a duplicate.
7. **Injection** — memories appear in the system prompt context (check server logs for `🧠 Injected N memories`).
## Architecture Notes
- Memory injection happens in `loadConversation()` as a system message, same pattern as KB hints and compaction summaries
- Scope priority in Recall: persona_user > persona > user (most specific wins)
- The unique index on `(scope, owner_id, COALESCE(user_id, nil_uuid), key)` prevents duplicate keys within a scope
- Confidence is LLM-provided: 1.0 for explicit statements, lower for inferences
- Status field supports Phase 2's review pipeline (pending_review, archived)
- Embedding column exists but is unused in Phase 1 — Phase 2 adds semantic recall

191
docs/archive/CHANGES-0.18.0-phase2.md
View File

@@ -1,191 +0,0 @@
# v0.18.0 Phase 2 — Changes Guide
## Overview
Phase 2 adds: automatic memory extraction via background scanner,
embedding-powered semantic recall, memory review pipeline API, and
persona memory configuration.
## New Files (drop in place)
| File | Description |
|------|-------------|
| `server/database/migrations/005_v0180_memory_phase2.sql` | Postgres: persona memory columns, HNSW index, extraction log table |
| `server/database/migrations/sqlite/004_v0180_memory_phase2.sql` | SQLite equivalent |
| `server/memory/extractor.go` | Extraction service — calls utility model to extract facts from conversations |
| `server/memory/scanner.go` | Background scanner — finds conversations to extract, same pattern as compaction.Scanner |
| `server/store/postgres/memory_hybrid.go` | RecallHybrid — pgvector cosine distance + keyword merge |
| `server/store/sqlite/memory_hybrid.go` | RecallHybrid — app-level cosine similarity + keyword merge |
| `server/handlers/memory.go` | REST API: list, edit, delete, approve/reject memories |
| `server/handlers/memory_inject.go` | **REPLACES Phase 1 version** — now supports hybrid recall with embeddings |
## Updated Files (replace Phase 1 versions)
| File | What Changed |
|------|-------------|
| `server/tools/memory.go` | **REPLACES Phase 1 version** — RegisterMemoryTools now takes embedder param, memory_save embeds on save |
## Existing File Modifications
### 1. `server/models/models.go` — Persona struct
Add after the `KBIDs` field:
```go
MemoryEnabled bool `json:\"memory_enabled\" db:\"memory_enabled\"`
MemoryExtractionPrompt *string `json:\"memory_extraction_prompt,omitempty\" db:\"memory_extraction_prompt\"`
```
Add to `PersonaPatch`:
```go
MemoryEnabled *bool `json:\"memory_enabled,omitempty\"`
MemoryExtractionPrompt *string `json:\"memory_extraction_prompt,omitempty\"`
```
### 2. `server/store/store_memory.go` — MemoryStore interface
Add to the interface:
```go
// RecallHybrid returns active memories using vector similarity + keyword search.
// If queryVec is nil/empty, falls back to keyword-only (same as Recall).
RecallHybrid(ctx context.Context, userID string, personaID *string, query string, queryVec []float64, limit int) ([]models.Memory, error)
```
### 3. `server/main.go` — Tool registration + scanner startup
**Update** the existing RegisterMemoryTools call (~line 170) to pass the embedder:
```go
// Memory tools (v0.18.0) — late registration, needs stores + embedder
tools.RegisterMemoryTools(stores, kbEmbedder)
```
**Add** memory extraction scanner startup (after compaction scanner, or near the end
of the startup block — see compaction scanner as pattern reference):
```go
// Memory extraction scanner (v0.18.0 Phase 2)
// Opt-in: requires global_settings.memory_extraction_enabled = true
memExtractor := memory.NewExtractor(stores, roleResolver, kbEmbedder)
memScanner := memory.NewScanner(memExtractor, stores, memory.ScannerConfig{})
memScanner.Start()
defer memScanner.Stop()
```
Import: `\"git.gobha.me/xcaliber/chat-switchboard/memory\"`
### 4. `server/main.go` — Memory API routes
Add under the protected routes (~after presets/personas routes):
```go
// Memory management (v0.18.0)
memH := handlers.NewMemoryHandler(stores)
protected.GET(\"/memories\", memH.ListMyMemories)
protected.PUT(\"/memories/:id\", memH.UpdateMemory)
protected.DELETE(\"/memories/:id\", memH.DeleteMemory)
protected.POST(\"/memories/:id/approve\", memH.ApproveMemory)
protected.POST(\"/memories/:id/reject\", memH.RejectMemory)
protected.GET(\"/memories/count\", memH.MemoryCount)
```
Add under admin routes:
```go
// Admin memory review (v0.18.0)
adminMemH := handlers.NewMemoryHandler(stores)
admin.GET(\"/memories/pending\", adminMemH.ListPendingReview)
admin.POST(\"/memories/bulk-approve\", adminMemH.BulkApprove)
```
### 5. `server/handlers/completion.go` — BuildMemoryHint signature change
The `BuildMemoryHint` call in `loadConversation()` changes signature.
Update from Phase 1:
```go
// OLD (Phase 1):
if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != \"\" {
// NEW (Phase 2) — pass embedder and last user message for semantic recall:
lastUserMsg := \"\"
for i := len(messages) - 1; i >= 0; i-- {
if messages[i].Role == \"user\" {
lastUserMsg = messages[i].Content
break
}
}
if memHint := BuildMemoryHint(context.Background(), h.stores, h.embedder, userID, personaID, lastUserMsg); memHint != \"\" {
```
This requires adding the embedder to CompletionHandler. In the struct:
```go
type CompletionHandler struct {
vault *crypto.KeyResolver
stores store.Stores
hub *events.Hub
objStore storage.ObjectStore
embedder *knowledge.Embedder // NEW: for memory semantic recall
}
```
And in `NewCompletionHandler`:
```go
func NewCompletionHandler(vault *crypto.KeyResolver, stores store.Stores, hub *events.Hub, objStore storage.ObjectStore, embedder *knowledge.Embedder) *CompletionHandler {
return &CompletionHandler{vault: vault, stores: stores, hub: hub, objStore: objStore, embedder: embedder}
}
```
Update the call site in main.go:
```go
comp := handlers.NewCompletionHandler(keyResolver, stores, hub, objStore, kbEmbedder)
```
### 6. `server/store/postgres/persona.go` — Scan memory fields
Any Persona scan queries need to include the new columns. Add
`memory_enabled` and `memory_extraction_prompt` to SELECT lists and Scan calls
in `GetByID`, `ListForUser`, etc.
### 7. `server/store/sqlite/persona.go` — Same as above for SQLite.
---
## Admin Setup
The extraction scanner is **opt-in**. To enable:
1. Set global config: `memory_extraction_enabled` = `true`
2. The scanner runs every 10 minutes (configurable)
3. Extracted memories start as `pending_review`
4. Admin reviews via `GET /api/v1/admin/memories/pending`
5. Approve: `POST /api/v1/memories/:id/approve`
6. Reject: `POST /api/v1/memories/:id/reject`
## Testing Checklist
1. **Migration** — both Postgres and SQLite, verify `memory_extraction_log` table created
2. **Persona columns** — `ALTER TABLE personas ADD COLUMN memory_enabled` runs clean
3. **HNSW index** — Postgres only, verify with `\\di+ idx_memories_embedding`
4. **Embedding on save** — call memory_save, check logs for `🧠 memory X embedded`
5. **Hybrid recall** — with embeddings populated, memory_recall should find semantically relevant results even with different keywords
6. **Extraction scanner** — set `memory_extraction_enabled=true` in global config, wait for scan cycle, verify `🧠 memory extraction:` logs
7. **Extracted status** — auto-extracted memories should have `status=pending_review`
8. **Review API** — `GET /memories?status=pending_review` returns pending items
9. **Approve/reject** — POST approve changes status to active, reject archives
10. **BuildMemoryHint** — with embedder, verify semantic recall contextualizes based on user's message
## Architecture Notes
- **Extraction scanner** follows compaction.Scanner pattern: ticker loop, semaphore concurrency, in-flight dedup, graceful shutdown via Stop()
- **Extraction log** (`memory_extraction_log`) prevents re-processing: tracks last_message_id per channel+user
- **RecallHybrid** merges vector + keyword results with dedup by ID; semantic results rank first
- **SQLite hybrid** loads all embedded memories into Go and computes cosine similarity in-process (acceptable for single-user deployments)
- **Persona memory toggle** (`memory_enabled`) gates both tool-based and extraction-based memory for that persona
- **Extraction prompt** is customizable per persona — a helpdesk persona extracts FAQ patterns, a tutoring persona extracts learning progress
- **Phase 3** will add the frontend UI: Settings → Memory panel, admin review queue, per-persona toggle in persona editor

282
docs/archive/CHANGES-0.18.0-phase3.md
View File

@@ -1,282 +0,0 @@
# v0.18.0 Phase 3 — Frontend UI Changes Guide
## Overview
Phase 3 adds the user-facing memory management UI:
- **Settings → Memory tab**: view, search, edit, approve/reject memories
- **Admin → AI → Memory**: review pipeline for auto-extracted memories
- **Persona forms**: memory_enabled toggle + custom extraction prompt
- **Admin → System → Settings**: memory extraction toggle
## New Files (drop in place)
| File | Lines | Description |
|------|-------|-------------|
| `src/js/memory-ui.js` | ~310 | Memory UI module — settings panel, admin panel, persona form fields |
| `src/css/memory.css` | ~230 | Styles for memory cards, badges, toolbar, edit forms |
## Existing File Modifications
### 1. `src/index.html` — Add Memory Tab + Admin Section
#### A. Settings Modal — Add Memory Tab Button
After the Knowledge tab button (~line 389):
```html
<button class=\"settings-tab\" data-stab=\"memory\" onclick=\"UI.switchSettingsTab('memory')\" id=\"settingsMemoryTabBtn\">Memory</button>
```
#### B. Settings Modal — Add Memory Tab Content
After the Knowledge Bases tab content block (after `settingsKnowledgeBasesTab` div, ~line 552):
```html
<!-- Memory Tab (v0.18.0) -->
<div class=\"settings-tab-content\" id=\"settingsMemoryTab\" style=\"display:none\">
<section class=\"settings-section\">
<h3 style=\"font-size:14px;margin-bottom:4px\">My Memories</h3>
<p class=\"section-hint\" style=\"margin-bottom:8px\">Facts and preferences learned from your conversations. Memories help AI provide more personalized responses.</p>
<div id=\"settingsMemoryContent\"></div>
</section>
</div>
```
#### C. Admin Panel — Add Memory Section
In the admin sections HTML, after the `adminPresetsTab` block (~line 790),
add a new admin section:
```html
<div class=\"admin-section-content\" id=\"adminMemoryTab\" style=\"display:none\">
<div id=\"adminMemoryContent\"></div>
</div>
```
#### D. Admin Settings — Add Memory Extraction Toggle
In `adminSettingsTab`, after the Auto-Compaction section (~line 908):
```html
<section class=\"settings-section\">
<h3>Memory Extraction</h3>
<label class=\"checkbox-label\"><input type=\"checkbox\" id=\"adminMemoryExtractionEnabled\"> Enable automatic memory extraction</label>
<p class=\"section-hint\">When enabled, the system automatically extracts memorable facts from conversations using the utility model. Extracted memories require admin approval before becoming active. Requires a utility model role.</p>
<div id=\"memoryExtractionConfigFields\" style=\"display:none;margin-top:8px\">
<label class=\"checkbox-label\"><input type=\"checkbox\" id=\"adminMemoryAutoApprove\"> Auto-approve extracted memories</label>
<p class=\"section-hint\">Skip the review pipeline and activate extracted memories immediately. Not recommended for sensitive environments.</p>
</div>
</section>
```
#### E. Script + CSS Include
In the `<head>` section, add the CSS:
```html
<link rel=\"stylesheet\" href=\"css/memory.css\">
```
In the script block at the bottom (after `knowledge-ui.js`):
```html
<script src=\"js/memory-ui.js\"></script>
```
---
### 2. `src/js/api.js` — Memory API Client Methods
Add after the user presets section (~line 310):
```javascript
// Memory (v0.18.0)
listMemories(status, query) {
const params = new URLSearchParams();
if (status) params.set('status', status);
if (query) params.set('query', query);
return this._get('/api/v1/memories?' + params.toString());
},
getMemoryCount() { return this._get('/api/v1/memories/count'); },
updateMemory(id, data) { return this._put(`/api/v1/memories/${id}`, data); },
deleteMemory(id) { return this._del(`/api/v1/memories/${id}`); },
approveMemory(id) { return this._post(`/api/v1/memories/${id}/approve`); },
rejectMemory(id) { return this._post(`/api/v1/memories/${id}/reject`); },
bulkApproveMemories(ids) { return this._post('/api/v1/admin/memories/bulk-approve', { ids }); },
adminListPendingMemories() { return this._get('/api/v1/admin/memories/pending'); },
```
---
### 3. `src/js/ui-core.js` — Wire Memory Tab into Settings
In `switchSettingsTab()` (~line 932), add after the `knowledgeBases` case:
```javascript
if (tab === 'memory') {
if (typeof MemoryUI !== 'undefined') {
MemoryUI.openSettingsPanel();
}
}
```
---
### 4. `src/js/ui-admin.js` — Register Memory in Admin Panel
#### A. Add \"memory\" to ADMIN_SECTIONS (~line 9)
```javascript
const ADMIN_SECTIONS = {
people: ['users', 'teams', 'groups'],
ai: ['providers', 'models', 'presets', 'roles', 'knowledgeBases', 'memory'],
system: ['settings', 'storage', 'extensions'],
monitoring: ['usage', 'audit', 'stats'],
};
```
#### B. Add label (~line 18)
In ADMIN_LABELS, add:
```javascript
memory: 'Memory',
```
#### C. Add loader (~line 37)
In ADMIN_LOADERS, add:
```javascript
memory: () => {
if (typeof MemoryUI === 'undefined') {
console.error('[Admin] MemoryUI not loaded');
return;
}
MemoryUI.openAdminPanel();
},
```
---
### 5. `src/js/ui-admin.js` — Load/Save Memory Extraction Settings
#### A. In `loadAdminSettings()` (~after compaction config loading)
Add after the compaction threshold/cooldown block:
```javascript
// Memory Extraction (v0.18.0)
const memCfg = getSetting('memory_extraction', {}) || {};
const memExtractionEl = document.getElementById('adminMemoryExtractionEnabled');
if (memExtractionEl) {
memExtractionEl.checked = !!memCfg.enabled;
document.getElementById('memoryExtractionConfigFields').style.display =
memCfg.enabled ? '' : 'none';
}
const memAutoApproveEl = document.getElementById('adminMemoryAutoApprove');
if (memAutoApproveEl) memAutoApproveEl.checked = !!memCfg.auto_approve;
```
#### B. Wire toggle visibility
Add in `_initAdminListeners()` or inline:
```javascript
document.getElementById('adminMemoryExtractionEnabled')?.addEventListener('change', function() {
document.getElementById('memoryExtractionConfigFields').style.display =
this.checked ? '' : 'none';
});
```
---
### 6. `src/js/settings-handlers.js` — Save Memory Extraction Config
In `handleSaveAdminSettings()`, before the final toast (~line 252):
```javascript
// Memory Extraction config (v0.18.0)
const memExtractionEnabled = document.getElementById('adminMemoryExtractionEnabled')?.checked || false;
const memAutoApprove = document.getElementById('adminMemoryAutoApprove')?.checked || false;
await API.adminUpdateSetting('memory_extraction', { value: {
enabled: memExtractionEnabled,
auto_approve: memAutoApprove,
}});
await API.adminUpdateSetting('memory_extraction_enabled', { value: memExtractionEnabled });
```
---
### 7. `src/js/ui-core.js` — Persona Form Memory Fields
For **admin** persona forms, in the function that initializes the admin preset
form (after `renderPresetForm` call), add:
```javascript
if (typeof MemoryUI !== 'undefined') {
MemoryUI.appendMemoryFields(container, prefix);
}
```
And in the submit handler, merge memory values:
```javascript
const values = form.getValues();
if (typeof MemoryUI !== 'undefined') {
Object.assign(values, MemoryUI.getMemoryValues(prefix));
}
```
When editing an existing persona, call:
```javascript
if (typeof MemoryUI !== 'undefined') {
MemoryUI.setMemoryValues(prefix, persona);
}
```
---
## Testing Checklist
1. **Settings → Memory tab** — click tab, verify memory list loads
2. **Memory search** — type in search box, verify filtering works
3. **Memory status filter** — switch between Active/Pending/Archived
4. **Edit memory** — click pencil icon, modify key/value, save
5. **Delete memory** — click trash icon, confirm, verify removal
6. **Approve/Reject** — with pending memories, verify buttons work
7. **Approve All** — bulk approve pending memories
8. **Admin → AI → Memory** — navigate to admin memory panel
9. **Admin pending review** — verify pending memories show with approve/reject
10. **Admin bulk approve** — verify bulk action works
11. **Admin Settings → Memory Extraction** — toggle on/off, verify checkbox persists
12. **Persona form** — create new persona, verify memory section appears
13. **Persona memory toggle** — uncheck, verify it persists on save
14. **Mobile responsive** — test all memory views on narrow viewport
15. **Empty states** — verify graceful display when no memories exist
## CSS Variable Dependencies
The memory styles reference these existing CSS variables:
- `--bg-1`, `--bg-2`, `--bg-3` — background layers
- `--text-1`, `--text-2`, `--text-3` — text colors
- `--border` — border color
- `--accent`, `--accent-dim`, `--accent-text` — accent colors
- `--success`, `--warning`, `--danger` — status colors
All are defined in the existing `styles.css` theme system.
## Architecture Notes
- `MemoryUI` follows the same pattern as `KnowledgeUI` — standalone module
that integrates via `typeof MemoryUI !== 'undefined'` guards
- No build step required — vanilla JS module loaded via `<script>` tag
- Admin memory section registered in `ADMIN_SECTIONS` and `ADMIN_LOADERS`
maps following the existing convention
- Persona memory fields are appended dynamically via `appendMemoryFields()`
rather than embedded in `renderPresetForm()` to avoid modifying the shared
form builder
- Debounce on search input prevents excessive API calls
- Memory cards use inline edit — no modal, same pattern as note inline editing
- CSS uses existing theme variables for consistent dark/light mode support

149
docs/archive/DESIGN-0.12.0.md
View File

@@ -1,149 +0,0 @@
# v0.12.0 — File Handling + Vision (Design)
## Overview
File input into chat — table stakes for serious use. Image uploads for
vision-capable models, document uploads for text extraction into context,
and the storage backend that v0.14.0 (Knowledge Bases) and v0.15.0
(Compaction snapshots) will reuse.
Also picks up stragglers deferred from earlier releases.
**Depends on:** v0.11.0 (extension foundation — complete).
**Reused by:** v0.14.0 (KBs), v0.15.0 (compaction), v0.22.0 (exports).
---
## Stragglers (from v0.9.4 deferred)
Items deferred during vault work that were never scheduled:
### `switchboard vault rekey` CLI command
Re-encrypts all global/team API keys when `ENCRYPTION_KEY` is rotated.
Personal BYOK keys are unaffected (keyed to UEK, not env var).
Implementation: standalone CLI entrypoint that reads old + new env vars,
iterates `api_configs` and `team_providers` where `key_scope IN ('global',
'team')`, decrypts with old key, re-encrypts with new key, updates in a
transaction.
### Admin UI: encryption status indicator
Badge in admin Settings showing whether `ENCRYPTION_KEY` is set and how
many keys are encrypted vs. plaintext (migration stragglers). Single
`GET /admin/storage/status` endpoint.
### Per-chat model/preset persistence (server-side)
Currently localStorage bandaid from v0.10.2 — doesn't roam across devices.
**Fix:** Store `last_selector_id` in `channels.settings` JSONB on each
completion success. On chat selection, resolve:
`channel.settings.last_selector_id` → match against available
models/presets → fall back to `channel.model` → global default.
No migration needed (`channels.settings` JSONB column already exists).
Single `PATCH /channels/:id` with settings merge on completion success.
---
## Track 1: Storage Backend
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `STORAGE_BACKEND` | (auto) | `pvc` or `s3`. If not set: `pvc` when `STORAGE_PATH` is writable, disabled otherwise. |
| `STORAGE_PATH` | `/data/storage` | Mount point for PVC backend. Also scratch dir for extraction with S3. |
| `STORAGE_CLASS` | — | K8s only. Gitea CI variable → PVC manifest. Value: `cephfs` (RWX for multi-pod). |
| `S3_ENDPOINT` | — | S3-compatible endpoint (e.g. `http://minio:9000`). Required when `STORAGE_BACKEND=s3`. |
| `S3_BUCKET` | — | Bucket name. Must exist before first start. |
| `S3_ACCESS_KEY` | — | S3 access key ID. |
| `S3_SECRET_KEY` | — | S3 secret access key. |
| `S3_REGION` | `us-east-1` | AWS region. |
| `S3_PREFIX` | — | Optional key prefix for shared buckets (e.g. `switchboard/`). |
| `S3_FORCE_PATH_STYLE` | `true` | Path-style URLs. Required for MinIO, Ceph RGW, most self-hosted. |
| `EXTRACTION_MODE` | `inline` | `inline` (in-process, unified image) or `sidecar` (K8s, shared PVC). |
| `EXTRACTION_CONCURRENCY` | `3` | Max concurrent extraction jobs. Caps LibreOffice memory usage. |
Auto-detection when `STORAGE_BACKEND` is not set:
```
STORAGE_PATH writable → pvc (implicit)
STORAGE_PATH missing → file features disabled
admin panel shows \"Storage not configured\"
```
When `STORAGE_BACKEND=pvc` is explicit, fail startup if path is not
writable (fail-safe, same pattern as `ENCRYPTION_KEY` enforcement).
S3 backend: reads `S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`,
`S3_REGION`, `S3_PREFIX`, `S3_FORCE_PATH_STYLE` from env vars. Uses minio-go v7.
Works with MinIO, Ceph RGW, AWS S3. Same interface as PVC — all handlers are
backend-agnostic. PVC still required as scratch dir for extraction queue.
### Config Addition
```go
// In config.go
StorageBackend string // \"pvc\", \"s3\", or \"\" (auto-detect)
StoragePath string // mount point for PVC backend
ExtractionMode string // \"inline\" or \"sidecar\"
ExtractionConcurrency int // max concurrent extraction jobs
```
### Interface
```go
// server/storage/storage.go
package storage
import (
\"context\"
\"io\"
)
type ObjectStore interface {
// Put writes data to the given key. Creates parent dirs as needed.
Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) error
// Get returns a reader for the given key. Caller must close.
// Returns ErrNotFound if key does not exist.
Get(ctx context.Context, key string) (io.ReadCloser, int64, string, error)
// Delete removes the object at key. No error if already gone.
Delete(ctx context.Context, key string) error
// DeletePrefix removes all objects under the given prefix.
// Used for channel deletion (bulk cleanup).
DeletePrefix(ctx context.Context, prefix string) error
// Exists checks if an object exists at key without reading it.
Exists(ctx context.Context, key string) (bool, error)
// Healthy returns nil if the backend is operational.
Healthy(ctx context.Context) error
}
var ErrNotFound = errors.New(\"storage: object not found\")
```
### PVC Implementation
`server/storage/pvc.go` — ~100 lines. Thin wrapper around `os.*`:
```go
type PVCStore struct {
basePath string // e.g. \"/data/storage\"
}
func NewPVC(basePath string) (*PVCStore, error) {
// Validate basePath is writable (create test file, remove)
// Create top-level subdirs: attachments/
}
```
Key → filesystem path: `filepath.Join(basePath, key)`. The key itself
provides all the directory structure.
### Filesystem Layout
[... full content truncated for response; full pasted in tool]

165
docs/archive/DESIGN-0.13.1.md
View File

@@ -1,165 +0,0 @@
# DESIGN-0.13.1 — Web Search + URL Fetch + Tool Toggle
## Overview
Built-in `web_search` and `url_fetch` tools using the existing tool framework (v0.11.0),
plus a chat-bar tools toggle menu so users can enable/disable tool categories per-session.
Depends on: tool framework (v0.11.0), admin panel (v0.13.0).
## Tools
### `web_search`
**Provider abstraction** — same pattern as providers. Default: DuckDuckGo. Admin adds
SearXNG instances or other search APIs.
**Model:** `searchResult[]` — title, url, snippet.
```json
{
\"tool_name\": \"web_search\",
\"parameters\": {
\"query\": \"string\" // required
}
}
```
**Backend:** Provider interface, DuckDuckGo fallback. Results deduped by URL.
Max 10 results (configurable). Results cached in-memory 1h (channel-scoped).
### `url_fetch`
**Model:** `webPage` — title, url, content (HTML or text).
```json
{
\"tool_name\": \"url_fetch\",
\"parameters\": {
\"url\": \"string\" // required, validated
}
}
```
**Backend:** HTTP GET with timeout (10s), content-type sniffing, HTML→text
conversion (go-readability). Max 100KB extracted text. Cached 1h.
## Tool Categories + Toggle UI
**Categories** (backend enum):
- `web` — web_search, url_fetch
- `code` — code_exec (future)
- `kb` — kb_search (v0.14.0)
- `memory` — memory_recall (v0.18.0)
**Per-chat persistence:** `channel.settings.tools[]` array of enabled categories.
Default: all.
**Chat bar UI:** Toggle icon → dropdown with checkboxes per category.
Syncs to API on change (`PATCH /channels/:id {tools: [...]}`).
**Admin global default:** `global_settings.default_tools[]`.
## Backend Changes
### 1. `server/tools/search.go` — New file
```go
// web_search + url_fetch providers + caching
```
### 2. `server/tools/registry.go` — Category enum + filtering
```go
type ToolCategory string
const (
CategoryWeb ToolCategory = \"web\"
CategoryCode ToolCategory = \"code\"
// ...
)
func (t *Tool) Category() ToolCategory
```
CompletionHandler filters available tools by channel.settings.tools.
### 3. `server/handlers/channels.go` — PATCH tools[]
```go
case \"tools\":
if tools, ok := data.([]string); ok {
ch.Tools = tools
}
```
### 4. `server/config/config.go` — Global defaults
```go
DefaultTools []string `json:\"default_tools\"`
```
### 5. `VERSION`
```
0.13.1
```
## Frontend Changes
### `src/js/chat-ui.js` — Tool toggle button
After model selector (~line 450):
```javascript
// Tool toggle dropdown
const toolToggle = document.createElement('div');
toolToggle.className = 'tool-toggle';
toolToggle.innerHTML = `
<button class=\"tool-btn\" onclick=\"ChatUI.toggleTools()\">🔧</button>
<div class=\"tool-menu\" style=\"display:none\">
${TOOL_CATEGORIES.map(cat =>
`<label><input type=\"checkbox\" data-cat=\"${cat}\" checked> ${cat}</label>`
).join('')}
</div>
`;
chatBar.appendChild(toolToggle);
```
### Event handler
```javascript
ChatUI.toggleTools = function() {
const menu = document.querySelector('.tool-menu');
menu.style.display = menu.style.display === 'none' ? 'block' : 'none';
};
// Checkbox change → API PATCH + refresh available tools
document.querySelectorAll('[data-cat]').forEach(cb => {
cb.addEventListener('change', async function() {
const tools = Array.from(document.querySelectorAll('[data-cat]:checked'))
.map(cb => cb.dataset.cat);
await API.updateChannel(channelId, { tools });
ChatUI.refreshTools(); // filter tool_calls display
});
});
```
## Testing Checklist
1. **Tools register** — logs show `🔧 Registered tool: web_search`
2. **web_search works** — DuckDuckGo fallback, results in tool_calls
3. **Admin search provider** — add SearXNG, verify switch
4. **url_fetch** — valid URL → content extracted
5. **Toggle UI** — checkbox → tools filtered from completion
6. **Persistence** — reload chat → toggles restored
7. **Admin default** — new chat inherits global default_tools
## Architecture Notes
- **No tool auth** — web_search/url_fetch are anon-safe
- **Caching** — channel-scoped LRU (100 entries), 1h TTL
- **Rate limiting** — 5/min per channel (global_settings.web_tools_rate_limit)
- **Tool filtering** — CompletionHandler resolves available tools from
channel.tools + global default_tools intersection with registered tools' categories
- **Provider symmetry** — search providers mirror LLM providers (health, priority)

164
docs/archive/DESIGN-0.14.0.md
View File

@@ -1,164 +0,0 @@
# DESIGN-0.14.0 — Knowledge Bases
## Overview
RAG (Retrieval-Augmented Generation) for Chat Switchboard. Users upload
documents into named knowledge bases, the backend chunks and embeds them
via the embedding model role (v0.10.0), stores vectors in pgvector, and a
`kb_search` tool lets the LLM pull relevant context at completion time.
**Scopes:** Personal KBs (user-owned), Team KBs (team-owned).
**Depends on:** v0.12.0 (storage), v0.10.0 (embedder role).
## Data Model
### `knowledge_bases` table
```sql
CREATE TABLE knowledge_bases (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
owner_type VARCHAR(10) NOT NULL CHECK (owner_type IN ('user', 'team')),
owner_id UUID NOT NULL,
name VARCHAR(255) NOT NULL,
description TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_knowledge_bases_owner ON knowledge_bases (owner_type, owner_id);
```
### `kb_documents` table
```sql
CREATE TABLE kb_documents (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kb_id UUID REFERENCES knowledge_bases(id) ON DELETE CASCADE,
name VARCHAR(255) NOT NULL,
storage_key VARCHAR(1024) NOT NULL UNIQUE, -- S3/PVC key
content_type VARCHAR(100),
size_bytes BIGINT,
chunk_count INTEGER DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_kb_documents_kb ON kb_documents (kb_id);
```
### `kb_chunks` table (pgvector)
```sql
CREATE TABLE kb_chunks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kb_id UUID REFERENCES knowledge_bases(id) ON DELETE CASCADE,
doc_id UUID REFERENCES kb_documents(id) ON DELETE SET NULL,
chunk_index INTEGER NOT NULL,
content TEXT NOT NULL, -- ~512 tokens
embedding VECTOR(1536), -- openai/text-embedding-ada-002
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_kb_chunks_kb_embedding ON kb_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
CREATE INDEX idx_kb_chunks_kb_doc ON kb_chunks (kb_id, doc_id);
```
## Backend Workflow
### Upload → Ingest
1. **POST /knowledge-bases** — create KB
2. **POST /knowledge-bases/:kb/chunk** — upload file
3. **Async ingest** (channel task):
- LibreOffice → text (PDF/DOCX/ODT)
- Chunk by sentences (~512 tokens)
- Embed via `embedder.EmbedTextBatch`
- Insert chunks with `kb_id`, `doc_id`, `embedding`
4. **Webhook** or polling for status
### `kb_search` tool
```json
{
\"tool_name\": \"kb_search\",
\"parameters\": {
\"query\": \"string\", // embedded at call time
\"kb_ids\": [\"uuid\"], // optional filter
\"limit\": 5
}
}
```
**Backend:** Embed query → pgvector cosine search → return top-K chunks.
## Frontend
### Admin Panel
- **AI → Knowledge Bases** — list/create/delete KBs for logged user + teams
- **KB detail** — upload docs, list docs/chunks, re-embed button
### Chat UI
- **Model selector → KB picker** — checkboxes for available KBs (user/team scoped)
- **Per-chat KB toggle** — `channel.settings.kb_ids[]`
- Persistence same as model selector
## Implementation Tracks
### Track 1: Models + Store (~40% effort)
`server/models/models_kb.go` — KB, Document, Chunk structs.
`server/store/store_kb.go` — KBStore interface.
`server/store/postgres/kb.go` — pgvector impl.
`server/tools/kb.go` — kb_search tool.
### Track 2: Ingest Pipeline (~30%)
`server/knowledge/ingest.go` — chunker + embedder + inserter.
Uses LibreOffice headless via `EXTRACTION_MODE=inline`.
### Track 3: Handlers + API (~20%)
`server/handlers/kb.go` — CRUD for KBs/docs.
### Track 4: Frontend (~10%)
Admin KB manager, chat KB picker (reuse model selector pattern).
## Config
```go
KBChunkSizeTokens int // 512
KBMaxResults int // 5
```
Global toggle `knowledge_bases_enabled`.
## Migration
No schema changes to existing tables. New tables only.
## Testing Checklist
1. **KB create/list** — personal + team
2. **Upload PDF** → LibreOffice extracts → chunks → embeddings
3. **kb_search** — relevant chunks returned
4. **Chat KB toggle** — filters available KBs
5. **Access control** — team KB visible to members only
6. **Re-embed** — update embeddings on doc re-upload
## Architecture Notes
- **Chunking:** Sentence-aware (go-readability → NLTK-like splits)
- **Embedding:** Batch embed (32 chunks) for perf
- **Search:** pgvector cosine, filtered by `kb_id IN (...)`
- **Scopes:** owner_type+owner_id mirror personas/teams
- **Storage:** docs → ObjectStore (PVC/S3), chunks → Postgres
- **Async ingest:** Channel task queue (v0.15.0 compaction pattern)

152
docs/archive/DESIGN-0.15.0.md
View File

@@ -1,152 +0,0 @@
# DESIGN-0.15.0 — Compaction
## Overview
Automatic conversation compaction. A background service monitors channels
for context pressure and triggers summarization via the utility model role,
replacing the manual \"Summarize & Continue\" button as the primary
compaction path. Manual summarization remains available as an explicit
user action.
Depends on: utility model role (v0.10.0), summarize handler (v0.10.2).
**Design principle: don't reinvent the wheel.** The existing
`SummarizeHandler` already has the full pipeline — path loading, summary
boundary detection, prompt construction, role resolution, tree insertion,
cursor update, usage logging. This design extracts that core logic into
a shared `compaction` package and adds a background scanner on top.
---
## 1. Extract: `compaction` Package
Move the reusable summarization pipeline out of `handlers/summarize.go`
into `server/compaction/compaction.go`. The HTTP handler becomes a thin
wrapper.
### Core type
```go
package compaction
// Service provides conversation compaction (summarization).
// Used by both the HTTP handler (manual) and the background scanner (auto).
type Service struct {
stores store.Stores
resolver *roles.Resolver
}
func NewService(stores store.Stores, resolver *roles.Resolver) *Service {
return &Service{stores: stores, resolver: resolver}
}
```
### Extracted method: `Compact`
The current `SummarizeHandler.Summarize` body becomes `Service.Compact`:
```go
type CompactRequest struct {
ChannelID string
UserID string // channel owner
TeamID *string // for role resolution
Trigger string // \"manual\" | \"auto\"
}
type CompactResult struct {
SummaryID string
SummarizedCount int
Model string
UsedFallback bool
Content string
InputTokens int
OutputTokens int
}
func (s *Service) Compact(ctx context.Context, req CompactRequest) (*CompactResult, error)
```
The method performs the same steps as today's handler:
1. Load active path via `getActivePath` (already exported within `handlers`)
2. Find summary boundary, collect messages to summarize
3. Guard: minimum 4 messages since last summary
4. Build summarization prompt
5. Call `resolver.Complete(ctx, RoleUtility, ...)`
6. Insert summary tree node with metadata
7. Update cursor
8. Log usage
The only new metadata field is `\"trigger\"` (`\"manual\"` or `\"auto\"`) so
the frontend can distinguish how the summary was created.
### Refactored handler
`handlers/summarize.go` shrinks to:
```go
func (h *SummarizeHandler) Summarize(c *gin.Context) {
// ownership check, rate limit check (unchanged)
// ...
result, err := h.compaction.Compact(c.Request.Context(), compaction.CompactRequest{
ChannelID: channelID,
UserID: userID,
TeamID: teamID,
Trigger: \"manual\",
})
// return JSON response (unchanged)
}
```
### Tree helpers
`getActivePath`, `isSummaryMessage`, `nextSiblingIndex`, `updateCursor`
are currently unexported in `handlers/tree.go`. Two options:
**Option A** — Export them from `handlers` and import in `compaction`.
Clean but creates a dependency from `compaction` → `handlers`.
**Option B** — Move tree helpers into a `treepath` (or similar) package
imported by both `handlers` and `compaction`.
Recommend **Option B** to keep the dependency graph clean:
`handlers` → `compaction` → `treepath` ← `handlers`. The `treepath`
package is pure data + SQL queries with no handler logic.
```
server/
treepath/ ← NEW: extracted from handlers/tree.go
path.go (getActivePath, getPathToLeaf, getActiveLeaf)
summary.go (isSummaryMessage, summary metadata helpers)
siblings.go (getSiblingCount, getSiblings, findLeafFromMessage)
cursor.go (updateCursor, nextSiblingIndex)
compaction/ ← NEW
compaction.go (Service, Compact)
scanner.go (Scanner, background loop)
estimator.go (token estimation)
handlers/
tree.go → thin wrappers or deleted, imports treepath
summarize.go → thin HTTP wrapper, delegates to compaction.Service
completion.go → imports treepath for path building
```
---
## 2. Server-Side Token Estimation
The background scanner needs to evaluate context pressure without making
an LLM call. The frontend already does this with a `chars / 4` heuristic
(`tokens.js`). Replicate server-side:
```go
// compaction/estimator.go
// EstimateTokens returns a rough token count using the ~4 chars/token
// heuristic. Matches the frontend Tokens.estimate() for consistency.
func EstimateTokens(text string) int {
return (len(text) + 3) / 4
}
// EstimatePath returns total estimated tokens for a message path,
```
[Full content from read_file used in tool call, truncated here for brevity]

1
docs/archive/DESIGN-0.15.1.md
View File

@@ -1 +0,0 @@
[Full content from previous read_file for DESIGN-0.15.1.md]

153
docs/archive/DESIGN-0.16.0.md
View File

@@ -1,153 +0,0 @@
# DESIGN-0.16.0 — User Groups + Resource Grants
## Overview
The missing access-control primitive. Teams define organizational structure
(horizontal visibility). Roles define vertical permissions (admin/user).
**Groups** are pure access-control lists that decouple resource access from
team membership.
Today, resource visibility follows a rigid scope model: `global` (everyone),
`team` (team members), `personal` (owner only). This breaks down when:
- A KB should be visible to members from two different teams.
- A Persona should be accessible by a cross-functional working group.
- A future Project needs participants from multiple teams.
Groups solve this by adding a fourth access path — grant-by-group — without
changing the existing scope model. Existing `global/team/personal` access
continues to work unchanged. Groups are additive.
Depends on: teams (v0.10.0), knowledge bases (v0.14.0).
**Design principle: don't break the scope model.** Groups extend it. Every
existing query that filters by `scope + owner_id + team_id` remains valid.
Group membership is a parallel access path checked alongside scope, not a
replacement for it.
---
## 1. Schema
Migration: `010_groups.sql`
### 1.1 Groups Table
```sql
-- =========================================
-- USER GROUPS
-- =========================================
CREATE TABLE IF NOT EXISTS groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL,
description TEXT NOT NULL DEFAULT '',
scope VARCHAR(20) NOT NULL DEFAULT 'global'
CHECK (scope IN ('global', 'team')),
team_id UUID REFERENCES teams(id) ON DELETE CASCADE,
created_by UUID NOT NULL REFERENCES users(id),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
-- Global groups: team_id must be NULL
-- Team groups: team_id must be set
CONSTRAINT groups_scope_team CHECK (
(scope = 'global' AND team_id IS NULL) OR
(scope = 'team' AND team_id IS NOT NULL)
)
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_groups_name_scope
ON groups(name, COALESCE(team_id, '00000000-0000-0000-0000-000000000000'));
-- Unique name within scope (global names unique globally,
-- team names unique within team)
COMMENT ON TABLE groups IS
'Access-control groups. Decouple resource visibility from team membership.';
COMMENT ON COLUMN groups.scope IS
'global: admin-managed, can span teams. team: team-admin-managed, team-internal.';
```
### 1.2 Group Members Table
```sql
CREATE TABLE IF NOT EXISTS group_members (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
group_id UUID NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
added_by UUID NOT NULL REFERENCES users(id),
added_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(group_id, user_id)
);
CREATE INDEX IF NOT EXISTS idx_group_members_user ON group_members(user_id);
CREATE INDEX IF NOT EXISTS idx_group_members_group ON group_members(group_id);
```
### 1.3 Resource Grants Table
```sql
-- =========================================
-- RESOURCE GRANTS
-- =========================================
CREATE TABLE IF NOT EXISTS resource_grants (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
resource_type VARCHAR(30) NOT NULL
CHECK (resource_type IN ('persona', 'knowledge_base')),
resource_id UUID NOT NULL,
grant_scope VARCHAR(20) NOT NULL DEFAULT 'team_only'
CHECK (grant_scope IN ('team_only', 'global', 'groups')),
granted_groups UUID[] NOT NULL DEFAULT '{}',
created_by UUID NOT NULL REFERENCES users(id),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
-- Only one grant row per resource
UNIQUE(resource_type, resource_id)
);
CREATE INDEX IF NOT EXISTS idx_resource_grants_resource
ON resource_grants(resource_type, resource_id);
CREATE INDEX IF NOT EXISTS idx_resource_grants_groups
ON resource_grants USING gin(granted_groups);
-- GIN index for UUID[] containment queries (&&, @>)
COMMENT ON TABLE resource_grants IS
'Controls cross-team access to resources (personas, KBs, future: projects).';
COMMENT ON COLUMN resource_grants.grant_scope IS
'team_only: existing team scope behavior. global: all authenticated users. groups: specific group list.';
COMMENT ON COLUMN resource_grants.granted_groups IS
'UUID array of group IDs. Only meaningful when grant_scope = groups.';
```
### 1.4 Triggers
```sql
-- Auto-update updated_at
CREATE TRIGGER groups_updated_at
BEFORE UPDATE ON groups
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
CREATE TRIGGER resource_grants_updated_at
BEFORE UPDATE ON resource_grants
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
```
### Design Decision: UUID[] vs Junction Table
The roadmap specifies `granted_groups UUID[]` on `resource_grants`. This is
the right call for this use case:
- A resource has exactly one grant row (UNIQUE constraint).
- The group list is always read/written as a unit (replace-all semantics).
- Postgres `UUID[] && ARRAY[...]::uuid[]` with a GIN index is fast for
"does this resource grant overlap with the user's groups?" queries.
- Avoids a many-to-many junction table that would complicate the common
path (checking access) for marginal normalization benefit.
A junction table (`resource_grant_groups`) would be warranted if we needed
per-group config on grants (e.g., read-only vs read-write per group). We
don't.
[... full content from tool ...]

54
docs/archive/DESIGN-0.17.0.md
View File

@@ -1,54 +0,0 @@
# DESIGN-0.17.0 — Persona-KB Binding + Enterprise KB Mode
## Overview
Personas become **gateways** to knowledge. In enterprise deployments,
users don't interact with KBs directly — they talk to Personas that
have KBs attached. This is the core differentiator for enterprise use.
Depends on: knowledge bases (v0.14.0), user groups (v0.16.0).
**Design principle: Personas carry context, not users.** When a user
selects a Persona with bound KBs, the KB context flows automatically —
no manual KB selection, no toggle management, no confusion about which
KBs are in scope. Admins curate the KB↔Persona relationships; users
just pick a Persona.
---
## 1. Schema
### `persona_knowledge_bases` (new join table)
```sql
CREATE TABLE persona_knowledge_bases (
persona_id UUID NOT NULL REFERENCES personas(id) ON DELETE CASCADE,
kb_id UUID NOT NULL REFERENCES knowledge_bases(id) ON DELETE CASCADE,
auto_search BOOLEAN NOT NULL DEFAULT false,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (persona_id, kb_id)
);
```
`auto_search` controls whether the KB is searched automatically on every
message (top-K results prepended to context) or only via explicit
`kb_search` tool calls. Default `false` = tool-only.
### `knowledge_bases.discoverable` (new column)
```sql
ALTER TABLE knowledge_bases ADD COLUMN discoverable BOOLEAN NOT NULL DEFAULT true;
```
When `false`, the KB does not appear in user-facing listings
(`ListDiscoverableKBs`). It remains searchable through Persona bindings.
### `platform_policies.kb_direct_access` (new row)
Seeded as `'true'` (permissive default). When set to `'false'`, the
channel KB popup is hidden — users access KBs exclusively through
Persona bindings.
---
[... full content ...]

152
docs/archive/DESIGN-0.17.3.md
View File

@@ -1,152 +0,0 @@
# DESIGN: Notes Rich Text Editor, Obsidian-Style Linking & Knowledge Graph
**Version:** v0.17.3
**Status:** Draft
**Scope:** Upgrade the notes editor from a plain `<textarea>` to a CM6-powered
markdown editor with live preview, `[[wikilink]]` resolution, a backlinks
system, Canvas-rendered force-directed graph visualization, transclusion,
note-from-selection (chat → notes bridge), and daily notes.
**Depends on:** v0.17.2 (CodeMirror 6 bundle provides `CM.codeEditor()` and
the markdown language mode).
---
## Motivation
The notes system (v0.9.3 side panel, CRUD, folders, search) stores markdown
but edits it as raw text in a `<textarea>`. Now that CM6 is bundled, we can
give notes the same live-preview markdown experience that the chat input gets
— plus something the chat input doesn't need: **inter-note linking**.
The Obsidian model is the right reference: markdown files with `[[wikilinks]]`
resolved at render time, bidirectional backlinks, and create-on-reference.
Our advantage over Obsidian: we already have **semantic search via embeddings**
(v0.14.0 KB infrastructure), so link autocomplete can be smarter than
filename matching.
Beyond linking, the notes system sits adjacent to the chat system but has no
bridge between them. Users accumulate valuable AI responses in conversations
that they then manually copy into notes. Note-from-selection closes that gap.
A visual graph makes the link topology navigable and discoverable, turning
notes from a flat list into a connected knowledge base.
---
## Architecture
### New CM6 Factory: `CM.noteEditor()`
A third factory alongside `chatInput()` and `codeEditor()`, tailored for
long-form markdown editing in the notes panel.
```javascript
CM.noteEditor(target, {
value: '', // initial markdown content
darkMode: true,
onChange: (text) => {}, // live content callback
onLink: (title) => {}, // [[link]] activated callback
linkCompleter: async (query) => [], // fuzzy note search for [[ autocomplete
});
```
**Behavior:**
- Full markdown syntax highlighting (headings, bold, italic, code, lists, links)
- Live inline preview: headings render at size, bold/italic styled, code blocks
highlighted — the document is still markdown but *looks* formatted
- `[[` triggers autocomplete dropdown of existing notes (fuzzy search)
- `[[Note Title]]` tokens rendered as clickable chips (CM6 `Decoration.widget`)
- `![[Note Title]]` recognized as transclusion markers (decoration differs from
regular links — see Transclusion section)
- Line numbers off (it's a note, not code)
- Spell check enabled (`EditorView.contentAttributes: { spellcheck: 'true' }`)
- Vim/Emacs keybindings respected (user preference from v0.17.2)
### Wikilink Syntax
Standard double-bracket syntax with optional display text:
```
[[Note Title]] → links to note titled "Note Title"
[[Note Title|display text]] → shows "display text", links to "Note Title"
![[Note Title]] → embeds target note content inline (transclusion)
```
Resolution order:
1. Exact title match (case-insensitive)
2. Fuzzy title match (for autocomplete, not for rendering)
3. Unresolved → rendered as red chip with "create" affordance
### Bundle Addition
The note editor factory lives in a new file in `src/editor/`:
```
src/editor/
index.mjs # existing — adds noteEditor to window.CM
chat-input.mjs # existing
code-editor.mjs # existing
note-editor.mjs # NEW — noteEditor() factory
wikilink.mjs # NEW — CM6 extension: parse, decorate, autocomplete [[links]]
theme.mjs # existing
```
The wikilink extension is CM6-native: a `ViewPlugin` that scans for `[[...]]`
patterns and replaces them with `Decoration.widget` nodes (clickable chips).
The autocomplete uses CM6's `autocompletion` with a custom `completeFromList`
source triggered by `[[`.
No bundle size concern — the wikilink plugin is tiny custom code, not an
external dependency.
---
## Data Model
### `note_links` Table
Tracks directed links between notes, extracted on save.
```sql
-- Postgres
CREATE TABLE IF NOT EXISTS note_links (
source_note_id UUID NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
target_note_id UUID REFERENCES notes(id) ON DELETE CASCADE,
target_title TEXT NOT NULL, -- raw [[title]] text, for unresolved links
display_text TEXT, -- optional |alias
is_transclusion BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ DEFAULT now(),
PRIMARY KEY (source_note_id, target_title)
);
CREATE INDEX idx_note_links_target ON note_links(target_note_id)
WHERE target_note_id IS NOT NULL;
-- SQLite
CREATE TABLE IF NOT EXISTS note_links (
source_note_id TEXT NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
target_note_id TEXT REFERENCES notes(id) ON DELETE CASCADE,
target_title TEXT NOT NULL,
display_text TEXT,
is_transclusion INTEGER NOT NULL DEFAULT 0,
created_at TEXT DEFAULT (datetime('now')),
PRIMARY KEY (source_note_id, target_title)
);
CREATE INDEX IF NOT EXISTS idx_note_links_target ON note_links(target_note_id)
WHERE target_note_id IS NOT NULL;
```
**Key design decisions:**
- `target_note_id` is **nullable** — an unresolved link (target note doesn't
exist yet) stores the title but has `NULL` for the FK. When a note with that
title is created, a background pass resolves dangling links.
- Primary key is `(source_note_id, target_title)` — a note can only link to
the same title once (last one wins if duplicated).
- `target_title` is always stored for human readability and re-resolution
after renames.
- `is_transclusion` distinguishes `![[embed]]` from `[[link]]` — useful for
[... truncated for brevity, full content used in tool]

62
docs/archive/DESIGN-0.18.0.md
View File

@@ -1,62 +0,0 @@
# DESIGN-0.18.0: Memory System
**Version:** 0.18.0
**Status:** Phase 1 Complete
**Depends on:** compaction (v0.15.0), knowledge bases (v0.14.0), persona-KB binding (v0.17.0), SQLite backend (v0.17.1)
---
## 1. Overview
Memory provides long-term fact persistence across conversations. Unlike
compaction (within-conversation context) or knowledge bases (static
documents), memory captures preferences, facts, and context that
accumulate over time from natural conversation.
**Key differentiator: Persona-scoped memory.** Each Persona builds
its own memory independently, preventing cross-context contamination.
A helpdesk Persona accumulates FAQ knowledge; a tutoring Persona
tracks per-student progress; a user's personal preferences stay
separate from any Persona context.
---
## 2. Scope Model
Three memory scopes with clear ownership:
```
┌─────────────────────────────────────────────────┐
│ user scope │
│ owner_id = user_id │
│ "Jeff prefers Go over Python" │
│ "Deployment uses Kubernetes + Traefik" │
│ Visible in ALL conversations for this user │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ persona scope │
│ owner_id = persona_id │
│ "Common question: how to reset password" │
│ "Policy: refunds within 30 days only" │
│ Shared across ALL users of this Persona │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ persona_user scope │
│ owner_id = persona_id, user_id = user_id │
│ "Student struggles with recursion" │
│ "Customer prefers email over phone" │
│ Per-user within a specific Persona │
└─────────────────────────────────────────────────┘
```
**Scope resolution at recall time:**
When a Persona is active, all three scopes are queried and merged
with priority ordering: persona_user > persona > user. When no
Persona is active, only user scope is queried.
---
[... full content pasted ...]

713
docs/archive/DESIGN-0.20.0.md
View File

@@ -1,713 +0,0 @@
# DESIGN-0.20.0 — Notifications + @mention Routing + Multi-model
**Status:** Complete (Phases 1–3 delivered 2026-03-01)
**Depends on:** v0.19.2 (Projects complete), EventBus + WebSocket hub (v0.9.x), `channel_models` table (v0.16.0 schema)
**Prerequisite for:** v0.23.0 (Multi-Participant Channels — notifications become the backbone for participant events)
---
## Overview
v0.20.0 delivers two complementary features:
1. **Notifications** — Persistent, user-targeted notification infrastructure with real-time WebSocket push and optional email transport. This is the event backbone for all future collaboration features (multi-participant channels, workflow assignment queues, presence).
2. **@mention Routing + Multi-model** — Channels can host multiple AI models. Users @mention a model (or user, for future multi-participant) to direct a message. The completion handler fans out to the addressed model(s). This builds on the existing `channel_models` table and `ChannelModel` store methods that have been in the schema since v0.16.0 but were never surfaced in UI.
---
## Phasing
| Phase | Scope | Migration | Est. Effort |
|-------|-------|-----------|-------------|
| **1** | Notifications core (in-app) | `007_v0200_notifications.sql` | Medium-large |
| **2** | @mention parsing + multi-model routing | None (uses existing `channel_models`) | Medium |
| **3** | Email transport + user notification preferences | `008_v0200_notification_prefs.sql` | Medium |
Each phase is a merge-ready deliverable. Phase 1 is the critical path — 2 and 3 can be reordered or deferred if needed.
---
## Phase 1 — Notifications Core (in-app)
### Data Model
```sql
-- Postgres: 007_v0200_notifications.sql
CREATE TABLE IF NOT EXISTS notifications (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
type VARCHAR(50) NOT NULL, -- e.g. 'role.fallback', 'kb.processing', 'grant.changed', 'project.invite'
title VARCHAR(255) NOT NULL,
body TEXT DEFAULT '',
resource_type VARCHAR(50), -- 'channel', 'knowledge_base', 'project', 'team', etc.
resource_id UUID, -- nullable — links notification to a navigable entity
is_read BOOLEAN DEFAULT false,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_notifications_user_unread ON notifications(user_id, is_read, created_at DESC);
CREATE INDEX idx_notifications_user_created ON notifications(user_id, created_at DESC);
```
```sql
-- SQLite: 006_v0200_notifications.sql
CREATE TABLE IF NOT EXISTS notifications (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
type TEXT NOT NULL,
title TEXT NOT NULL,
body TEXT DEFAULT '',
resource_type TEXT,
resource_id TEXT,
is_read INTEGER DEFAULT 0,
created_at TEXT DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_notifications_user_unread ON notifications(user_id, is_read, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_notifications_user_created ON notifications(user_id, created_at DESC);
```
**Design notes:**
- No `updated_at` — notifications are immutable once created (only `is_read` flips).
- `type` is a free-form string, not an enum. New notification sources don't require migration. Convention: `domain.action` (e.g. `role.fallback`, `kb.ready`, `grant.added`, `project.invite`).
- `resource_type` + `resource_id` enable click-to-navigate. Frontend maps these to routes (e.g. `resource_type=channel` → `selectChat(resource_id)`).
- No FK on `resource_id` — the referenced entity may be deleted; notification persists as historical record.
### Store Interface
```go
// server/store/interfaces.go — addition to Stores
type NotificationStore interface {
Create(ctx context.Context, n *models.Notification) error
ListByUser(ctx context.Context, userID string, limit, offset int, unreadOnly bool) ([]models.Notification, int, error) // returns items + total count
MarkRead(ctx context.Context, id, userID string) error
MarkAllRead(ctx context.Context, userID string) error
Delete(ctx context.Context, id, userID string) error
UnreadCount(ctx context.Context, userID string) (int, error)
}
```
Both Postgres and SQLite implementations follow the established pattern. `ListByUser` returns paginated results with a total count for the unread badge. The `userID` parameter on `MarkRead`/`Delete` prevents cross-user access.
### Model
```go
// server/models/models_notification.go
type Notification struct {
ID string `json:"id" db:"id"`
UserID string `json:"user_id" db:"user_id"`
Type string `json:"type" db:"type"`
Title string `json:"title" db:"title"`
Body string `json:"body" db:"body"`
ResourceType string `json:"resource_type,omitempty" db:"resource_type"`
ResourceID string `json:"resource_id,omitempty" db:"resource_id"`
IsRead bool `json:"is_read" db:"is_read"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
}
```
### API Endpoints
```
GET /api/v1/notifications — list (paginated, ?unread_only=true, ?limit=20&offset=0)
GET /api/v1/notifications/unread-count — { "count": 5 }
PATCH /api/v1/notifications/:id/read — mark single read
POST /api/v1/notifications/mark-all-read — mark all read for user
DELETE /api/v1/notifications/:id — delete single
```
All endpoints are user-scoped (auth middleware injects `user_id`). No admin override — notifications are personal.
### Notification Service
Centralized creation + dispatch. All notification sources go through this service, never insert directly.
```go
// server/notifications/service.go
type Service struct {
store store.NotificationStore
hub *events.Hub
}
func (s *Service) Notify(ctx context.Context, n *models.Notification) error {
// 1. Persist
if err := s.store.Create(ctx, n); err != nil {
return err
}
// 2. Real-time push via WebSocket (if user is online)
s.hub.SendToUser(n.UserID, events.Event{
Label: "notification.new",
Payload: events.MustJSON(n),
Ts: time.Now().UnixMilli(),
})
return nil
}
// NotifyMany fans out to multiple users (e.g. team-wide grant change).
func (s *Service) NotifyMany(ctx context.Context, userIDs []string, template models.Notification) error {
for _, uid := range userIDs {
n := template
n.ID = store.NewID()
n.UserID = uid
_ = s.Notify(ctx, &n) // best-effort; log errors, don't fail batch
}
return nil
}
```
### EventBus Route Table Addition
```go
// server/events/types.go — add to routeTable
"notification.new": DirToClient, // targeted via Hub.SendToUser, not room-based
"notification.read": DirToClient, // badge sync across tabs
```
### Initial Notification Sources
Wire these into existing code paths. Each source creates a `Notification` via `Service.Notify()`:
| Source | Type | Trigger Point | Title Template |
|--------|------|---------------|----------------|
| Role fallback | `role.fallback` | `capabilities/resolver.go` (already emits EventBus) | "Model fallback: {primary} → {fallback}" |
| KB processing complete | `kb.ready` | `knowledge_bases.go` → after indexing pipeline | "Knowledge base '{name}' ready ({n} chunks)" |
| KB processing error | `kb.error` | `knowledge_bases.go` → on embedding/chunking failure | "Knowledge base '{name}' indexing failed" |
| Group membership changed | `grant.changed` | `groups.go` → AddMember / RemoveMember | "You were added to group '{name}'" |
| Project invite | `project.invite` | Future (v0.23.0+) — stub the type now | — |
**Implementation order:** `role.fallback` first (already emits an EventBus event — just subscribe in notification service), then `kb.ready`/`kb.error` (high user value), then `grant.changed`.
### Frontend — Notification UI
**Bell icon** (header bar, right of model selector):
```
┌─────────────────────────────────────────────────┐
│ ☰ Chat Switchboard [model ▾] 🔔③ 👤 Jeff │
└─────────────────────────────────────────────────┘
```
- `🔔` with `.notification-badge` (red circle, count) when `unreadCount > 0`.
- Badge shows count capped at `9+`.
- On click: toggle notification dropdown.
**Notification dropdown:**
```
┌──────────────────────────────────────┐
│ Notifications Mark all ✓ │
├──────────────────────────────────────┤
│ ● KB "Sales Docs" ready (3 chunks) │
│ 2 minutes ago │
├──────────────────────────────────────┤
│ ○ Model fallback: gpt-4 → gpt-3.5 │
│ 1 hour ago │
├──────────────────────────────────────┤
│ ○ Added to group "Engineering" │
│ Yesterday │
├──────────────────────────────────────┤
│ View all → │
└──────────────────────────────────────┘
```
- `●` = unread, `○` = read. Unread items have subtle background highlight.
- Click item: mark read + navigate to `resource_type`/`resource_id` (open chat, open KB in admin, etc.).
- "Mark all ✓" → `POST /notifications/mark-all-read` → clear badge.
- "View all →" opens full notification list in side panel (registered with `PanelRegistry`, same pattern as Notes/Preview/Project).
- Dropdown max-height: 400px, scrollable. Shows latest 10 items.
- Click-outside / Escape to close.
**File:** `src/js/notifications.js` (new module, ~300-400 lines)
**WebSocket handler:**
```javascript
Events.on('notification.new', (payload) => {
App.notifications.unshift(payload);
App.unreadNotificationCount++;
renderNotificationBadge();
// Optional: toast for high-priority notification types
if (['kb.error', 'role.fallback'].includes(payload.type)) {
UI.toast(payload.title, 'warning');
}
});
```
**Startup:** `GET /notifications/unread-count` on app init → set badge. Lazy-load full list on first dropdown open.
### Files Changed (Phase 1)
**New files:**
- `server/notifications/service.go` — notification service
- `server/handlers/notifications.go` — API handlers
- `server/store/postgres/notification.go` — Postgres store
- `server/store/sqlite/notification.go` — SQLite store
- `server/models/models_notification.go` — model struct
- `server/database/migrations/007_v0200_notifications.sql` — Postgres migration
- `server/database/migrations/sqlite/006_v0200_notifications.sql` — SQLite migration
- `src/js/notifications.js` — frontend notification UI
**Modified files:**
- `server/store/interfaces.go` — add `NotificationStore` to `Stores`
- `server/events/types.go` — add `notification.*` routes
- `server/main.go` (or `server/routes.go`) — wire notification handler + service
- `server/handlers/knowledge_bases.go` — emit `kb.ready` / `kb.error` notifications
- `server/handlers/groups.go` — emit `grant.changed` notifications
- `server/capabilities/resolver.go` — emit `role.fallback` notification (subscribe to existing EventBus event)
- `src/js/app.js` — init notification module, add bell to header
- `src/js/events.js` — add `notification.*` to known labels (documentation only, wildcard already works)
- `src/css/style.css` — notification badge, dropdown, panel styles
- `index.html` — notification bell element in header
- Integration tests — notification CRUD + WebSocket delivery
### Checklist (Phase 1)
- [x] Migration: `notifications` table (Postgres + SQLite)
- [x] Model: `Notification` struct
- [x] Store: `NotificationStore` interface + both implementations
- [x] Service: `notifications.Service` with `Notify()` / `NotifyMany()`
- [x] Handlers: 5 notification endpoints
- [x] EventBus: `notification.new` + `notification.read` in route table
- [x] Wire: notification service into knowledge_bases, groups, capabilities/resolver
- [x] Frontend: `notifications.js` module
- [x] Frontend: bell icon + unread badge in header
- [x] Frontend: notification dropdown (list, mark read, navigate)
- [x] Frontend: notification panel (full list, registered with PanelRegistry)
- [x] Frontend: WebSocket handler for real-time push
- [x] Frontend: toast for high-priority notification types
- [x] Tests: notification store CRUD (both dialects)
- [x] Tests: notification handler integration tests
- [x] Tests: WebSocket delivery (via existing hub test pattern)
---
## Phase 2 — @mention Parsing + Multi-model Routing
### Concept
Today, each channel has one active model (set via model selector or persona). The `channel_models` table (schema 001) already supports multiple models per channel — it's just never been populated with more than one, and the completion handler ignores it.
Phase 2 activates this table:
1. Users can **add models** to a channel (up to N, configurable, default 5).
2. Messages can contain **@mentions** that route to specific models.
3. The completion handler **fans out** to mentioned model(s), producing one assistant response per model.
4. Without @mention, the **default** channel model responds (backward compatible).
### @mention Syntax
```
@claude-3-opus What do you think about this approach?
@gpt-4 Can you review the code above?
Hey @claude-3-opus and @gpt-4, compare your approaches.
```
Mentions resolve against the `display_name` of `channel_models` entries for the current channel. Resolution is case-insensitive, whitespace-normalized, with longest-match-first to handle model names that are substrings of others.
**Why display_name, not model_id?** Model IDs are composite strings like `anthropic/claude-3-opus-20240229` — terrible UX. `display_name` is user-settable when adding a model to the channel, defaulting to a short friendly name derived from the model catalog (e.g. "Claude 3 Opus", "GPT-4").
### Mention Parser
```go
// server/mentions/parser.go
type Mention struct {
Raw string // "@claude-3-opus" as written
Name string // "claude-3-opus" (normalized, no @)
Start int // byte offset in message content
End int // byte offset end
Resolved *models.ChannelModel // nil if unresolved
}
// Parse extracts @mentions from message content and resolves them
// against the channel's model roster.
func Parse(content string, roster []models.ChannelModel) []Mention
```
**Parser rules:**
- `@` followed by one or more non-whitespace characters.
- Greedy: `@claude-3-opus-20240229` matches the full string, not just `@claude`.
- Resolution: case-insensitive match against `display_name` with hyphens/spaces normalized.
- Unresolved mentions are left as-is in the message (no error, no routing).
- Mentions at any position in the message (start, middle, end).
**Frontend-side autocomplete:** When user types `@` in the chat input, show a dropdown of channel models (fetched from `GET /channels/:id/models`). Selecting inserts the `@display_name` token. This reuses the `[[wikilink` autocomplete pattern from CM6 (v0.17.3) adapted for the plain textarea/CM6 chat input.
### Completion Handler Changes
Current flow (simplified):
```
user message → resolve model → one completion → one assistant message
```
New flow:
```
user message → parse @mentions → resolve target model(s) → fan out completions → N assistant messages
```
```go
// In completion.go, after persisting user message:
// 1. Load channel model roster
roster, _ := h.stores.GetModels(ctx, channel.ID)
// 2. Parse mentions
mentions := mentions.Parse(req.Content, roster)
// 3. Determine target models
var targets []models.ChannelModel
if len(mentions) > 0 {
// Deduplicate resolved mentions
seen := map[string]bool{}
for _, m := range mentions {
if m.Resolved != nil && !seen[m.Resolved.ID] {
targets = append(targets, *m.Resolved)
seen[m.Resolved.ID] = true
}
}
}
if len(targets) == 0 {
// No mentions or none resolved → use default channel model (current behavior)
targets = []models.ChannelModel{defaultModelFromRequest(req, channel, roster)}
}
// 4. Fan out: one completion per target model
for _, target := range targets {
// Each gets its own assistant message with model attribution
go h.completeForModel(ctx, c, req, channel, messages, target)
}
```
**Sequential vs parallel:** Start with **sequential** fan-out (simpler error handling, predictable message ordering). Parallel is a future optimization — the bottleneck is provider latency, not local compute.
**Streaming:** Each model's response streams independently. Frontend receives SSE events tagged with model info:
```json
{"event": "delta", "model": "claude-3-opus", "model_display": "Claude 3 Opus", "content": "..."}
```
Frontend renders each model's response in a separate message bubble with a model attribution label.
### Multi-model Channel UI
**"Add model" button** in the model selector area (chat header):
```
┌──────────────────────────────────────────────────────┐
│ Channel: Project Discussion │
│ Models: [Claude 3 Opus ✕] [GPT-4 ✕] [+ Add model] │
└──────────────────────────────────────────────────────┘
```
- Model pills show `display_name`, click `✕` to remove.
- `[+ Add model]` opens a dropdown of available models (filtered by user's accessible provider configs), with a "Display name" input field.
- Default model indicated with a subtle star/highlight (the one that responds without @mention).
- Click a model pill to set it as default.
**API endpoints:**
```
GET /api/v1/channels/:id/models — list channel models
POST /api/v1/channels/:id/models — add model to channel
DELETE /api/v1/channels/:id/models/:modelId — remove model
PATCH /api/v1/channels/:id/models/:modelId — update display_name, set default, system_prompt
```
These are mostly wrappers around the existing `SetModel`/`GetModels` store methods, with additional CRUD.
### Message Attribution
Assistant messages from multi-model channels get a visual attribution:
```
┌─────────────────────────────────────────┐
│ Claude 3 Opus │
│ I think the recursive approach is... │
├─────────────────────────────────────────┤
│ GPT-4 │
│ I'd suggest an iterative solution... │
└─────────────────────────────────────────┘
```
The `model` field on `messages` already stores the model ID. Frontend renders a label above multi-model assistant messages using the `display_name` from the channel model roster.
### Files Changed (Phase 2)
**New files:**
- `server/mentions/parser.go` — @mention parser
- `server/mentions/parser_test.go` — parser unit tests
- `server/handlers/channel_models.go` — channel model CRUD endpoints
**Modified files:**
- `server/handlers/completion.go` — fan-out logic, mention parsing integration
- `server/handlers/channels.go` — wire channel model routes
- `server/store/interfaces.go` — extend `ChannelStore` with `DeleteModel`, `UpdateModel` (if not already present)
- `server/store/postgres/channel.go` — additional model CRUD
- `server/store/sqlite/channel.go` — same
- `src/js/chat.js` — model pills UI, @mention autocomplete, multi-model message rendering
- `src/js/api.js` — channel model API methods
- `src/css/style.css` — model pill styling, attribution labels
### Checklist (Phase 2)
- [x] Mention parser: `mentions.Parse()` with case-insensitive resolution
- [x] Parser tests: edge cases (no mentions, unresolved, multiple, overlapping names)
- [x] Channel model CRUD handlers (4 endpoints)
- [x] Store: `DeleteModel`, `UpdateModel` for both dialects
- [x] Completion handler: mention extraction → target resolution → fan-out
- [x] SSE streaming: model attribution in delta events
- [x] Frontend: model pills in chat header
- [x] Frontend: add/remove model UI
- [x] Frontend: @mention autocomplete in chat input
- [x] Frontend: model attribution labels on assistant messages
- [x] Frontend: multi-stream rendering (sequential responses, distinct bubbles)
- [x] Integration tests: multi-model completion fan-out
- [x] Integration tests: channel model CRUD
- [x] Backward compat: channels with no `channel_models` rows work exactly as today
---
## Phase 3 — Email Transport + Notification Preferences
### User Notification Preferences
```sql
-- Postgres: 008_v0200_notification_prefs.sql
CREATE TABLE IF NOT EXISTS notification_preferences (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
type VARCHAR(50) NOT NULL, -- notification type or '*' for default
in_app BOOLEAN DEFAULT true,
email BOOLEAN DEFAULT false,
UNIQUE(user_id, type)
);
```
```sql
-- SQLite: 007_v0200_notification_prefs.sql
CREATE TABLE IF NOT EXISTS notification_preferences (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
type TEXT NOT NULL,
in_app INTEGER DEFAULT 1,
email INTEGER DEFAULT 0,
UNIQUE(user_id, type)
);
```
**Resolution chain:** Specific type preference → user's `*` default → system default (in_app=true, email=false).
### SMTP Configuration
Admin settings (stored in `platform_settings` JSONB, same pattern as existing admin config):
```json
{
"notifications": {
"email_enabled": false,
"smtp_host": "",
"smtp_port": 587,
"smtp_user": "",
"smtp_password": "", // encrypted via vault
"smtp_from": "noreply@chat.example.com",
"smtp_tls": true,
"instance_name": "Chat Switchboard",
"digest_enabled": false,
"digest_interval": "daily" // "hourly" | "daily" | "weekly"
}
}
```
**Implementation:** `server/notifications/email.go` — wraps `net/smtp` with TLS support. Template rendering via Go `html/template` (no external dependency). Two templates per notification type: HTML + plaintext fallback.
### Email Service
```go
// server/notifications/email.go
type EmailTransport struct {
config SMTPConfig
vault *crypto.KeyResolver // decrypt SMTP password
}
func (t *EmailTransport) Send(ctx context.Context, to, subject, htmlBody, textBody string) error
```
### Notification Service Updates
The `Service.Notify()` method gains a preference check:
```go
func (s *Service) Notify(ctx context.Context, n *models.Notification) error {
prefs := s.resolvePrefs(ctx, n.UserID, n.Type)
if prefs.InApp {
s.store.Create(ctx, n)
s.hub.SendToUser(n.UserID, ...)
}
if prefs.Email && s.email != nil {
user, _ := s.stores.GetUser(ctx, n.UserID)
if user.Email != "" {
s.email.Send(ctx, user.Email, n.Title, renderHTML(n), renderText(n))
}
}
return nil
}
```
### Digest Mode
Low-priority notifications (e.g. `grant.changed`) can batch into a periodic email digest:
- Background goroutine checks on interval (hourly/daily).
- Queries `notifications WHERE is_read = false AND created_at > last_digest_at`.
- Groups by user, renders a single digest email per user.
- Marks digested notifications with a `digested_at` timestamp (add column in this migration).
**Deferred if complexity is too high for v0.20.0.** The core value is in-app + individual email. Digest is nice-to-have.
### User Preferences UI
Settings → Notifications:
```
┌────────────────────────────────────────────────┐
│ Notification Preferences │
├────────────────────────────────────────────────┤
│ Type In-App Email │
│ ───────────────────── ─────── ───── │
│ Model fallback [✓] [ ] │
│ KB processing [✓] [✓] │
│ Group membership [✓] [ ] │
│ Default (all others) [✓] [ ] │
└────────────────────────────────────────────────┘
```
### Admin Settings UI
Admin panel → System → Notifications:
- Enable/disable email transport globally.
- SMTP configuration form (host, port, user, password, from address, TLS toggle).
- Test email button (sends to current admin's email).
- Digest toggle + interval selector.
### Files Changed (Phase 3)
**New files:**
- `server/notifications/email.go` — SMTP transport
- `server/notifications/templates.go` — HTML/text email templates
- `server/notifications/digest.go` — digest aggregation (if not deferred)
- `server/store/postgres/notification_prefs.go` — preference store
- `server/store/sqlite/notification_prefs.go` — same
- `server/database/migrations/008_v0200_notification_prefs.sql`
- `server/database/migrations/sqlite/007_v0200_notification_prefs.sql`
**Modified files:**
- `server/notifications/service.go` — preference resolution, email dispatch
- `server/store/interfaces.go` — `NotificationPreferenceStore`
- `server/handlers/notifications.go` — preference CRUD endpoints
- `server/handlers/admin.go` — SMTP config in admin settings
- `src/js/admin-handlers.js` — notification admin UI
- `src/js/app.js` — user notification preferences in settings modal
### Checklist (Phase 3)
- [x] Migration: `notification_preferences` table (both dialects)
- [x] Store: `NotificationPreferenceStore` interface + implementations
- [x] Handlers: preference CRUD endpoints (GET/PUT per type)
- [x] Notification service: preference resolution chain
- [x] Email transport: SMTP with TLS, template rendering
- [x] Email templates: HTML + plaintext for each notification type
- [x] Admin UI: SMTP config + test email button
- [x] User UI: notification preferences in Settings
- [ ] Optional: digest mode (background goroutine + batched email) — deferred
- [x] Tests: preference resolution chain
- [x] Tests: email transport (mock SMTP)
---
## Cross-cutting Concerns
### Retention / Cleanup
Notifications accumulate. Add a cleanup policy:
- Default: retain 90 days, configurable via admin settings (`notifications.retention_days`).
- Cleanup: background goroutine (daily) deletes `WHERE created_at < NOW() - retention`.
- Same pattern as compaction scanner — register in `server/main.go` startup.
### Notification Count Performance
The unread badge polls on app init, then stays current via WebSocket. No periodic polling. Badge syncs across tabs via `notification.read` event (one tab marks read → other tabs update badge).
### SQLite Considerations
- No `LISTEN/NOTIFY` — WebSocket push uses `Hub.SendToUser()` directly (already in-process, no PG dependency).
- `datetime('now')` for `created_at` (standard SQLite pattern).
- `INTEGER` for booleans (standard).
### Mobile
- Notification bell fits in mobile header (replace one of the existing icons or use hamburger menu).
- Notification dropdown becomes full-screen overlay on mobile (same pattern as other dropdowns).
- @mention autocomplete: same touch-friendly dropdown pattern as wikilink autocomplete.
### Security
- Notifications are user-scoped. No cross-user access via API (every query includes `user_id` from auth context).
- SMTP password encrypted at rest via vault (same pattern as API keys in v0.9.4).
- Email transport validates `smtp_host` against SSRF allowlist (same pattern as `url_fetch`).
- @mention parsing sanitized — no injection via display names (names are alphanumeric + hyphens only, validated on `channel_models` write).
---
## Dependency Map
```
Phase 1: Notifications Core
├── notifications table (migration)
├── NotificationStore (postgres + sqlite)
├── Notification Service (create + dispatch)
├── EventBus route (notification.new)
├── API endpoints (5)
├── Frontend: bell + dropdown + panel
└── Wire: KB, groups, resolver → service
Phase 2: @mention + Multi-model ← can start in parallel with Phase 1
├── mentions/parser.go
├── channel_models CRUD endpoints ← table already exists
├── completion handler fan-out
├── Frontend: model pills + autocomplete
└── Frontend: multi-stream rendering
Phase 3: Email + Preferences ← depends on Phase 1
├── notification_preferences table (migration)
├── SMTP transport
├── Email templates
├── Preference resolution in Service
└── Admin + User settings UI
```
Phases 1 and 2 have no code-level dependency on each other — they touch different files and can be developed in parallel or in either order. Phase 3 strictly depends on Phase 1 (extends the notification service).
---
## Version Bump
- `VERSION`: `0.19.2` → `0.20.0` after Phase 2 merge (or after Phase 1 if shipping incrementally as `0.20.0-rc1`).
- Changelog entry after each phase merge.
- No breaking API changes — all additions.
---
## What This Unblocks
- **v0.23.0 Multi-Participant Channels:** Notifications become the backbone for participant join/leave, new message alerts, and assignment queue events.
- **v0.25.0 Workflow Engine:** Stage transition notifications, assignment notifications, completion webhooks all route through the notification service.
- **v0.22.0 Smart Routing:** Multi-model channels demonstrate the fan-out pattern that routing policies will generalize.

976
docs/archive/DESIGN-0.21.0.md
View File

@@ -1,976 +0,0 @@
# Design — v0.21.x: Workspaces + Extension Surfaces
**Status:** Planning
**Depends on:** v0.20.0 (notifications), v0.17.2 (CM6), v0.11.0 (extension foundation)
**Feeds into:** v0.22.0 (smart routing), v0.23.0 (multi-participant channels)
---
## Guiding Principle
The **workspace** is a platform primitive — not an editor-mode concept. Even in
pure chat mode, uploading a zip, having the AI operate on real files at real
paths, and downloading the result as an archive is a first-class workflow. Every
surface (chat, editor, article, future modes) benefits from workspace access.
This changes the layering: workspace infrastructure lands first as foundational
storage, then surfaces consume it.
---
## Release Decomposition
```
v0.21.0 Workspace Storage (platform primitive)
│
v0.21.1 Workspace Tools + Channel/Project Binding
│
v0.21.2 Workspace Indexing + Semantic Search
│
v0.21.3 Surface Infrastructure + REPL
│
v0.21.4 Git Integration
│
v0.21.5 Editor Surface (Development Mode)
│
v0.21.6 Article Surface + Document Output Pipeline
```
---
## v0.21.0 — Workspace Storage
Pure backend. No tools, no UI, no surfaces. Just the storage primitive with
full CRUD API and archive support.
### Data Model
**`workspaces` table**
| Column | Type | Notes |
|--------|------|-------|
| `id` | UUID | PK |
| `owner_type` | TEXT | `user`, `project`, `channel`, `team` |
| `owner_id` | UUID | FK polymorphic |
| `name` | TEXT | display name |
| `root_path` | TEXT | PVC-relative path: `workspaces/{id}` |
| `max_bytes` | BIGINT | quota (NULL = system default) |
| `status` | TEXT | `active`, `archived`, `deleting` |
| `created_at` | TIMESTAMPTZ | |
| `updated_at` | TIMESTAMPTZ | |
Index: `(owner_type, owner_id)` — lookup workspace(s) for a given owner.
**`workspace_files` table** (metadata index — filesystem is source of truth)
| Column | Type | Notes |
|--------|------|-------|
| `id` | UUID | PK |
| `workspace_id` | UUID | FK → workspaces |
| `path` | TEXT | relative path from workspace root: `src/main.go` |
| `is_directory` | BOOLEAN | |
| `content_type` | TEXT | MIME type (detected or inferred) |
| `size_bytes` | BIGINT | 0 for dirs |
| `sha256` | TEXT | content hash (NULL for dirs) |
| `created_at` | TIMESTAMPTZ | |
| `updated_at` | TIMESTAMPTZ | |
Index: `UNIQUE (workspace_id, path)` — one entry per path.
Index: `(workspace_id, content_type)` — filter by type.
Design note: the DB index is a metadata cache. The PVC filesystem is the source
of truth. Syncing happens on write operations and optionally via a reconcile
endpoint. This avoids the complexity of fsnotify while keeping queries fast.
### Filesystem Layout (PVC)
```
/data/storage/
attachments/ # existing (v0.12.0)
processing/ # existing extraction queue
workspaces/ # NEW
{workspace_id}/
.workspace.json # metadata: owner, quotas, created_at
files/ # actual file tree root
src/
main.go
README.md
```
Why a separate `files/` subdirectory: keeps workspace metadata (`.workspace.json`,
future `.git/` in v0.21.4) out of the user's file namespace.
### Store Interface
```go
type WorkspaceStore interface {
// CRUD
Create(ctx context.Context, w *models.Workspace) error
GetByID(ctx context.Context, id string) (*models.Workspace, error)
Update(ctx context.Context, id string, patch models.WorkspacePatch) error
Delete(ctx context.Context, id string) error
// Ownership lookup
GetByOwner(ctx context.Context, ownerType, ownerID string) (*models.Workspace, error)
ListByOwner(ctx context.Context, ownerType, ownerID string) ([]models.Workspace, error)
// File index
UpsertFile(ctx context.Context, f *models.WorkspaceFile) error
DeleteFile(ctx context.Context, workspaceID, path string) error
DeleteFilesByPrefix(ctx context.Context, workspaceID, prefix string) error
GetFile(ctx context.Context, workspaceID, path string) (*models.WorkspaceFile, error)
ListFiles(ctx context.Context, workspaceID, prefix string, recursive bool) ([]models.WorkspaceFile, error)
// Stats
GetDiskUsage(ctx context.Context, workspaceID string) (int64, error)
}
```
### Workspace FS (Filesystem Operations)
Separate from the store — this operates on PVC, not DB.
```go
type WorkspaceFS struct {
basePath string // e.g. /data/storage/workspaces
store WorkspaceStore // for metadata sync
}
// File CRUD — all paths relative to workspace root
func (fs *WorkspaceFS) ReadFile(ctx context.Context, w *models.Workspace, path string) (io.ReadCloser, int64, error)
func (fs *WorkspaceFS) WriteFile(ctx context.Context, w *models.Workspace, path string, r io.Reader, size int64) error
func (fs *WorkspaceFS) DeleteFile(ctx context.Context, w *models.Workspace, path string) error
func (fs *WorkspaceFS) Mkdir(ctx context.Context, w *models.Workspace, path string) error
func (fs *WorkspaceFS) Stat(ctx context.Context, w *models.Workspace, path string) (*models.WorkspaceFile, error)
func (fs *WorkspaceFS) ListDir(ctx context.Context, w *models.Workspace, prefix string, recursive bool) ([]models.WorkspaceFile, error)
// Tree: returns full file listing with sizes (for file tree UI)
func (fs *WorkspaceFS) Tree(ctx context.Context, w *models.Workspace) ([]models.WorkspaceFile, error)
// Archive operations
func (fs *WorkspaceFS) ExtractArchive(ctx context.Context, w *models.Workspace, r io.Reader, format string) (int, error)
func (fs *WorkspaceFS) CreateArchive(ctx context.Context, w *models.Workspace, format string) (io.ReadCloser, error)
// Maintenance
func (fs *WorkspaceFS) Reconcile(ctx context.Context, w *models.Workspace) error // sync FS → DB index
func (fs *WorkspaceFS) Destroy(ctx context.Context, w *models.Workspace) error // rm -rf + DB cleanup
```
WriteFile updates the DB index (upsert workspace_files) after successful write.
DeleteFile removes the DB row. Reconcile walks the FS and patches any drift.
### API Endpoints
```
POST /api/v1/workspaces # create workspace
GET /api/v1/workspaces/:id # get metadata
PATCH /api/v1/workspaces/:id # update name/quota
DELETE /api/v1/workspaces/:id # delete (async cleanup)
GET /api/v1/workspaces/:id/files?path=&recursive= # list files
GET /api/v1/workspaces/:id/files/read?path= # read file content
PUT /api/v1/workspaces/:id/files/write?path= # write file (body = content)
DELETE /api/v1/workspaces/:id/files?path= # delete file/dir
POST /api/v1/workspaces/:id/files/mkdir?path= # create directory
POST /api/v1/workspaces/:id/archive/upload # upload + extract zip/tar.gz
GET /api/v1/workspaces/:id/archive/download?format= # download as zip/tar.gz
POST /api/v1/workspaces/:id/reconcile # force FS → DB sync
GET /api/v1/workspaces/:id/stats # disk usage, file count
```
### Security
- Workspace access inherits from owner: if you can access the project/channel,
you can access its workspace.
- Path traversal guard: all paths are cleaned and must resolve within the
workspace root. Symlinks are rejected.
- File size limit: configurable per-workspace `max_bytes` with system default
(`WORKSPACE_MAX_BYTES`, default 500MB).
- Archive extraction bomb protection: max files (10,000), max total extracted
size (workspace quota), max single file (100MB).
### Migrations
- `xxx_workspace_tables.sql` (Postgres + SQLite): `workspaces`, `workspace_files`
### Checklist
- [ ] `models.Workspace`, `models.WorkspaceFile`, `models.WorkspacePatch` structs
- [ ] `WorkspaceStore` interface in `store/interfaces.go`
- [ ] Postgres implementation: `store/postgres/workspace.go`
- [ ] SQLite implementation: `store/sqlite/workspace.go`
- [ ] `WorkspaceFS` in new `server/workspace/` package
- [ ] Path traversal validation (must not escape workspace root)
- [ ] Archive extract (zip, tar.gz) with bomb protection
- [ ] Archive create (zip, tar.gz)
- [ ] MIME type detection on write (`net/http.DetectContentType` + extension fallback)
- [ ] Workspace CRUD handlers (6 endpoints)
- [ ] File CRUD handlers (5 endpoints)
- [ ] Archive handlers (2 endpoints)
- [ ] Reconcile + stats handlers (2 endpoints)
- [ ] `Stores.Workspaces` wired in `store/postgres/stores.go` + `store/sqlite/stores.go`
- [ ] Integration tests: workspace CRUD, file CRUD, archive round-trip, path traversal rejection
- [ ] PVC directory creation on startup (`/data/storage/workspaces/`)
---
## v0.21.1 — Workspace Tools + Channel/Project Binding
Make workspaces useful in chat mode. AI can operate on files through tool calls.
Channels and projects get workspace bindings.
### Tools (category: `workspace`)
**`workspace_ls`** — list files in workspace
```json
{
"path": "src/",
"recursive": false
}
```
Returns: array of `{ path, is_directory, content_type, size_bytes }`.
**`workspace_read`** — read a file
```json
{
"path": "src/main.go"
}
```
Returns: file content as text (with truncation guard for large files, e.g. 50KB
soft limit with "file truncated" indicator). Binary files return a base64
summary or "binary file, N bytes" message.
**`workspace_write`** — create or overwrite a file
```json
{
"path": "src/main.go",
"content": "package main\n\nfunc main() {\n}\n"
}
```
Returns: confirmation with path and size. Creates parent directories
automatically.
**`workspace_rm`** — delete a file or directory
```json
{
"path": "src/old_module/",
"recursive": false
}
```
Returns: confirmation. Recursive required for non-empty directories.
**`workspace_mv`** — rename or move a file
```json
{
"source": "src/old.go",
"destination": "src/new.go"
}
```
**`workspace_patch`** — apply a text diff/patch to a file
```json
{
"path": "src/main.go",
"operations": [
{ "find": "oldFunction()", "replace": "newFunction()" },
{ "find": "// TODO: remove", "replace": "" }
]
}
```
The `find`/`replace` approach (same pattern as `str_replace` in many AI
coding tools) is more reliable for LLMs than unified diffs. Each operation's
`find` string must match exactly once in the file.
### Channel Workspace Binding
**Schema change:** `channels` table gains optional `workspace_id` FK.
**Behavior:**
- When a workspace is bound to a channel, workspace tools are injected into
the tool set for that channel's completions (same pattern as KB tools via
persona binding).
- New channel setting: "Workspace" toggle in channel settings.
- Creating a workspace from channel settings auto-binds it.
**Archive upload flow (chat mode):**
1. User uploads a zip/tar.gz to a channel with a workspace.
2. Upload handler detects archive MIME type + workspace binding.
3. Extracts into workspace (not attachment store).
4. Workspace tools become available for AI to operate on files.
5. User or AI can call archive download to get results.
Non-archive files still go to attachment store as before (backward compatible).
### Project Workspace Binding
**Schema change:** `projects` table gains optional `workspace_id` FK.
**Behavior:**
- Project detail panel gets a "Files" tab showing workspace file tree.
- All channels in a project inherit the project workspace (unless the channel
has its own).
- Workspace resolution: channel workspace > project workspace (same override
pattern as persona/KB resolution).
### Frontend
- Channel settings: workspace toggle, create/bind workspace.
- Project detail panel: Files tab with tree view (reusable component for
editor surface later).
- Chat bar: workspace indicator when active (similar to KB indicator).
### Checklist
- [ ] `workspace_ls` tool implementation
- [ ] `workspace_read` tool with text truncation guard + binary detection
- [ ] `workspace_write` tool with auto-mkdir
- [ ] `workspace_rm` tool with recursive guard
- [ ] `workspace_mv` tool
- [ ] `workspace_patch` tool with find/replace operations
- [ ] Tool registration in `workspace` category
- [ ] Tool injection in completion handler when workspace bound
- [ ] `channels.workspace_id` column (migration, both dialects)
- [ ] `projects.workspace_id` column (migration, both dialects)
- [ ] Archive-to-workspace upload handler (detect archive + workspace → extract)
- [ ] Workspace resolution: channel → project fallback
- [ ] Channel settings UI: workspace toggle
- [ ] Project detail panel: Files tab
- [ ] Chat bar: workspace indicator
- [ ] Integration tests: tool execution, workspace resolution, archive upload flow
- [ ] Update `ExecutionContext` with workspace ID
---
## v0.21.2 — Workspace Indexing + Semantic Search
Embed text files in workspace for semantic search. Reuses the existing
`knowledge/` chunker and embedder — they're already decoupled from KB-specific
concerns.
### Data Model
**`workspace_chunks` table**
| Column | Type | Notes |
|--------|------|-------|
| `id` | UUID | PK |
| `workspace_id` | UUID | FK → workspaces |
| `file_id` | UUID | FK → workspace_files |
| `chunk_index` | INT | ordinal within file |
| `content` | TEXT | chunk text |
| `token_count` | INT | estimated |
| `embedding` | vector(3072) / JSON | pgvector or JSON text (SQLite) |
| `metadata` | JSONB / JSON | extensible: line_start, heading, etc. |
Index: pgvector IVFFlat on `embedding` (Postgres), app-level cosine (SQLite).
Index: `(file_id)` — re-index on file change.
### Indexing Pipeline
```
file write/upload
→ is text file? (same textMIMETypes + code extensions)
→ extract content (already in memory from write, or read from FS)
→ chunk (reuse knowledge.SplitText with configurable ChunkConfig)
→ embed (reuse knowledge.Embedder)
→ store in workspace_chunks (DELETE old chunks for file_id, INSERT new)
→ update workspace_files.sha256 (content-addressed skip on no-change)
```
**Code-aware MIME types** (extend `textMIMETypes`):
The existing KB ingester recognizes `text/plain`, `text/markdown`, `text/csv`,
`text/html`. For workspace indexing, we also recognize source code by extension:
```go
var indexableExtensions = map[string]bool{
".go": true, ".py": true, ".js": true, ".ts": true, ".jsx": true,
".tsx": true, ".rs": true, ".c": true, ".cpp": true, ".h": true,
".hpp": true, ".java": true, ".rb": true, ".php": true, ".swift": true,
".kt": true, ".scala": true, ".sh": true, ".bash": true, ".zsh": true,
".sql": true, ".yaml": true, ".yml": true, ".toml": true, ".json": true,
".xml": true, ".css": true, ".scss": true, ".less": true,
".md": true, ".txt": true, ".csv": true, ".html": true, ".htm": true,
".dockerfile": true, ".tf": true, ".hcl": true, ".proto": true,
".graphql": true, ".vue": true, ".svelte": true,
".pl": true, ".pm": true, // Perl
".lua": true, ".zig": true, ".nim": true, ".ex": true, ".exs": true,
}
```
**Content-addressed skip:** if `workspace_files.sha256` matches the new write,
skip re-chunking and re-embedding. This makes bulk operations (archive extract)
efficient — only changed files get re-indexed.
**Background indexing:** file writes trigger async indexing (same goroutine
pattern as KB ingestion with semaphore). The write API returns immediately; the
`workspace_files` row gets an `index_status` column: `pending`, `indexing`,
`ready`, `error`, `skipped` (binary/non-text).
### Code-Aware Chunking
For source code, the default chunker works reasonably well but we add a
code-specific separator hierarchy:
```go
func CodeChunkConfig() ChunkConfig {
return ChunkConfig{
ChunkSize: 1500, // larger chunks for code (functions can be long)
ChunkOverlap: 200,
Separators: []string{"\n\nfunc ", "\n\nclass ", "\n\ndef ", "\n\n", "\n", " "},
}
}
```
This is a simple heuristic — it splits on function/class boundaries when
possible, falling back to paragraph and line boundaries. It doesn't require
an AST parser and handles most languages well enough for search.
### Search Tool
**`workspace_search`** — semantic search across workspace files
```json
{
"query": "database connection pooling",
"top_k": 10,
"file_pattern": "*.go"
}
```
Returns: array of `{ file_path, chunk_content, score, line_hint }`.
The `file_pattern` is optional glob filtering (applied post-search on the
`workspace_files.path` join). `line_hint` is approximate line number from
chunk metadata.
### Workspace Store Extensions
```go
// Added to WorkspaceStore interface
InsertChunks(ctx context.Context, chunks []models.WorkspaceChunk) error
DeleteChunksByFile(ctx context.Context, fileID string) error
SimilaritySearch(ctx context.Context, workspaceID string, embedding []float64, topK int) ([]models.WorkspaceChunkResult, error)
UpdateFileIndexStatus(ctx context.Context, fileID, status string) error
```
### `workspace_files` Schema Addition
| Column | Type | Notes |
|--------|------|-------|
| `index_status` | TEXT | `pending`, `indexing`, `ready`, `error`, `skipped` |
| `chunk_count` | INT | number of chunks after indexing |
### Indexing Configuration
- **Per-workspace toggle:** `workspaces.indexing_enabled` (default: `true`
when embedding role is configured, `false` otherwise).
- **System setting:** `WORKSPACE_INDEXING_ENABLED` env var (global kill switch).
- **Concurrency:** shared semaphore with KB ingestion (don't overwhelm the
embedding provider).
### Checklist
- [ ] `workspace_chunks` table (Postgres + SQLite migrations)
- [ ] `workspace_files.index_status` and `chunk_count` columns
- [ ] `WorkspaceChunk`, `WorkspaceChunkResult` model structs
- [ ] Store methods: `InsertChunks`, `DeleteChunksByFile`, `SimilaritySearch`, `UpdateFileIndexStatus`
- [ ] Postgres implementation with pgvector
- [ ] SQLite implementation with app-level cosine
- [ ] `workspace.Indexer` — reuses `knowledge.SplitText` + `knowledge.Embedder`
- [ ] Indexable type detection: MIME type + extension map
- [ ] Code-aware chunk config
- [ ] Content-addressed skip (sha256 check)
- [ ] Background indexing goroutine with semaphore
- [ ] `workspace_search` tool
- [ ] `workspace_search` glob filtering on file path
- [ ] Hook indexer into `WorkspaceFS.WriteFile` and `ExtractArchive`
- [ ] Workspace index status endpoint: `GET /api/v1/workspaces/:id/index-status`
- [ ] Integration tests: index pipeline, semantic search, content-addressed skip
- [ ] Notification on index complete/error (reuse notification service)
---
## v0.21.3 — Surface Infrastructure + REPL
Pure UI architecture. No workspace dependency. Can develop in parallel with
v0.21.0–v0.21.2 on the frontend side.
### Surface Registration API
Extends the existing `ctx.*` extension API from v0.11.0.
```js
Extensions.register({
id: 'my-mode',
init(ctx) {
ctx.surfaces.register('my-surface', {
label: 'My Mode',
icon: 'code', // lucide icon name
regions: ['surface-main', 'sidebar-content'],
activate() { /* take over regions */ },
deactivate() { /* restore regions */ }
});
}
});
```
### Region Management
```js
ctx.ui.replace(regionId, element) // saves current content, swaps in element
ctx.ui.restore(regionId) // restores saved content
```
Regions are DOM containers identified by `data-surface-region` attributes.
`replace()` detaches the current children (preserved in memory, not destroyed),
inserts the new element. `restore()` re-attaches the saved children.
This is critical for CM6 — editor instances preserve state across mode switches
because the DOM nodes aren't destroyed, just detached.
### Surface Regions
```
┌──────────────────────────────────────────────────┐
│ [surface-header] │
├──────────────┬───────────────────────────────────┤
│ │ │
│ [sidebar- │ [surface-main] │
│ content] │ (chat messages / editor / │
│ │ article / custom) │
│ │ │
│ ├───────────────────────────────────┤
│ │ [surface-footer] │
│ │ (chat input / editor toolbar) │
└──────────────┴───────────────────────────────────┘
```
The sidebar-nav region (mode selector) is managed by the surface system itself,
not by individual extensions.
### Mode Selector
Appears in the sidebar below the brand logo, above chat history, when ≥1
extension surface is registered. Shows icon buttons for each mode. Chat is
always the first (default) mode.
```
[💬] [</>] [📄] ← mode selector (chat, editor, article)
──────────────
Chat History... ← sidebar-content (replaced per mode)
```
### Events
```js
Events.on('surface.activated', ({ surface, previous }) => { ... });
Events.on('surface.deactivated', ({ surface }) => { ... });
```
### REPL / Debug Console (#70)
Fourth tab in debug modal (Ctrl+Shift+L). See roadmap for full spec:
- `AsyncFunction` wrapper for top-level `await`
- Injected globals: `API`, `Events`, `Extensions`, `DebugLog`, `Surfaces`
- Pretty-print results (collapsible JSON, red errors with stack)
- Command history: ↑/↓ with `sessionStorage`
- Tab-completion on live object graphs
- Event label hints on `Events.publish('`/`Events.on('`
- Multi-line: Shift+Enter for newline, Enter to run
- Admin-gated OR `?debug=1` URL param
### Checklist
- [ ] `data-surface-region` attributes on index.html containers
- [ ] `surfaces.js` — SurfaceRegistry: register, activate, deactivate, getCurrent
- [ ] `ctx.ui.replace()` / `ctx.ui.restore()` with DOM preservation
- [ ] `ctx.surfaces.register()` API for extensions
- [ ] Mode selector component in sidebar
- [ ] Chat as default surface (implicit, always registered)
- [ ] `surface.activated` / `surface.deactivated` EventBus events
- [ ] REPL tab: AsyncFunction wrapper, global injection
- [ ] REPL tab: pretty-print results
- [ ] REPL tab: command history (sessionStorage)
- [ ] REPL tab: tab-completion on object graphs
- [ ] REPL tab: event label hints
- [ ] REPL tab: admin gate
- [ ] REPL tab: toolbar (clear, copy, help)
- [ ] Mobile: mode selector collapses to hamburger/bottom nav
- [ ] Integration with extension loader (surfaces from manifest)
- [ ] Documentation in EXTENSIONS.md §6 update
---
## v0.21.4 — Git Integration
Layer git operations onto workspace. A workspace can optionally track a git
remote. Credentials go through the vault.
### Schema Changes
**`workspaces` table additions:**
| Column | Type | Notes |
|--------|------|-------|
| `git_remote_url` | TEXT | nullable — HTTPS or SSH URL |
| `git_branch` | TEXT | nullable — default branch after clone |
| `git_credential_id` | UUID | nullable — FK to git_credentials |
| `git_last_sync` | TIMESTAMPTZ | nullable — last pull/push |
**`git_credentials` table:**
| Column | Type | Notes |
|--------|------|-------|
| `id` | UUID | PK |
| `user_id` | UUID | FK → users (owner) |
| `name` | TEXT | display name: "GitHub PAT", "GitLab token" |
| `auth_type` | TEXT | `https_pat`, `https_basic`, `ssh_key` |
| `encrypted_data` | BYTEA | AES-256-GCM (same vault pattern as BYOK) |
| `created_at` | TIMESTAMPTZ | |
Encrypted data contains: `{ "token": "..." }` for PAT, `{ "username": "...",
"password": "..." }` for basic, `{ "private_key": "...", "passphrase": "..." }`
for SSH.
### Git Operations (server-side)
Uses `os/exec` calling `git` binary (not a Go git library — keeps binary size
down and avoids CGO). The git binary is already in the Docker image for other
purposes.
```go
type GitOps struct {
fs *WorkspaceFS
credentials CredentialResolver // vault decryption
}
func (g *GitOps) Clone(ctx context.Context, w *models.Workspace, url, branch string, credID string) error
func (g *GitOps) Pull(ctx context.Context, w *models.Workspace) error
func (g *GitOps) Push(ctx context.Context, w *models.Workspace, message string) error
func (g *GitOps) Status(ctx context.Context, w *models.Workspace) (*GitStatus, error)
func (g *GitOps) Diff(ctx context.Context, w *models.Workspace, path string) (string, error)
func (g *GitOps) Commit(ctx context.Context, w *models.Workspace, message string, paths []string) error
func (g *GitOps) Log(ctx context.Context, w *models.Workspace, n int) ([]GitLogEntry, error)
func (g *GitOps) BranchList(ctx context.Context, w *models.Workspace) ([]string, string, error)
func (g *GitOps) Checkout(ctx context.Context, w *models.Workspace, branch string) error
```
Credential injection: temporary `.git-credentials` file or `GIT_ASKPASS` script
created for the duration of the operation, then securely wiped. SSH keys use
`GIT_SSH_COMMAND` with a temp key file.
### Git Tools (category: `workspace`)
**`git_status`** — show working tree status
**`git_diff`** — show diff for file or all changes
**`git_commit`** — stage and commit files
**`git_log`** — recent commit history
**`git_branch`** — list/switch branches
These are only injected when the workspace has `git_remote_url` set.
### API Endpoints
```
POST /api/v1/workspaces/:id/git/clone # { url, branch, credential_id }
POST /api/v1/workspaces/:id/git/pull
POST /api/v1/workspaces/:id/git/push
GET /api/v1/workspaces/:id/git/status
GET /api/v1/workspaces/:id/git/diff?path=
POST /api/v1/workspaces/:id/git/commit # { message, paths }
GET /api/v1/workspaces/:id/git/log?n=
GET /api/v1/workspaces/:id/git/branches
POST /api/v1/workspaces/:id/git/checkout # { branch }
# Credential management
POST /api/v1/git-credentials # create
GET /api/v1/git-credentials # list (user-scoped)
DELETE /api/v1/git-credentials/:id # delete
```
### Post-Operation Hooks
After `clone`, `pull`, `checkout`: trigger workspace reconcile (sync FS → DB
index) and re-index changed files (v0.21.2). This ensures the file tree and
search index stay current after git operations.
### Frontend
- Workspace settings: git remote URL, credential picker, clone button.
- Git credential management in user Settings.
- Future (editor surface): git status indicators in file tree, commit UI.
### Checklist
- [ ] `git_credentials` table (Postgres + SQLite migrations)
- [ ] `workspaces` git columns (migration)
- [ ] `GitCredentialStore` interface + both implementations
- [ ] Vault integration for credential encryption/decryption
- [ ] `workspace.GitOps` with exec-based git operations
- [ ] Credential injection: temp `.git-credentials` / `GIT_SSH_COMMAND`
- [ ] Clone with depth option (shallow clone for large repos)
- [ ] Pull, push, status, diff, commit, log, branch, checkout
- [ ] Post-operation reconcile + re-index hook
- [ ] Git tool implementations (5 tools)
- [ ] Git API endpoints (9 endpoints)
- [ ] Credential CRUD endpoints (3 endpoints)
- [ ] Workspace settings UI: git config section
- [ ] User Settings: git credentials management
- [ ] Integration tests: clone, commit, push/pull cycle
- [ ] `.gitignore` respect in workspace indexing (skip ignored files)
- [ ] Security: reject `file://` and local path URLs in clone
---
## v0.21.5 — Editor Surface (Development Mode)
The IDE-like experience. Built entirely as a browser-tier extension consuming
workspace primitives from v0.21.0–v0.21.4 and surface infrastructure from
v0.21.3.
### Layout
```
┌──────────────────────────────────────────────────────┐
│ [surface-header] workspace name | branch | status │
├──────────────┬───────────────────┬───────────────────┤
│ File Tree │ Code Editor │ AI Chat Panel │
│ (sidebar) │ (CM6, main) │ (resizable) │
│ │ │ │
│ 📁 src/ │ 1│ package main │ You: refactor │
│ 📄 main.go│ 2│ │ the handler... │
│ 📄 util.go│ 3│ func main() { │ │
│ 📁 tests/ │ 4│ ... │ AI: I'll update │
│ 📄 go.mod │ 5│ } │ main.go... │
│ │ │ │
├──────────────┴───────────────────┴───────────────────┤
│ [surface-footer] file: src/main.go | Ln 4, Col 12 │
└──────────────────────────────────────────────────────┘
```
### Components
**File Tree** (sidebar-content region)
- Tree view backed by `workspace_ls` API (or cached from `Tree()`)
- File icons by extension/MIME type
- Click to open in editor
- Right-click context menu: rename, delete, new file, new folder
- Git status indicators (if git-enabled): modified, staged, untracked
- Drag-drop file reorder (within folders)
**Code Editor** (surface-main region)
- CM6 `codeEditor()` factory (already exists from v0.17.2)
- Language auto-detection from file extension
- Tab bar for multiple open files
- Modified indicator (dot on tab)
- Save: Ctrl/Cmd+S → `workspace_write` API
- Auto-save on tab switch / mode switch (configurable)
**AI Chat Panel** (split pane, resizable)
- Reuses the existing chat rendering pipeline
- Same channel context — conversation history carries over between modes
- Workspace tools auto-injected
- Tool call results update the editor live (if the modified file is open)
**Status Bar** (surface-footer region)
- Current file path
- Cursor position (line, column)
- Language mode
- Encoding
- Git branch (if applicable)
### Extension Manifest
```json
{
"id": "editor-mode",
"name": "Development",
"version": "1.0.0",
"type": "browser",
"surfaces": [{
"id": "editor",
"label": "Editor",
"icon": "code-2",
"regions": ["surface-main", "surface-footer", "sidebar-content", "surface-header"]
}],
"permissions": ["workspace"]
}
```
### Live Update on Tool Calls
When the AI calls `workspace_write` or `workspace_patch` and the target file
is open in the editor, the editor content updates live. Implementation:
1. Tool call completes → workspace write handler fires EventBus event
`workspace.file.changed` with `{ workspaceId, path }`.
2. Editor surface subscribes to this event.
3. On match (file is open), fetch fresh content from API and update CM6
state via `view.dispatch({ changes })`.
### Checklist
- [ ] `editor-mode` extension directory structure
- [ ] File tree component (tree view, icons, context menu)
- [ ] Tab bar component (open files, modified indicator)
- [ ] Split pane layout (editor + chat, resizable divider)
- [ ] CM6 editor integration with language detection
- [ ] Save flow: Ctrl/Cmd+S → API → DB index update
- [ ] Auto-save on tab/mode switch
- [ ] Live update on `workspace.file.changed` event
- [ ] Status bar: file, cursor, language, branch
- [ ] Surface registration via manifest
- [ ] Git status indicators in file tree (if v0.21.4 landed)
- [ ] Mobile: single-pane mode (editor or chat, toggle)
- [ ] Keyboard shortcuts: Ctrl+P (quick open), Ctrl+Shift+F (workspace search)
- [ ] Quick open: fuzzy file finder using `workspace_files` index
---
## v0.21.6 — Article Surface + Document Output Pipeline
Writing-focused surface. The conversation is secondary — the document is the
artifact.
### Document Output Pipeline
Before article mode, we need a way for extensions to produce downloadable
output. This is the pipeline:
```
extension generates content
→ POST /api/v1/workspaces/:id/files/write (save to workspace)
→ or POST /api/v1/workspaces/:id/export (convert + save)
→ user downloads from workspace or gets direct download link
```
Export endpoint supports format conversion (markdown → HTML, markdown → PDF via
pandoc if available, markdown → DOCX). This is a stretch goal — initial version
just saves to workspace and lets users download the raw file.
### Layout
```
┌──────────────────────────────────────────────────────┐
│ [surface-header] Document Title [Export ▾] │
├──────────────┬───────────────────────────────────────┤
│ Outline │ Rich Text Editor (CM6 markdown) │
│ │ │
│ § Intro │ # My Article │
│ § Methods │ │
│ § Data │ Compelling introduction paragraph │
│ § Analysis│ about the topic at hand... │
│ § Results │ │
│ § Discussion│ ## Methods │
│ │ │
│──────────────│ ### Data Collection │
│ AI Assistant │ │
│ │ We gathered data from... │
│ "Expand the │ │
│ methods │ │
│ section..." │ │
├──────────────┴───────────────────────────────────────┤
│ [surface-footer] Words: 1,234 | Reading: ~5 min │
└──────────────────────────────────────────────────────┘
```
### Components
**Outline** (sidebar-content, top section)
- Auto-generated from heading structure in document
- Click to scroll to section
- Drag to reorder sections
**AI Assistant** (sidebar-content, bottom section)
- Lightweight chat interface for document-focused prompts
- Tools: `workspace_read`, `workspace_write`, `workspace_search`
- Additional article tools:
- `suggest_outline` — propose structure for a topic
- `expand_section` — flesh out a heading with content
- `check_citations` — verify URLs are reachable (via url_fetch)
**Rich Text Editor** (surface-main)
- CM6 `noteEditor()` factory (from v0.17.3) with enhanced live preview
- Heading decorations, blockquotes, code blocks, links
- Focus mode: dim everything except current section
**Export** (surface-header dropdown)
- Download as Markdown (.md)
- Download as HTML (rendered)
- Download as PDF (if pandoc available)
- Copy to clipboard (markdown)
### Checklist
- [ ] `article-mode` extension directory structure
- [ ] Outline component (auto-generated from headings, click-to-scroll)
- [ ] Sidebar split: outline (top) + AI assistant (bottom)
- [ ] Rich text editor with enhanced markdown live preview
- [ ] Export dropdown: markdown, HTML, PDF
- [ ] Article-specific tools: `suggest_outline`, `expand_section`
- [ ] Focus mode (dim non-active sections)
- [ ] Word count + reading time in status bar
- [ ] Surface registration via manifest
- [ ] Auto-save document to workspace
- [ ] Mobile: outline collapses to top dropdown
---
## Cross-Cutting Concerns
### Migration Strategy
Each sub-release gets its own migration file(s), both Postgres and SQLite.
Migrations run on backend startup (existing pattern). Numbering continues
from v0.20.0's last migration.
### Test Strategy
- Backend integration tests against both Postgres and SQLite for every store
method and handler (existing CI matrix pattern).
- Workspace FS tests: use a temp directory, not the real PVC.
- Tool execution tests: mock workspace with pre-populated files.
- Surface tests: manual testing + REPL verification (surface infrastructure
is frontend-only, no backend tests).
### Backward Compatibility
- Channels without workspaces work exactly as before.
- Projects without workspaces work exactly as before.
- Archive uploads to channels without workspaces go to attachment store.
- Workspace tools only appear when a workspace is bound.
### Documentation Updates
Each sub-release updates:
- `VERSION` → `0.21.N`
- `CHANGELOG.md` — detailed entry
- `ARCHITECTURE.md` — workspace section, surface system section
- `EXTENSIONS.md` — §6 update for surface implementation
- `ROADMAP.md` — mark completed items
### Deployment
- PVC: ensure `workspaces/` directory is created at startup
(same pattern as `attachments/` directory creation).
- Git binary: verify `git` is in the Docker image (add to Dockerfile
if not present).
- Embedding role: workspace indexing gracefully degrades when no embedding
model is configured (files still work, search doesn't).
- Quota defaults: `WORKSPACE_MAX_BYTES=524288000` (500MB),
`WORKSPACE_MAX_FILES=10000`.

1153
docs/archive/DESIGN-0.24.0.md
View File

File diff suppressed because it is too large Load Diff

394
docs/archive/DESIGN-0.26.0.md
View File

@@ -1,394 +0,0 @@
# DESIGN — v0.26.0: Workflow Engine
**Status:** Shipped (v0.26.0–v0.26.5, 15 changesets)
**Branch:** `0.26.0`
**Scope:** Team-owned staged processes, AI intake, human assignment, visitor experience
**Depends on:** Dynamic surfaces (v0.25.0), multi-participant channels (v0.23.x),
anonymous identity (v0.24.3), permissions (v0.24.2)
---
## Phasing
Six sub-releases. Each is shippable and testable independently.
Deferred tech debt from v0.21–v0.25 absorbed into v0.26.0 (phase 0).
```
v0.26.0 Foundation: cleanup + context-aware tool system
│
v0.26.1 Workflow definitions + versioning (schema + CRUD)
│
v0.26.2 Workflow instances + stage transitions (runtime)
│
v0.26.3 Visitor experience (surfaces + branded chat)
│
v0.26.4 AI intake + assignment queue
│
v0.26.5 Team collaboration + workflow builder UI
```
---
## v0.26.0 — Foundation
Cleanup debt, then build the tool infrastructure that workflows need.
### Deferred Cleanup (absorb from v0.21–v0.25)
**Session cleanup job**
- Background goroutine: delete `session_participants` older than N days
with no associated messages
- Configurable via `SESSION_EXPIRY_DAYS` env var (default: 30)
- Runs on startup + every 6h
- Closes v0.24.3 deferred item
**Stale code TODOs**
- `surfaces.go:108-109` — surface delete: clean up static assets + templates
from `SURFACE_ASSET_DIR/{id}/` and `SURFACE_TEMPLATE_DIR/{id}/`
- `completion.go:164` — remove misleading TODO comment (pricing IS populated
in `recordUsage()` at line ~1996)
- `intrinsic.go:19` — remove stale "TODO: 0.9.2" comment (shipped in v0.22.0)
**Roadmap housekeeping**
- Mark shipped items in "Technical Debt + Deferred Items" section:
- Rate limit tracking (v0.22.4)
- Auto-disable policy (v0.22.4)
- Tool health recording (v0.22.4)
- PDF export via pandoc (v0.22.4)
- Persona tool grant enforcement (v0.25.0)
- Clean up stale deferred-into version markers
### Context-Aware Tool System
_Absorbed from v0.25.0 roadmap. Prerequisite for workflow tool scoping._
**`ToolContext` struct** (`tools/types.go`)
- Fields: `ChannelType`, `WorkspaceID`, `WorkflowID`, `TeamID`,
`PersonaID`, `IsVisitor`
- Populated from channel record in completion handler before tool execution
**`Require` predicates**
- `Availability() Require` method on `Tool` interface
- `BaseTool` embed: defaults to `AlwaysAvailable` (backward compat)
- Built-in predicates: `RequireWorkspace`, `RequireWorkflow`,
`RequireTeam`, `DenyVisitor`, `All()`
- `tools.AvailableFor(tctx, disabled)` replaces `AllDefinitionsFiltered()`
**Self-declaring tools**
- Workspace tools: `RequireWorkspace + DenyVisitor`
- Git tools: `RequireWorkspace + DenyVisitor`
- `workspace_create`: available only when no workspace bound
- Memory/notes tools: `DenyVisitor`
- Eliminates `WorkspaceToolNames()` / `GitToolNames()` manual suppression
**`ExecutionContext` extension**
- Add `WorkflowID`, `TeamID` to existing `ExecutionContext`
- Populated from channel record in completion handler
**Persona tool grant enforcement hardening**
- v0.25.0 shipped the second-pass allowlist in completion.go:917
- For workflows: version snapshot must include persona tool grants
at snapshot time (frozen for running instances)
### Route Namespaces
Two distinct namespaces. No collision.
**`/s/:slug` — Extension Surfaces (application extensions)**
- Single-page, self-contained sub-applications
- Examples: custom dashboard, kanban board, form builder
- One template, one JS entry point, scoped CSS
- Registered via surface manifest (v0.25.0 `surface_registry`)
**`/w/:scope/:slug` — Workflows (business logic automation)**
- `:scope` is `team-slug` or `global` — workflows belong to a team or
are org-wide. Prevents slug collisions across teams.
- Multi-page: landing, visitor chat, team review, admin tracking —
different views depending on who is accessing (visitor vs. team
member vs. admin)
- Involves customers and team members, possibly spanning multiple teams
- Purpose-built Go templates per view, not a single-page pattern
Both use Go template rendering. They share `base.html` but have
independent template trees and JS entry points.
### Migration
- 023_v0260_foundation.sql: session cleanup additions (if schema changes
needed), no new tables in this phase
---
## v0.26.1 — Workflow Definitions + Versioning
Schema and CRUD for defining workflows. No runtime yet.
### Tables
**`workflows`**
- `id`, `team_id` (nullable — NULL = global), `name`,
`slug` (unique within scope: `UNIQUE(team_id, slug)` with
partial index for global), `description`
- `branding` (JSONB: accent color, logo URL, tagline)
- `entry_mode` (enum: `public_link`, `team_only`)
- `is_active`, `version` (auto-increment on edit)
- `on_complete` (JSONB, nullable — v0.27.0 chaining hook, NULL for now.
Future schema: `{"action": "start_workflow", "target_slug": "...",
"data_map": {...}}`. Column exists early so the table doesn't need
migration when chaining lands.)
- `retention` (JSONB — `{"mode": "archive"|"delete", "delete_after_days": N}`.
Default: `{"mode": "archive"}`)
- `created_by`, `created_at`, `updated_at`
**`workflow_stages`**
- `id`, `workflow_id`, `ordinal`, `name`
- `persona_id` (FK — persona drives this stage)
- `assignment_team_id` (FK — which team handles human review)
- `form_template` (JSONB — fields the persona should collect)
- `history_mode` (enum: `full`, `summary`, `fresh` — default `full`)
- `auto_transition` (bool — advance automatically when form complete)
- `transition_rules` (JSONB — conditions, round-robin config)
**`workflow_versions`**
- `id`, `workflow_id`, `version_number`
- `snapshot` (JSONB — full serialized definition + stages + tool grants)
- `created_at`
### API
- Team-admin CRUD: create/edit/delete workflows and stages
- Reorder stages (PATCH ordinal)
- `workflow.create` permission required
- Publish action: snapshot current definition → `workflow_versions`
- Slug validation: lowercase, alphanumeric + hyphens, unique within scope
(same slug can exist under different teams). URL resolves as
`/w/team-slug/workflow-slug` or `/w/global/workflow-slug`.
### Design Decisions (to flesh out)
- Branding schema: minimal (accent + logo + tagline) or extensible JSONB?
- Stage persona: must be team-scoped persona? Or any accessible persona?
### Migration
- 023_v0261_workflows.sql: `workflows`, `workflow_stages`, `workflow_versions`
---
## v0.26.2 — Workflow Instances + Stage Transitions
Channels become workflow runtime containers.
### Channel Extensions
- `workflow_id`, `workflow_version` columns on `channels`
(populated on workflow channel creation)
- `current_stage` (ordinal), `stage_data` (JSONB)
- `workflow_status` enum: `active`, `completed`, `stale`, `cancelled`
- `last_activity_at` tracking
### Entry Point
- Public link: `/w/:scope/:slug` → creates workflow channel, adds anonymous
session participant (v0.24.3), binds stage 1 persona, auto-creates
workspace. `:scope` is team slug or `global`.
- Team-only: same flow but requires authenticated user
### Stage Transitions
- AI-triggered: `workflow_advance` tool (see v0.26.4)
- Human-triggered: button in channel header
- Rejection: return to previous stage with reason text
- On transition: swap active persona, notify assignment team,
update channel metadata, create channel-scoped note with collected data
**History mode** (per-stage, configurable in `workflow_stages`):
- `full` — new persona sees complete conversation history with a system
boundary message (same pattern as @mention context boundaries, v0.23.0).
Simplest to implement, best for continuity-sensitive workflows.
- `summary` — utility-role summarization of prior stages injected as a
system message. New persona starts with context but not raw history.
- `fresh` — new persona sees nothing from prior stages. Clean slate.
Simplest for independent review stages.
Default: `full`. Stored as `history_mode` enum on `workflow_stages`.
### Staleness Sweep
- Background goroutine (like health accumulator pattern)
- Marks idle instances as `stale` after configurable threshold
- Stale UX: visitor sees "Continue or Start Over"
(start over = new instance on latest version)
### Migration
- Adds columns to `channels`, or separate `workflow_instances` table
(TBD — inline columns simpler, separate table cleaner)
---
## v0.26.3 — Visitor Experience
Purpose-built surfaces for anonymous workflow participants.
### Landing Page
- `GET /w/:scope/:slug` — branded page with persona avatar, description, "Start"
- Go template: `workflow-landing.html`
- Reads `workflow.branding` JSONB for accent color, logo, tagline
- System dark/light mode (no theme toggle — visitors don't have prefs)
### Visitor Chat
- `GET /w/:scope/:slug/c/:channelId` — bubble chat (NOT full ChatPane)
- `workflow-chat.js`: lightweight WebSocket client
- Message send/receive
- Markdown rendering (marked.js + DOMPurify, already available)
- Typing indicator
- Stage transition animation (visual feedback on advance)
- Scoped: no sidebar, no settings, no navigation
- Session auth via `AuthOrSession` middleware (v0.24.3)
### Tool Scoping
- `DenyVisitor` predicates prevent visitors from accessing workspace,
git, memory, notes tools
- Persona tool grants further restrict per-stage
### Design Decisions (to flesh out)
- Visitor can upload files? (attachments to workflow channel)
- Visitor can see previous stage history? (probably not — each stage
is a clean persona conversation)
- Mobile-first layout for visitor surfaces
---
## v0.26.4 — AI Intake + Assignment Queue
The AI does the data collection. Humans review and act.
### `workflow_advance` Tool
- `RequireWorkflow` availability predicate
- Input: collected form data (JSONB matching `form_template`)
- Validates all required fields present
- Triggers stage transition (same path as human-triggered)
- Creates channel-scoped note with structured form response
### Form Template → System Prompt
- Stage `form_template` JSONB injected into persona system prompt:
"Collect the following information from the visitor: [field list]"
- Persona conducts conversational intake
- When all fields gathered → calls `workflow_advance` with data
- Rejection: persona sees rejection reason in conversation, re-collects
### Assignment Queue
**`workflow_assignments` table**
- `id`, `channel_id`, `stage` (ordinal), `team_id`
- `assigned_to` (nullable user_id), `status` (unassigned/claimed/completed)
- `created_at`, `claimed_at`, `completed_at`
**Claim model**
- Unassigned workflow channels visible to all team members
- Optimistic lock: `UPDATE ... WHERE status = 'unassigned'`
- Round-robin auto-assignment (optional, per-stage `transition_rules`)
**Notifications**
- New workflow assignment → notification to team (existing infra)
- WebSocket push to online team members
- Claim confirmation notification to claimer
### Migration
- `workflow_assignments` table
---
## v0.26.5 — Team Collaboration + Workflow Builder UI
Frontend surfaces for building and managing workflows.
### Team Member View
- Assigned member sees full history (AI intake + artifacts + notes)
- Persona remains active — assists both visitor and team member
- Member can: advance stage, reject (with reason), add notes,
invoke tools, reassign, escalate
### Workflow Builder
- Team admin panel section
- Stage list editor: add/remove/reorder stages
- Per-stage config: persona picker, assignment team, form template editor
- Branding editor: accent color, logo upload, tagline
- Publish button (creates version snapshot)
- Read-only tool grants display per stage persona
### Channel Header (Workflow Mode)
- Workflow stage indicator (step N of M, stage name)
- Advance / Reject buttons (permission-gated)
- Assignment info (who claimed, when)
- History mode indicator (full/summary/fresh)
### Queue UI
Prototype both approaches in v0.26.5, decide based on usage:
**Option A: Sidebar section** for team members — shows unassigned workflow
channels for user's teams, count badge, click opens in main pane.
Low friction, always visible.
**Option B: Dedicated surface** (`/workflows` or team-admin view) for
tracking and management — full table with filters, assignment history,
metrics. Better for high-volume operations.
Likely outcome: both. Sidebar for team members (day-to-day claims),
dedicated surface for team admins (tracking, reporting, bulk ops).
---
## Resolved Decisions
1. **Workflow channel lifecycle.** Configurable per-workflow retention
policy. Default: archive (read-only, `ai_mode='off'`, no new messages).
Optional: auto-delete after N days. Stored as `retention` JSONB on
`workflows` table.
2. **Multi-stage persona conversations.** Three modes, configurable
per-stage via `history_mode` enum on `workflow_stages`:
- `full` — complete history + boundary message (default, simplest)
- `summary` — utility-role summarization injected as system message
- `fresh` — clean slate, no prior context
First and third are simplest to implement; ship those first,
`summary` can land as a follow-up.
3. **Visitor re-entry.** Resume if session cookie matches and instance
is `active`. Otherwise start new on latest version.
4. **Workflow-to-workflow chaining.** Deferred to v0.27.0 (tasks/agents).
`on_complete` JSONB column on `workflows` table ships as nullable in
v0.26.1 migration so the schema is pre-wired. v0.27.0 populates it
and adds the trigger logic. No migration needed when chaining lands.
5. **Route namespaces.** `/s/:slug` for extension surfaces (single-page
sub-apps). `/w/:scope/:slug` for workflows (multi-page, role-dependent
views). No collision. See "Route Namespaces" in v0.26.0.
---
## Cross-Cutting Concerns
**Testing strategy:** Integration tests per phase. v0.26.0 tool context
tests. v0.26.1 workflow CRUD + versioning tests. v0.26.2 stage transition
tests. v0.26.4 assignment claim concurrency test.
**Migration numbering:** Single migration per phase (023, 024, ...) or
consolidated? Leaning toward: one per phase, squash at tag time if needed.
**Backward compat:** Non-workflow channels unaffected. `ToolContext` with
`BaseTool` defaults means existing tools work without changes. Workflow
columns on `channels` are nullable.

842
docs/archive/DESIGN-0.27.0.md
View File

@@ -1,842 +0,0 @@
# DESIGN — v0.27.0+: Debt Clearance, Extension Surfaces, Tasks
**Status:** Draft
**Scope:** Deferred debt clearance (v0.27.0), Tasks / Autonomous Agents (v0.27.x), TBD pull-forward (v0.28.0)
**Depends on:** Workflow Engine (v0.26.0), Dynamic Surfaces (v0.25.0), Permissions (v0.24.2)
**Current version:** 0.26.5
---
## Versioning Plan
```
v0.27.0 Debt Clearance: Extension Surface Routes + Workflow Polish
│
v0.27.1 Tasks Foundation: Service Channels + Scheduler
│
v0.27.2 Task Execution: Budgets + Admin Controls
│
v0.27.3 Task Chaining: Webhooks + Workflow-to-Workflow
│
v0.27.4 Personal Tasks: BYOK Scheduling + User Task UI
│
v0.28.0 Platform Polish (TBD pull-forward)
```
---
## Code Audit — Items Already Shipped
The ROADMAP lists these as deferred but the code proves they're done:
| Item | Evidence | Listed as deferred in |
|------|----------|----------------------|
| Persona tool grant enforcement in completion handler | `completion.go:928–949` — second-pass allowlist via `GetToolGrants()` | v0.25.0, v0.26.0 |
| Version snapshot includes persona tool grants | `workflows.go:263–273` — `stageSnapshot.ToolGrants` populated at publish | v0.25.0 |
| Session cleanup job | Shipped in v0.26.0 (background goroutine, `SESSION_EXPIRY_DAYS`) | v0.24.3 |
**Action:** Cross off all three in the ROADMAP's deferred sections and in
the v0.25.0 / v0.26.0 milestone checklists.
---
## v0.27.0 — Debt Clearance
All deferred items from v0.25.0 and v0.26.0 plus outstanding tech debt
that blocks future work. No new features — just finishing what's owed.
### Phase 1: Extension Surface Routes (`/s/:slug`)
The longest-deferred item (originally v0.21.3). Infrastructure exists:
`surface_registry` table, `InstallSurface` handler (zip upload + asset
extraction), `ListEnabledSurfaces` API. What's missing is the runtime
route mounting, template rendering, and frontend nav integration.
**Problem: Hardcoded surface conditionals in `base.html`**
Lines 61–66 and 118–122 of `base.html` use `{{if eq .Surface "chat"}}`
chains. Extension surfaces fall through to "Unknown surface". Same
pattern for CSS includes (line 25–26) and script includes (118–122).
**Solution: Generic extension surface template + dynamic blocks**
```
base.html changes:
{{if eq .Surface "chat"}}...
{{else if eq .Surface "admin"}}...
...
{{else if .Manifest}}{{template "surface-extension" .}}
{{else}}<div>Unknown surface: {{.Surface}}</div>
{{end}}
```
New template `templates/surfaces/extension.html`:
```html
{{define "surface-extension"}}
<div id="extension-surface" class="extension-surface"
data-surface-id="{{.Surface}}"
data-manifest='{{.Manifest | toJSON}}'>
{{/* Extension JS mounts into this container */}}
<div id="extension-mount"></div>
</div>
{{end}}
{{define "css-extension"}}
{{/* Extension CSS loaded dynamically from surfacesDir */}}
{{if .Manifest}}
<link rel="stylesheet" href="{{.BasePath}}/surfaces/{{.Surface}}/css/main.css?v={{.Version}}">
{{end}}
{{end}}
{{define "scripts-extension"}}
{{if .Manifest}}
<script nonce="{{.CSPNonce}}" src="{{.BasePath}}/surfaces/{{.Surface}}/js/main.js?v={{.Version}}"></script>
{{end}}
{{end}}
```
**Dynamic route registration at startup:**
```go
// pages.go — New() after registerCoreSurfaces():
func (e *Engine) loadExtensionSurfaces() {
if e.stores.Surfaces == nil {
return
}
ctx := context.Background()
surfaces, err := e.stores.Surfaces.List(ctx)
if err != nil {
log.Printf("[pages] Failed to load extension surfaces: %v", err)
return
}
for _, sr := range surfaces {
if sr.Source == "core" {
continue // Core surfaces already registered
}
route, _ := sr.Manifest["route"].(string)
if route == "" {
route = "/s/" + sr.ID // Default to /s/:id
}
manifest := SurfaceManifest{
ID: sr.ID,
Route: route,
Title: sr.Title,
Template: "surface-extension",
Auth: authFromManifest(sr.Manifest), // default "authenticated"
Layout: layoutFromManifest(sr.Manifest),
Source: "extension",
}
e.surfaces = append(e.surfaces, manifest)
}
log.Printf("[pages] Loaded %d extension surfaces from registry",
len(surfaces) - countCore(surfaces))
}
```
**nginx config — serve extension static assets:**
```nginx
# Extension surface assets
location /surfaces/ {
alias /data/surfaces/;
expires 1h;
add_header Cache-Control "public, immutable";
}
```
**Frontend nav integration:**
`ListEnabledSurfaces` API already returns route + title for all enabled
surfaces. The sidebar/nav needs to render extension surfaces as
additional items. Extension surfaces appear in a "Surfaces" section
of the sidebar, below the core nav items.
```js
// app.js or pages.js — on init:
const surfaces = await App.api.get('/api/v1/surfaces');
surfaces.filter(s => !['chat','admin','settings','editor','notes'].includes(s.id))
.forEach(s => renderExtensionNavItem(s));
```
**Extension surface manifest contract (`.surface` archive):**
```
my-dashboard.surface (zip)
├── manifest.json
├── js/
│ └── main.js ← entry point, mounts into #extension-mount
├── css/
│ └── main.css ← scoped styles
└── assets/ ← images, fonts, etc.
```
`manifest.json`:
```json
{
"id": "my-dashboard",
"title": "Dashboard",
"route": "/s/dashboard",
"auth": "authenticated",
"layout": "single",
"components": ["chat-pane"],
"hooks": ["surface"]
}
```
The extension JS receives the standard `ctx` object (same extension API
from v0.11.0) plus access to platform components:
```js
// main.js — extension surface entry point
(function() {
const mount = document.getElementById('extension-mount');
const manifest = JSON.parse(
document.getElementById('extension-surface').dataset.manifest
);
// Full access to platform primitives + components
const chatPane = window.ChatPane?.create(mount, { role: 'assist' });
// Build custom UI
mount.innerHTML = '<h1>My Dashboard</h1>';
})();
```
**Admin UI additions:**
The admin surfaces section (already exists) gains:
- Upload button (already wired to `InstallSurface`)
- Enable/disable toggles (already wired)
- Uninstall button (already wired to `DeleteSurface`)
- Route display showing the `/s/:slug` URL
**Deliverables:**
- [ ] `surface-extension` template (HTML + CSS + scripts blocks)
- [ ] `base.html` conditional chain extended with `{{if .Manifest}}` fallthrough
- [ ] `loadExtensionSurfaces()` in page engine
- [ ] nginx location block for `/surfaces/` static assets
- [ ] Frontend nav renders extension surfaces from `ListEnabledSurfaces`
- [ ] Admin surfaces section shows upload/enable/disable/uninstall
- [ ] Sample `.surface` archive (hello-world dashboard) for testing
- [ ] CSP nonce propagation for extension scripts
### Phase 2: Workflow Engine Polish
Eight items deferred from v0.26.0. All backend infrastructure exists;
these are wiring, UI, and enforcement.
**Channel header stage indicator + advance/reject controls**
- Workflow channels show: stage name, step N of M, assignment info
- Advance / Reject buttons (permission-gated by `workflow.manage`)
- Renders in the chat header area (same slot as `ai_mode` context banner)
**Stage persona auto-switch in chat UI**
- On stage transition, chat UI updates the active persona display
- Model selector reflects the stage persona's bound model
- System prompt injection already works (v0.26.4); this is FE-only
**Assignment notifications via WebSocket**
- New workflow assignment → `workflow.assigned` WS event to team members
- Claim confirmation → `workflow.claimed` WS event to claimer
- Uses existing notification infrastructure (v0.20.0)
**Round-robin auto-assignment**
- Per-stage `transition_rules.auto_assign: "round_robin"` in `workflow_stages`
- On stage entry: query team members, pick least-recently-assigned
- `workflow_assignments.assigned_to` populated automatically
- Falls back to unassigned pool if round-robin fails
**`on_complete` workflow chaining**
- Column exists on `workflows` table (nullable JSONB, v0.26.1)
- Schema: `{"action": "start_workflow", "target_slug": "...", "data_map": {...}}`
- On workflow completion: if `on_complete` is non-null, start target workflow
with mapped `stage_data` from completed instance
- Reuses existing `StartInstance()` path
**Workflow retention enforcement**
- Column exists on `workflows` table (`retention` JSONB, v0.26.1)
- Background sweep (extend staleness goroutine): completed instances
older than `retention.delete_after_days` → hard delete
- `retention.mode = "archive"` → set `workflow_status = 'archived'`,
`ai_mode = 'off'` (already implemented for channel archive)
**Drag-and-drop stage reorder in builder**
- Backend `PATCH /api/v1/workflows/:id/stages/reorder` already exists
- Frontend: HTML5 drag events on stage list items in workflow builder
- Same DnD pattern as channel/folder reorder (v0.23.1)
**Team-scoped workflow management UI**
- Team admin settings surface gains "Workflows" section
- Lists workflows owned by the team, quick-edit name/description
- Links to full builder for stage editing
- Mirrors the admin-level builder but scoped to `team_id`
**Deliverables:**
- [ ] Channel header workflow stage indicator component
- [ ] Advance/reject buttons with permission gate
- [ ] Stage persona auto-switch in chat UI
- [ ] `workflow.assigned` / `workflow.claimed` WS events
- [ ] Round-robin auto-assignment in stage transition handler
- [ ] `on_complete` chaining trigger in workflow completion path
- [ ] Retention enforcement in staleness sweep goroutine
- [ ] Drag-and-drop stage reorder in workflow builder UI
- [ ] Team settings → Workflows section
### Phase 3: Workspace + Editor Debt
Items deferred from v0.21.x that are low-hanging fruit.
- [ ] `.gitignore` respect in workspace indexing (`workspace/indexer.go` — skip paths matching patterns from `.gitignore` in workspace root)
- [ ] Workspace settings UI: git config section in channel/project settings (remote URL, branch, auto-commit toggle)
- [ ] Pane state persistence per-user/per-project (save pane layout to `user_settings` JSONB, restore on surface load)
- [ ] Editor drag-drop file reorder (workspace file tree — reorder via DnD, persist order in workspace metadata)
**Explicitly deferred (not v0.27.0):**
- User settings git credentials management UI — needs vault-encrypted git credential store (v0.28.0 candidate)
- Integration tests: clone/commit/push/pull — needs git binary in CI container
- Article-specific AI tools — needs article surface rebuild
- Mobile hamburger nav — needs responsive audit pass
**Deliverables:**
- [ ] `.gitignore` filter in `workspace/indexer.go`
- [ ] Git config section in workspace/project settings UI
- [ ] Pane layout persistence in `user_settings`
- [ ] Editor file tree DnD reorder
### ROADMAP Cleanup
Update the ROADMAP itself:
- [ ] Cross off tool grant enforcement (3 locations)
- [ ] Move "Deferred to v0.27.0" items under their proper v0.27.0 section
- [ ] Remove duplicate entries (extension surface routes listed 4× across the file)
- [ ] Add v0.27.0 section with phase structure
- [ ] Update dependency graph
---
## v0.27.1 — Tasks Foundation: Service Channels + Scheduler
The core primitive: a channel with no human participant, driven by
a scheduler.
### Service Channels
`type: 'service'` — a new channel type. Service channels:
- Have zero human participants (only persona participants)
- Cannot be joined by users (read-only view for authorized users)
- Are created by the scheduler or the `task_create` tool
- Inherit the workflow engine's stage model (optional — simple tasks
skip stages entirely)
- Appear in a "Tasks" sidebar section (collapsed by default)
```sql
-- No new migration needed — channel type CHECK already allows extension.
-- Add 'service' to the channel type CHECK constraint:
ALTER TABLE channels DROP CONSTRAINT IF EXISTS channels_type_check;
ALTER TABLE channels ADD CONSTRAINT channels_type_check
CHECK (type IN ('direct','dm','group','channel','workflow','service'));
```
### Task Definitions
A task is a lightweight scheduled job that creates a service channel
and runs a completion (or instantiates a workflow).
**`tasks` table:**
```sql
CREATE TABLE tasks (
id TEXT PRIMARY KEY DEFAULT gen_random_uuid()::text,
owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
team_id TEXT REFERENCES teams(id) ON DELETE SET NULL,
name TEXT NOT NULL,
description TEXT,
scope TEXT NOT NULL DEFAULT 'personal'
CHECK (scope IN ('personal', 'team', 'global')),
-- What to run
task_type TEXT NOT NULL DEFAULT 'prompt'
CHECK (task_type IN ('prompt', 'workflow')),
persona_id TEXT REFERENCES personas(id) ON DELETE SET NULL,
model_id TEXT, -- explicit model override (nullable)
system_prompt TEXT, -- additional system prompt (prepended)
user_prompt TEXT, -- the message to send
workflow_id TEXT REFERENCES workflows(id) ON DELETE SET NULL,
tool_grants JSONB, -- explicit tool allowlist (nullable = inherit all)
-- Schedule
schedule TEXT NOT NULL, -- cron expression (5-field)
timezone TEXT NOT NULL DEFAULT 'UTC',
is_active BOOLEAN NOT NULL DEFAULT true,
-- Execution policy
max_tokens INTEGER NOT NULL DEFAULT 4096,
max_tool_calls INTEGER NOT NULL DEFAULT 10,
max_wall_clock INTEGER NOT NULL DEFAULT 300, -- seconds
output_mode TEXT NOT NULL DEFAULT 'channel'
CHECK (output_mode IN ('channel', 'note', 'webhook')),
output_channel_id TEXT REFERENCES channels(id) ON DELETE SET NULL,
webhook_url TEXT,
-- Provider routing
provider_config_id TEXT REFERENCES provider_configs(id) ON DELETE SET NULL,
-- Notifications
notify_on_complete BOOLEAN NOT NULL DEFAULT false,
notify_on_failure BOOLEAN NOT NULL DEFAULT true,
-- Bookkeeping
last_run_at TIMESTAMPTZ,
next_run_at TIMESTAMPTZ,
run_count INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_tasks_next_run ON tasks (next_run_at) WHERE is_active = true;
CREATE INDEX idx_tasks_owner ON tasks (owner_id);
```
**Two task types:**
1. **Prompt task** (`task_type = 'prompt'`): Create a service channel,
send `user_prompt` with `system_prompt` prepended, collect response.
Simple, single-turn. Good for: news digest, stock screener, daily
standup prep, report generation.
2. **Workflow task** (`task_type = 'workflow'`): Instantiate a workflow
in a service channel. Multi-stage, tool-using, potentially long-running.
Good for: data pipeline, automated review, research agent.
### Scheduler
Background goroutine (same pattern as compaction scanner, staleness sweep):
```go
type TaskScheduler struct {
stores store.Stores
interval time.Duration // poll interval: 30s
stop chan struct{}
}
func (s *TaskScheduler) Run() {
ticker := time.NewTicker(s.interval)
for {
select {
case <-ticker.C:
s.poll()
case <-s.stop:
return
}
}
}
func (s *TaskScheduler) poll() {
// SELECT * FROM tasks WHERE is_active AND next_run_at <= now()
// ORDER BY next_run_at LIMIT 10
due, _ := s.stores.Tasks.ListDue(ctx)
for _, task := range due {
go s.execute(task)
}
}
```
Cron parsing: use `github.com/robfig/cron/v3` for 5-field expressions.
Compute `next_run_at` after each execution.
### Provider Resolution for Tasks
Task execution needs a provider. Resolution order:
1. Explicit `provider_config_id` on task (user chose a specific provider)
2. Personal BYOK provider (if `scope = 'personal'` and user has BYOK keys)
3. Team provider (if `scope = 'team'`)
4. Global provider (fallback)
5. Routing policy (if task's model matches a routing policy)
This is the same resolution chain as completions, reusing the existing
`resolveProvider()` path in `completion.go`.
### Migration
- 026_tasks.sql: `tasks` table, service channel type extension
**Deliverables:**
- [ ] `tasks` table + store interface + PG/SQLite implementations
- [ ] `type: 'service'` channel type
- [ ] `TaskScheduler` background goroutine
- [ ] Cron expression parsing + `next_run_at` computation
- [ ] Task execution path (create service channel, run completion)
- [ ] Provider resolution for task context
- [ ] Tasks sidebar section (read-only view of service channels)
---
## v0.27.2 — Task Execution: Budgets + Admin Controls
### Execution Budgets
Three limits enforced in the task execution path:
1. **`max_tokens`** — total output tokens. Tracked via existing
`usage_log` infrastructure. Execution halts mid-stream if exceeded.
2. **`max_tool_calls`** — total tool invocations per run. Counter
incremented in `streamWithToolLoop`. Execution halts on breach.
3. **`max_wall_clock`** — seconds. `context.WithTimeout` on the
execution goroutine. Hard kill on timeout.
Budget breach → task marked as `budget_exceeded` in run history,
notification to owner.
### Task Run History
**`task_runs` table:**
```sql
CREATE TABLE task_runs (
id TEXT PRIMARY KEY DEFAULT gen_random_uuid()::text,
task_id TEXT NOT NULL REFERENCES tasks(id) ON DELETE CASCADE,
channel_id TEXT REFERENCES channels(id) ON DELETE SET NULL,
status TEXT NOT NULL DEFAULT 'running'
CHECK (status IN ('running','completed','failed',
'budget_exceeded','cancelled')),
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
completed_at TIMESTAMPTZ,
tokens_used INTEGER DEFAULT 0,
tool_calls INTEGER DEFAULT 0,
wall_clock INTEGER DEFAULT 0, -- seconds
error TEXT
);
```
### Admin Controls
**Global config keys:**
- `tasks.enabled` — master kill switch (default: true)
- `tasks.allow_personal` — whether non-admin users can create personal
tasks (default: false). Same pattern as `personas.allow_personal`.
- `tasks.max_concurrent` — max simultaneous task executions (default: 5)
- `tasks.default_budget.max_tokens` — default ceiling (overridable per task)
- `tasks.default_budget.max_tool_calls` — default ceiling
- `tasks.default_budget.max_wall_clock` — default ceiling (seconds)
**Admin panel — Tasks section:**
- List all tasks (filterable by owner, team, scope, status)
- View task run history with budget usage
- Pause/resume individual tasks
- Kill running executions
- Edit default budgets
- Toggle `tasks.allow_personal`
**Permission integration:**
- New permission: `tasks.create` — required to create tasks
- New permission: `tasks.admin` — required to manage others' tasks
- `Everyone` group gets `tasks.create` if `tasks.allow_personal` is true
**Deliverables:**
- [ ] Execution budget enforcement (tokens, tool calls, wall clock)
- [ ] `task_runs` table + store
- [ ] Budget breach notification to task owner
- [ ] Global config keys for task admin controls
- [ ] Admin panel Tasks section (CRUD, history, kill switch)
- [ ] `tasks.create` and `tasks.admin` permissions
- [ ] `tasks.allow_personal` toggle (mirrors `personas.allow_personal`)
---
## v0.27.3 — Task Chaining: Webhooks + Workflow-to-Workflow
### Completion Webhooks
When a task (or workflow) completes, optionally POST to an external URL:
```go
type WebhookPayload struct {
TaskID string `json:"task_id"`
TaskName string `json:"task_name"`
Status string `json:"status"`
ChannelID string `json:"channel_id"`
CompletedAt time.Time `json:"completed_at"`
Output string `json:"output"` // last assistant message
StageData any `json:"stage_data"` // workflow stage data (if applicable)
}
```
- Webhook URL on task or workflow (`webhook_url` column)
- Retry: 3 attempts with exponential backoff (1s, 5s, 25s)
- Timeout: 10s per attempt
- HMAC signature header (`X-Switchboard-Signature`) using a per-task secret
### `task_create` Tool
AI-invocable tool that spawns sub-tasks:
```json
{
"name": "task_create",
"description": "Create a new scheduled or one-shot task",
"parameters": {
"name": "string",
"prompt": "string",
"schedule": "string (cron or 'once')",
"persona": "string (handle, optional)",
"max_tokens": "integer (optional)"
}
}
```
- `RequireWorkflow` or `RequireTeam` predicate (not available in
personal chats — prevents runaway task creation)
- One-shot tasks: `schedule = 'once'`, `next_run_at = now()`
- Created tasks inherit the parent's scope (team/global)
- Depth limit: tasks cannot create tasks (no recursive spawning)
### Workflow-to-Workflow Chaining
Wires the `on_complete` column (v0.26.1) into the task system:
- On workflow completion with `on_complete` set: create a one-shot task
that instantiates the target workflow with mapped data
- Data mapping: `data_map` keys in `on_complete` JSONB map source
`stage_data` fields to target workflow's initial `stage_data`
**Deliverables:**
- [ ] Webhook delivery with retry + HMAC signature
- [ ] `task_create` tool with depth limit
- [ ] `on_complete` chaining via task system
- [ ] Webhook secret generation per task/workflow
---
## v0.27.4 — Personal Tasks: BYOK Scheduling + User Task UI
The user-facing task experience. Users create personal tasks that run
against their BYOK providers.
### Use Cases
- **Morning news digest:** Daily at 6am, prompt "Summarize top tech news",
output to a personal service channel. Persona: "News Analyst" with
`web_search` tool grant.
- **Stock screener:** Weekdays at market open, prompt "Check my watchlist
for unusual pre-market activity", BYOK OpenAI key for cost control.
- **Weekly project summary:** Friday at 5pm, prompt "Summarize this
week's activity across my projects", uses `conversation_search` +
`workspace_search` tools.
- **Daily standup prep:** Weekday mornings, reviews yesterday's notes
and generates standup talking points.
### Settings Surface — Tasks Section
New section in user settings (same pattern as BYOK, Personas):
- Task list: name, schedule (human-readable), last run status, next run
- Create task: name, persona picker, model picker, prompt editor,
schedule builder (preset crons + custom), budget overrides
- Task detail: run history, output channel link, edit, pause/delete
- BYOK indicator: shows which provider the task will use
### Schedule Builder
Preset schedules + custom cron for power users:
| Preset | Cron |
|--------|------|
| Every morning (6am) | `0 6 * * *` |
| Weekday mornings (8am) | `0 8 * * 1-5` |
| Every hour | `0 * * * *` |
| Weekly (Monday 9am) | `0 9 * * 1` |
| Monthly (1st at midnight) | `0 0 1 * *` |
| Custom... | user-entered 5-field cron |
Timezone selector defaults to browser timezone.
### BYOK Task Routing
Personal tasks prefer the user's BYOK provider:
1. If task has explicit `provider_config_id` → use it
2. If user has a BYOK provider config for the task's model → use BYOK
3. Fall through to team/global provider (if allowed by admin)
Admin can restrict personal tasks to BYOK-only via
`tasks.personal_require_byok` config key (default: false). When true,
personal tasks without a BYOK provider fail with a clear error instead
of falling through to org providers.
### Output Modes
Three output destinations:
1. **Channel** (default): Output goes to a persistent service channel.
User views it like a read-only chat. Channel accumulates run outputs
over time (scrollable history).
2. **Note**: Output saved as a channel-scoped note (good for structured
data, reports). Note title includes timestamp.
3. **Webhook**: Output POSTed to external URL (for integration with
other tools — Slack, email, etc.).
### Task Sidebar Section
New sidebar section (below Channels, above Chats):
```
▾ Tasks (collapsible)
◷ Morning News Digest ✓ 6:02am
◷ Stock Screener ✓ 9:30am
◷ Weekly Summary ⏳ Fri 5pm
+ New task
```
Click opens the task's output channel. Status indicators: ✓ (last run
succeeded), ✗ (failed), ⏳ (next run time), ▶ (running now).
**Deliverables:**
- [ ] Settings → Tasks section (CRUD, schedule builder, budget config)
- [ ] BYOK provider routing for personal tasks
- [ ] `tasks.personal_require_byok` config key
- [ ] Output modes: channel, note, webhook
- [ ] Tasks sidebar section with status indicators
- [ ] "Run Now" button for manual trigger
- [ ] Task output channel read-only view
---
## v0.28.0 — Platform Polish (TBD Pull-Forward)
Candidates pulled from the TBD section, prioritized by value and
dependency readiness.
### Tier 1 — High value, dependencies met
**Virtual scroll for long conversations**
Conversations with 500+ messages bog down the DOM. Virtual scroll renders
only visible messages + buffer zone. Prerequisite for heavy task output
channels.
**KB auto-injection (context-aware)**
Top-K chunk prepend to system prompt, context budget aware, per-channel
toggle. Useful for tasks that need domain knowledge without explicit
`kb_search` tool calls. Latency budgeting required (embedding lookup
adds ~200ms).
**Helm chart**
Raw k8s manifests with `${VAR}` substitution → proper Helm chart.
`values.yaml` for replicas, image tags, ingress, storage, secrets,
feature flags. Subchart for dev/test Postgres. Target: `helm install
switchboard ./chart`. Now makes sense because the feature set is
stabilizing post-tasks.
**Per-provider model preferences**
`user_model_settings` unique key is `(user_id, model_id)` — same model
from different providers shares one visibility toggle. Needs
`provider_config_id` dimension in DB constraint, store, API, and
frontend `hiddenModels` keying. Natural fit now that BYOK tasks need
per-provider model selection.
### Tier 2 — Medium value, some design work needed
**Memory compaction**
Summarize old memories to save context tokens. Confidence decay: reduce
confidence over time, prune low-confidence entries. Natural extension of
the memory system (v0.18.0).
**`capability_match` routing policy**
"Cheapest model with tool_calling" — a new routing policy type. Useful
for tasks that need tool use but don't need the strongest model.
**New provider types via config file**
OpenAI-compatible endpoints registrable via YAML config (no Go code).
Enables users to point at local LLMs (Ollama, vLLM, llama.cpp) without
a provider code change.
### Tier 3 — Future (v0.29+)
- Desktop app (Tauri) — large scope, independent track
- Full PWA with offline — needs service worker rewrite
- Plugin/extension marketplace — needs extension surface routes first (v0.27.0)
- Starlark/sidecar extension tiers — needs extension surface routes first
- Git credentials vault store — needs vault extension design
- Cross-persona memory sharing — needs memory system redesign
---
## Design Decision: Task Permission Model
Tasks introduce a new resource type that intersects with multiple
existing permission boundaries:
| Scope | Who can create | Provider used | Visibility |
|-------|---------------|---------------|------------|
| Personal | User (if `tasks.allow_personal`) | User's BYOK or team fallback | Owner only |
| Team | User with `tasks.create` + team membership | Team provider | Team members |
| Global | Admin | Global provider | All users (read) |
Personal tasks are the BYOK sweet spot. The admin toggle
(`tasks.allow_personal`) mirrors the existing `personas.allow_personal`
pattern — a single boolean in global config, surfaced in the admin
settings panel.
**Why not just a permission?** The `tasks.create` permission already
gates the ability. `tasks.allow_personal` is a _policy_ control —
admins might allow task creation for team tasks but prohibit personal
tasks to prevent uncontrolled BYOK spending. Two orthogonal knobs.
---
## Design Decision: Task vs. Workflow Relationship
Tasks and workflows are related but distinct:
| | Task | Workflow |
|---|------|---------|
| Trigger | Cron schedule or `task_create` tool | Human click or task trigger |
| Participants | Zero humans (service channel) | Humans + AI |
| Complexity | Single prompt or workflow instantiation | Multi-stage, assignment queue |
| Output | Channel, note, or webhook | Channel (interactive) |
| Lifecycle | Recurring (cron) or one-shot | Single instance |
A **prompt task** is simpler than a workflow — it's a scheduled
completion. A **workflow task** is a task that instantiates a workflow.
This keeps the task system lean (scheduling + budgets + output routing)
and the workflow engine rich (stages + assignment + personas).
The `task_type` column makes this explicit. Prompt tasks don't need
workflows at all — they're a direct scheduler → completion path.
Workflow tasks reuse the full workflow engine.
---
## Resolved Decisions
1. **Task output retention.** Use channel-level retention from v0.26.0.
No per-task retention policy needed — the channel's `retention` JSONB
handles archive/delete semantics. Notes created by task output survive
channel removal (notes are channel-scoped but not cascade-deleted).
Storage quotas per user are a future concern (v0.29+ candidate) — not
blocking for initial task support.
2. **Concurrent task execution.** Skip with a warning log. If
`task_runs` has a row with `status = 'running'` for the task, the
scheduler skips that tick. Log: `[scheduler] Skipping task %s — previous
run still active`. No queue — if a task consistently overruns its
schedule, the user needs to adjust the cron interval or budget.
3. **Task templates.** Ship 3–5 optional starter templates in v0.27.4.
Presented as suggestions during task creation ("Start from a
template..." option alongside blank task). Not auto-created — user
must explicitly choose one. Templates are just pre-filled form values,
not persistent DB records.
4. **Task notifications.** Opt-in per task. Two booleans on the `tasks`
table: `notify_on_complete` (default false), `notify_on_failure`
(default true). Uses existing notification infrastructure (v0.20.0
bell + WebSocket push).

810
docs/archive/DESIGN-0.32.0.md
View File

@@ -1,810 +0,0 @@
# DESIGN-0.32.0 — Multi-Replica HA
Run 2–3 backend replicas across nodes for node-level availability.
Five changesets. No sub-versions — the work is tightly coupled and
each CS builds on the previous.
Depends on: v0.31.2 (current HEAD).
**Design decision:** PG `SKIP LOCKED` replaces Kubernetes Lease-based
leader election for the task scheduler. Every replica polls; PG
serializes the claims. Extends to all shared mutable state — tickets,
rate counters — keeping PG as the sole coordination layer. No Redis,
no K8s API dependency, no new infrastructure.
---
## What Already Works Multi-Replica
These require zero changes:
- **REST API** — stateless, JWT auth, any replica serves any request.
- **PG + S3 + CephFS** — shared storage infrastructure.
- **`pg_broadcast` LISTEN/NOTIFY** — cross-pod event bus. `Publish()`
→ `broadcastHook` → `pg_notify` → remote pod `publishLocal()` →
local WS subscribers. Fully wired, just never tested at N>1.
---
## What Needs Work
Five areas of in-process mutable state that break at replica count > 1:
| State | Current | Problem at N>1 | Fix |
|-------|---------|----------------|-----|
| Task scheduler | Single goroutine polls `ListDue` | All replicas fire same tasks | `SKIP LOCKED` atomic claim |
| Task run guard | `GetActiveRun` check → `CreateRun` | TOCTOU race window | Conditional `INSERT ... WHERE NOT EXISTS` |
| WS ticket store | `sync.Map` per-pod | Ticket from pod-1 invalid on pod-2 | PG table with TTL reaper |
| Rate limiter | In-memory token bucket per-pod | Effective limit = N × configured | PG counter with time bucket |
| `SendToUser` | Local hub lookup only | User on pod-2 never reached | Route through bus → `pg_broadcast` |
---
## CS0 — Schema: `020_ha.sql`
New migration for both PG and SQLite. No changes to existing 019
task schema — `next_run_at` nullable already supports the claim
mechanism.
### PG: `server/database/migrations/020_ha.sql`
```sql
-- ==========================================
-- Chat Switchboard — 020 Multi-Replica HA
-- ==========================================
-- Shared state tables for multi-replica operation.
-- v0.32.0: ws_tickets, rate_limit_counters.
-- ==========================================
-- =========================================
-- WEBSOCKET TICKETS (replaces sync.Map)
-- =========================================
CREATE TABLE IF NOT EXISTS ws_tickets (
id TEXT PRIMARY KEY, -- 128-bit hex token
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
ON ws_tickets (expires_at);
-- =========================================
-- RATE LIMIT COUNTERS (replaces in-memory)
-- =========================================
-- Token bucket approximation using time-windowed counters.
-- Key format: "{scope}:{identifier}" e.g. "auth:192.168.1.1"
-- Window is truncated to the second for the configured rate.
CREATE TABLE IF NOT EXISTS rate_limit_counters (
key TEXT NOT NULL,
window TIMESTAMPTZ NOT NULL, -- truncated timestamp (window start)
tokens REAL NOT NULL DEFAULT 0, -- tokens consumed in this window
PRIMARY KEY (key, window)
);
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
ON rate_limit_counters (window);
```
### SQLite: `server/database/migrations/sqlite/020_ha.sql`
SQLite parity — structurally identical but with dialect adjustments.
These tables are functional in SQLite for single-process test parity,
though multi-replica is PG-only in production.
```sql
CREATE TABLE IF NOT EXISTS ws_tickets (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
ON ws_tickets (expires_at);
CREATE TABLE IF NOT EXISTS rate_limit_counters (
key TEXT NOT NULL,
window TEXT NOT NULL,
tokens REAL NOT NULL DEFAULT 0,
PRIMARY KEY (key, window)
);
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
ON rate_limit_counters (window);
```
### Deliverables
- [x] `020_ha.sql` (PG)
- [x] `020_ha.sql` (SQLite)
- [x] CI green on both pipelines (migration runs, tables exist)
---
## CS1 — Task Scheduler: SKIP LOCKED
Replace the read-then-execute scheduler with atomic claim semantics.
Every replica runs the poll loop. PG serializes task handoff.
### 1.1 — Atomic `ClaimDueTask` (replaces `ListDue`)
New store method. Single atomic statement — SELECT + UPDATE in one
round trip. Returns at most one task per call.
**PG implementation:**
```sql
UPDATE tasks
SET next_run_at = NULL
WHERE id = (
SELECT id FROM tasks
WHERE is_active = true
AND next_run_at <= NOW()
ORDER BY next_run_at ASC
LIMIT 1
FOR UPDATE SKIP LOCKED
)
RETURNING <taskColumns>
```
Mechanics:
- `FOR UPDATE SKIP LOCKED` — if another replica holds a lock on a
candidate row, skip it instantly (no wait, no deadlock).
- `SET next_run_at = NULL` — claimed task disappears from future
polls. `advanceNextRun` restores it after execution.
- Returns zero rows if nothing is due → `sql.ErrNoRows` → no-op.
- The existing partial index `idx_tasks_next_run` covers the WHERE
clause (`is_active = true AND next_run_at IS NOT NULL` by
implication — NULL rows won't satisfy `<= NOW()`).
**SQLite implementation:**
SQLite has no `SKIP LOCKED`. Single-process, so not needed. Keep
the current `ListDue` behavior unchanged for SQLite. The store
interface accommodates both:
```go
// TaskStore additions
ClaimDueTask(ctx context.Context) (*models.Task, error)
```
SQLite `ClaimDueTask` just does:
```sql
SELECT <taskColumns> FROM tasks
WHERE is_active = 1 AND next_run_at <= datetime('now')
ORDER BY next_run_at ASC LIMIT 1
```
Then sets `next_run_at = NULL` in a second statement (single writer,
no contention).
### 1.2 — Conditional `CreateRunExclusive`
Belt-and-suspenders: prevent double-execution even if two replicas
somehow both claim the same task (shouldn't happen with SKIP LOCKED,
but defense in depth).
```sql
INSERT INTO task_runs (task_id, status)
SELECT $1, 'running'
WHERE NOT EXISTS (
SELECT 1 FROM task_runs
WHERE task_id = $1 AND status IN ('running', 'queued')
)
RETURNING id, started_at
```
Returns `sql.ErrNoRows` if a run already exists → skip execution.
New store method:
```go
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
```
### 1.3 — Scheduler Loop Rewrite
`scheduler.go` changes:
```go
func (s *Scheduler) poll() {
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
cfg := taskutil.LoadTaskConfig(ctx, s.stores.GlobalConfig)
if !cfg.Enabled {
return
}
// Claim tasks one at a time until none remain or budget exhausted.
// Each replica claims independently — PG SKIP LOCKED serializes.
claimed := 0
for claimed < cfg.MaxConcurrent {
task, err := s.stores.Tasks.ClaimDueTask(ctx)
if err != nil {
break // sql.ErrNoRows or real error — either way, done
}
claimed++
go s.execute(ctx, task)
}
}
```
`execute()` changes:
- Remove the `GetActiveRun` TOCTOU check — replaced by
`CreateRunExclusive`.
- `GetQueuedRun` stays — webhook-triggered runs still need adoption.
But adoption also uses `CreateRunExclusive` semantics (transition
only if still `queued`).
- `advanceNextRun` still computes and sets `next_run_at`, making the
task visible to future polls again.
### 1.4 — Remove `ListDue` from Interface
`ListDue` removed from `TaskStore` interface. Replaced by
`ClaimDueTask`. `CreateRun` remains for webhook trigger path;
`CreateRunExclusive` added alongside it.
Updated interface:
```go
type TaskStore interface {
// ... existing CRUD ...
// Scheduler queries
ClaimDueTask(ctx context.Context) (*models.Task, error)
SetNextRun(ctx context.Context, id string, nextRun interface{}) error
SetLastRun(ctx context.Context, id string) error
IncrementRunCount(ctx context.Context, id string) error
// Run history
CreateRun(ctx context.Context, r *models.TaskRun) error
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
UpdateRun(ctx context.Context, id string, status string, ...) error
TransitionRunStatus(ctx context.Context, id string, status string) error
GetActiveRun(ctx context.Context, taskID string) (*models.TaskRun, error)
GetQueuedRun(ctx context.Context, taskID string) (*models.TaskRun, error)
ListRuns(ctx context.Context, taskID string, limit int) ([]models.TaskRun, error)
}
```
### Deliverables
- [ ] `ClaimDueTask` — PG (`FOR UPDATE SKIP LOCKED`) + SQLite (simple select)
- [ ] `CreateRunExclusive` — PG + SQLite
- [ ] Scheduler loop rewrite — claim-per-iteration, no TOCTOU
- [ ] Remove `ListDue` from interface + both stores
- [ ] Unit tests: concurrent claim (PG only — two goroutines, verify
disjoint task sets)
- [ ] CI green
---
## CS2 — PG Ticket Store
Replace `events.TicketStore` (in-memory `sync.Map`) with a PG-backed
implementation. The middleware already programs to the `TicketValidator`
interface — the swap is clean.
### 2.1 — Store Interface
New interface in `store/`:
```go
// TicketStore manages short-lived single-use WebSocket auth tickets.
type TicketStore interface {
Issue(ctx context.Context, userID string) (string, error)
Validate(ctx context.Context, ticketID string) (string, bool)
Reap(ctx context.Context) (int, error)
}
```
### 2.2 — PG Implementation
**Issue:**
```sql
INSERT INTO ws_tickets (id, user_id, expires_at)
VALUES ($1, $2, NOW() + INTERVAL '30 seconds')
```
Token generation stays in Go (`crypto/rand`, 16 bytes, hex-encoded).
**Validate (atomic consume):**
```sql
DELETE FROM ws_tickets
WHERE id = $1 AND expires_at > NOW()
RETURNING user_id
```
Single statement — delete + return. If expired or already consumed,
zero rows → `("", false)`.
**Reap (TTL cleanup):**
```sql
DELETE FROM ws_tickets WHERE expires_at <= NOW()
```
Called by a system task or inline during poll. No dedicated goroutine
needed — the scheduler's 30s tick can piggyback, or register as a
lightweight system function.
### 2.3 — SQLite Implementation
Identical SQL with dialect adjustments (`datetime('now', '+30 seconds')`
instead of `NOW() + INTERVAL`).
### 2.4 — Adapter for Middleware Interface
The middleware `TicketValidator` interface is:
```go
type TicketValidator interface {
Validate(ticketID string) (string, bool)
}
```
The store's `Validate` takes a `context.Context`. Thin adapter:
```go
// TicketValidatorAdapter bridges store.TicketStore → middleware.TicketValidator.
type TicketValidatorAdapter struct {
Store store.TicketStore
}
func (a *TicketValidatorAdapter) Validate(ticketID string) (string, bool) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
return a.Store.Validate(ctx, ticketID)
}
```
### 2.5 — Wiring in `main.go`
Replace:
```go
ticketStore := events.NewTicketStore()
defer ticketStore.Stop()
```
With:
```go
ticketAdapter := &events.TicketValidatorAdapter{Store: stores.Tickets}
```
The `POST /ws/ticket` endpoint calls `stores.Tickets.Issue(ctx, userID)`
directly. The WS auth middleware receives `ticketAdapter` (satisfies
`TicketValidator`). No reaper goroutine — use system function or
scheduler piggyback.
### 2.6 — Reap Strategy
Option A: Register `ticket_reap` as a system function
(`taskutil.SystemRegistry`), add a system task with `schedule: "@hourly"`
or `"*/5 * * * *"`. Pros: visible in admin UI, uses existing
infrastructure. Cons: requires task to exist.
Option B: Inline reap — call `stores.Tickets.Reap(ctx)` at the top
of the scheduler `poll()` loop (every 30s, cheap no-op when table is
empty). Pros: zero config. Cons: slightly couples scheduler to tickets.
**Recommendation: Option B.** At 50 users, the table will have at most
a few dozen rows. A `DELETE WHERE expires_at <= NOW()` every 30s is
free. Revisit if ticket volume grows.
### 2.7 — Delete `events/tickets.go`
The in-memory `TicketStore` struct, `NewTicketStore()`, and reaper
goroutine are fully replaced. Remove the file. The `TicketValidatorAdapter`
lives in `events/` (or alongside the middleware — whichever avoids
import cycles).
### Deliverables
- [ ] `store.TicketStore` interface
- [ ] PG implementation + SQLite implementation
- [ ] `TicketValidatorAdapter` — bridges `context`-aware store to
`TicketValidator` interface
- [ ] `main.go` wiring — swap in PG store, remove `events.NewTicketStore()`
- [ ] Inline reap in scheduler poll
- [ ] Delete `events/tickets.go`
- [ ] Add `Tickets` field to `store.Stores`
- [ ] CI green
---
## CS3 — PG Rate Limiter
Replace the in-memory token bucket (`middleware.RateLimiter`) with a
PG-backed counter. Currently only applied to `/auth` routes (5 req/s,
burst 8), so transaction volume is trivially low.
### 3.1 — Store Interface
```go
// RateLimitStore manages distributed rate limit counters.
type RateLimitStore interface {
// Allow checks if a request is within the rate limit.
// Returns (allowed bool, tokensRemaining float64).
// Atomically increments the counter if allowed.
Allow(ctx context.Context, key string, rate float64, burst int) (bool, error)
// Cleanup removes expired windows.
Cleanup(ctx context.Context, maxAge time.Duration) error
}
```
### 3.2 — PG Implementation: Sliding-Window Token Bucket
The in-memory implementation uses a classic token bucket (refill based
on elapsed time). The PG version approximates this with time-bucketed
counters.
**`Allow` — atomic check-and-increment:**
```sql
-- Upsert the current window's counter and check burst limit.
-- Window = current second (truncated).
INSERT INTO rate_limit_counters (key, window, tokens)
VALUES ($1, date_trunc('second', NOW()), 1)
ON CONFLICT (key, window)
DO UPDATE SET tokens = rate_limit_counters.tokens + 1
RETURNING tokens
```
Go logic after the RETURNING:
1. Query returns `tokens` (count in the current 1-second window).
2. If `tokens > burst` → denied, return `(false, nil)`.
3. Else → allowed.
This is a fixed-window approximation, not a sliding window. For auth
rate limiting at 5 req/s burst 8, the practical difference is
negligible — worst case allows 2× burst at window boundaries, which
is acceptable for this use case. A true sliding window would require
reading adjacent windows and interpolating, adding complexity for
near-zero benefit at this scale.
**Cleanup:**
```sql
DELETE FROM rate_limit_counters
WHERE window < NOW() - $1::interval
```
Called alongside ticket reap in the scheduler poll loop.
### 3.3 — SQLite Implementation
Same logic, `datetime('now')` for window truncation. Single-process,
so the PG atomicity guarantees are naturally satisfied.
### 3.4 — Middleware Swap
Replace `middleware.RateLimiter` struct with one backed by the store:
```go
type RateLimiter struct {
store store.RateLimitStore
rate float64
burst int
}
func NewRateLimiter(store store.RateLimitStore, rate float64, burst int) *RateLimiter {
return &RateLimiter{store: store, rate: rate, burst: burst}
}
func (rl *RateLimiter) Limit() gin.HandlerFunc {
return func(c *gin.Context) {
key := "auth:" + c.ClientIP()
allowed, err := rl.store.Allow(c.Request.Context(), key, rl.rate, rl.burst)
if err != nil {
// DB error — fail open (don't block auth on rate limit DB failure)
c.Next()
return
}
if !allowed {
c.Header("Retry-After", "1")
c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
"error": "rate limit exceeded",
})
return
}
c.Next()
}
}
```
**Fail-open policy:** if PG is down, allow the request. The auth
endpoints have their own protections (bcrypt, lockout). Blocking
legitimate logins because the rate limit table is unreachable is worse
than allowing a brief burst.
### 3.5 — Wiring in `main.go`
Replace:
```go
authLimiter := middleware.NewRateLimiter(5, 8)
```
With:
```go
authLimiter := middleware.NewRateLimiter(stores.RateLimits, 5, 8)
```
Delete the in-memory cleanup goroutine (replaced by scheduler reap).
### Deliverables
- [ ] `store.RateLimitStore` interface
- [ ] PG implementation + SQLite implementation
- [ ] `middleware.RateLimiter` rewrite — backed by store, fail-open
- [ ] `main.go` wiring
- [ ] Add `RateLimits` field to `store.Stores`
- [ ] Scheduler poll cleanup (rate_limit_counters + ws_tickets)
- [ ] Delete in-memory visitor map + cleanup goroutine
- [ ] CI green
---
## CS4 — WebSocket Cross-Pod Delivery
The event bus → `pg_broadcast` → remote `publishLocal` pipeline is
already implemented and handles room-scoped events correctly. The gap:
**`hub.SendToUser()` bypasses the bus entirely** — it writes directly
to local connection send channels. Users on other pods are never reached.
### Affected call sites (7 total)
| File | Context |
|------|---------|
| `main.go` | DM typing indicators |
| `handlers/completion.go` (×3) | Streaming chunks, tool results, mentions |
| `handlers/stream_loop.go` | SSE stream events |
| `handlers/tool_loop.go` | Browser tool call delivery |
| `notifications/service.go` | Push notification delivery |
### 4.1 — Add `TargetUserID` to Event
```go
type Event struct {
Label string `json:"event"`
Room string `json:"room,omitempty"`
Payload json.RawMessage `json:"payload"`
Ts int64 `json:"ts"`
// Server-side metadata (not serialized to clients)
SenderID string `json:"-"`
ConnID string `json:"-"`
TargetUserID string `json:"-"` // v0.32.0: cross-pod targeted delivery
}
```
`TargetUserID` is included in the JSON serialization for `pg_broadcast`
(so the remote pod knows who to deliver to) but stripped before sending
to WebSocket clients. Add a custom `MarshalJSON` for the broadcast
path, or use a separate internal envelope.
**Simpler approach:** Use `json:"target_user_id,omitempty"` — it's
harmless if clients see it (they already ignore unknown fields), and
avoids a second serialization path. The field is empty for room-scoped
events and only populated for targeted delivery.
### 4.2 — Update WS Subscriber Filter
In `subscribeToBus`, add target filtering:
```go
// Targeted delivery: if event has a target user, only deliver to that user
if e.TargetUserID != "" && e.TargetUserID != c.userID {
return
}
```
This goes before the room filter. Targeted events skip room filtering
entirely (they're user-scoped, not room-scoped).
### 4.3 — Rewrite `SendToUser` → `PublishToUser`
New method on `Hub`:
```go
// PublishToUser sends an event to a specific user via the bus.
// Cross-pod safe: the bus broadcast hook fans out via pg_notify.
func (h *Hub) PublishToUser(userID string, event Event) {
event.TargetUserID = userID
h.bus.Publish(event)
}
```
This replaces all 7 `hub.SendToUser` call sites. The event flows:
1. `hub.PublishToUser(userID, evt)`
2. `bus.Publish(evt)` → local subscribers + `broadcastHook`
3. Local pod: `subscribeToBus` filter matches `TargetUserID` → deliver
4. `broadcastHook` → `pg_notify` → remote pod
5. Remote pod: `publishLocal(evt)` → `subscribeToBus` filter → deliver
### 4.4 — Keep `SendToUser` as Local-Only Optimization
Don't delete `SendToUser` — rename to `sendToUserLocal` (unexported).
`PublishToUser` can optionally try local delivery first (fast path)
and only broadcast if the user isn't connected locally. But this
optimization is premature at 50 users — just always go through the
bus. Revisit post-MVP if latency matters.
### 4.5 — Audit: `tool.call` + `WaitFor`
`tool.call.*` events use `DirToClient` routing and are delivered via
`SendToUser`. After the rewrite, they go through the bus and will
broadcast to all pods. The `WaitFor` on the originating pod still
works — `tool.result.*` comes back from the client on the same pod
(the WS connection is sticky to a pod).
Verify: if the user's browser tool is connected to pod-2, but the
completion handler calling `WaitFor` is on pod-1, the tool call event
needs to reach pod-2 (via pg_broadcast), and the result needs to come
back to pod-1 (via pg_broadcast of `tool.result.*`). Check
`RouteFor("tool.result.*")` — currently `DirFromClient`, which means
the broadcast hook skips it. **This needs to change to `DirBoth`** so
tool results cross pod boundaries for `WaitFor` to work.
### Deliverables
- [ ] `TargetUserID` field on `Event`
- [ ] `subscribeToBus` target filter
- [ ] `Hub.PublishToUser` — replaces all `SendToUser` call sites
- [ ] Update `tool.result.*` routing to `DirBoth`
- [ ] Rename `SendToUser` → `sendToUserLocal` (unexported)
- [ ] Verify `WaitFor` works cross-pod (tool bridge scenario)
- [ ] CI green
---
## CS5 — Health Probes + Helm Validation
### 5.1 — Readiness Probe Refinement
Current health endpoint: basic HTTP 200. Add a PG ping:
```go
func (h *HealthHandler) Readiness(c *gin.Context) {
ctx, cancel := context.WithTimeout(c.Request.Context(), 2*time.Second)
defer cancel()
if err := database.DB.PingContext(ctx); err != nil {
c.JSON(503, gin.H{"error": "database unavailable"})
return
}
c.JSON(200, gin.H{"status": "ok"})
}
```
Wire as `/healthz/ready` (separate from the existing `/healthz/live`
liveness probe). Kubernetes pulls the pod from the service on readiness
failure — new requests route to healthy replicas.
### 5.2 — Helm Changes
`chart/values.yaml`:
```yaml
backend:
replicaCount: 2 # was 1
```
`chart/templates/deployment-backend.yaml` additions:
**Pod anti-affinity** — spread replicas across nodes:
```yaml
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: switchboard-backend
topologyKey: kubernetes.io/hostname
```
`preferredDuring` (not `requiredDuring`) — if fewer nodes than
replicas, pods still schedule on the same node (degraded HA is better
than no scheduling).
**Readiness probe:**
```yaml
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
```
### 5.3 — Startup Jitter
Add random jitter (0–15s) to the scheduler's initial tick to stagger
replica polling:
```go
func (s *Scheduler) Run() {
// ...
jitter := time.Duration(rand.Intn(15000)) * time.Millisecond
time.Sleep(jitter)
log.Printf("[scheduler] Started (jitter=%s, interval=%s)", jitter, s.interval)
// ...
}
```
Not strictly necessary with `SKIP LOCKED` (contention is harmless),
but reduces unnecessary lock acquisition at startup.
### 5.4 — Validation Checklist
Manual validation on the `gobha-ai-chat` cluster:
- [ ] Scale to 2 replicas: `kubectl scale deploy switchboard-backend --replicas=2`
- [ ] Verify pods land on different nodes (`kubectl get pods -o wide`)
- [ ] Create a cron task (every 1 min), confirm only one run per tick
- [ ] WebSocket: connect on pod-1, send message that completes on
pod-2, verify streaming events arrive
- [ ] Issue WS ticket on pod-1, connect WS on pod-2 — ticket validates
- [ ] Rate limit: hit `/auth/login` 10× rapidly — confirm 429 after
burst regardless of which replica serves each request
- [ ] Kill one pod (`kubectl delete pod ...`), confirm the other
continues serving + picks up scheduler duties immediately
- [ ] `kubectl top pod` — memory baseline at 2 replicas under light load
### Deliverables
- [ ] `/healthz/ready` endpoint with PG ping
- [ ] Helm: `replicaCount: 2`, pod anti-affinity, readiness probe
- [ ] Scheduler startup jitter
- [ ] Validation checklist executed and documented
- [ ] CI green, tag `v0.32.0`
---
## Summary: Store Interface Additions
```go
type Stores struct {
// ... existing fields ...
Tickets TicketStore // v0.32.0: WS auth tickets (PG-backed)
RateLimits RateLimitStore // v0.32.0: Distributed rate limiting
}
```
`TaskStore` changes:
- **Add:** `ClaimDueTask(ctx) (*Task, error)`
- **Add:** `CreateRunExclusive(ctx, taskID) (*TaskRun, error)`
- **Remove:** `ListDue(ctx, limit) ([]Task, error)`
---
## Changeset Sequence
| CS | Scope | Key Files | Gate |
|----|-------|-----------|------|
| CS0 | Schema | `020_ha.sql` (PG + SQLite) | Migrations run, tables exist |
| CS1 | Task scheduler | `store/task_iface.go`, `store/postgres/tasks.go`, `store/sqlite/tasks.go`, `scheduler/scheduler.go` | Concurrent claim test passes |
| CS2 | Ticket store | `store/ticket_iface.go`, `store/postgres/tickets.go`, `store/sqlite/tickets.go`, `events/tickets.go` (delete), `main.go` | WS auth works with PG tickets |
| CS3 | Rate limiter | `store/ratelimit_iface.go`, `store/postgres/ratelimit.go`, `store/sqlite/ratelimit.go`, `middleware/ratelimit.go`, `main.go` | Rate limit shared across replicas |
| CS4 | WS fan-out | `events/types.go`, `events/ws.go`, all `SendToUser` call sites | Cross-pod event delivery |
| CS5 | Health + Helm | `handlers/health.go`, `chart/`, `scheduler/scheduler.go` | 2-replica cluster validated |
---
## Risk Assessment
**Low risk:** CS0 (schema), CS2 (ticket store — simple CRUD), CS3
(rate limiter — auth-only, fail-open).
**Medium risk:** CS1 (scheduler rewrite — core behavior change, but
well-scoped and testable), CS5 (Helm — infra changes, but rollback is
`--replicas=1`).
**Higher risk:** CS4 (WS fan-out — touches 7 call sites across
handlers, completion, notifications). The `tool.result` routing change
(`DirFromClient` → `DirBoth`) needs careful testing — if tool results
broadcast to all pods, the non-originating pod's `WaitFor` must not
accidentally consume the result. Verify: `WaitFor` subscribes to
`tool.result.{specific-id}` (exact label match), so only the waiting
goroutine receives it. Safe.
**Fallback:** If any CS breaks multi-replica, `replicaCount: 1` is
always a safe rollback. Each CS should leave CI green at single-replica.

238
docs/archive/DESIGN-0.33.0.md
View File

@@ -1,238 +0,0 @@
# DESIGN-0.33.0 — Observability
Operate the platform without reading Go source code. Six changesets
covering structured logging, Prometheus metrics, OpenAPI docs, Grafana
dashboards, alerting rules, and a built-in admin dashboard.
Depends on: v0.32.0 (Multi-Replica HA, current HEAD).
**Design decision:** Zero new database migrations. All metrics are
in-memory per-pod, scraped by Prometheus. Existing tables
(`provider_health`, `tool_health_windows`, `usage_log`, `audit_log`)
continue serving the admin dashboard via stores.
**Library choices:**
- `log/slog` (Go 1.22 stdlib) — zero-dep structured logging
- `prometheus/client_golang` v1.20+ — canonical Go Prometheus client
- Hand-curated OpenAPI 3.0.3 YAML — avoids annotation sprawl across 97 handlers
- Swagger UI v5 via CDN — single HTML embed, zero build step
---
## What Already Existed
These required no structural changes, only instrumentation:
- **Health accumulator** — in-memory provider+tool metrics, 60s DB flush
- **`Hub.ConnCount()`** — active WebSocket connection count
- **`database.DB.Stats()`** — Go stdlib DB pool metrics
- **K8s probes** — `/healthz/ready` + `/healthz/live` in Helm chart
- **Middleware skip paths** — `/metrics` already in `SkipPaths` list
- **`google/uuid`** — already in go.mod for request ID generation
---
## CS0 — Structured Logging + Request ID
Foundation changeset. All subsequent work emits structured logs.
### `server/logging/logger.go` (new)
`Init(format, level)` configures the global `slog` logger:
- `format=json` → `slog.NewJSONHandler(os.Stdout, opts)`
- `format=text` → `slog.NewTextHandler(os.Stdout, opts)`
- Levels: debug, info, warn, error
### `server/middleware/request_id.go` (new)
Generates UUID per request. Reuses existing `google/uuid`.
Honors inbound `X-Request-Id` header for trace propagation.
Sets `request_id` in Gin context and echoes header on response.
### Middleware chain rewrite
Changed `gin.Default()` → `gin.New()` with explicit chain:
```
RequestID → Prometheus → Logger → Recovery → CORS
```
Logger middleware rewritten from `log.Printf` to `slog.Info("request", ...)`
with structured fields: method, path, status, latency_ms, client_ip,
request_id, user_id.
### Config additions
- `LogFormat` (env: `LOG_FORMAT`, default: `text`)
- `LogLevel` (env: `LOG_LEVEL`, default: `info`)
---
## CS1 — Prometheus `/metrics` Endpoint
### `server/metrics/metrics.go` (new)
All metric definitions in one file using `promauto`:
| Metric | Type | Labels |
|--------|------|--------|
| `switchboard_http_requests_total` | Counter | method, path_pattern, status |
| `switchboard_http_request_duration_seconds` | Histogram | method, path_pattern |
| `switchboard_websocket_connections` | Gauge | — |
| `switchboard_completion_tokens_total` | Counter | direction, model_id |
| `switchboard_completions_total` | Counter | provider_config_id, model_id, status |
| `switchboard_completion_duration_seconds` | Histogram | provider_config_id, model_id |
| `switchboard_provider_status` | Gauge | provider_config_id |
| `switchboard_db_open_connections` | Gauge | — |
| `switchboard_db_in_use_connections` | Gauge | — |
| `switchboard_db_idle_connections` | Gauge | — |
| `switchboard_db_wait_count` | Gauge | — |
| `switchboard_db_wait_duration_seconds` | Gauge | — |
| `switchboard_task_executions_total` | Counter | status |
| `switchboard_sandbox_executions_total` | Counter | entry_point, status |
### `server/metrics/db_collector.go` (new)
Background goroutine reads `database.DB.Stats()` every 15s, updates
Prometheus gauges.
### `server/middleware/prometheus.go` (new)
Gin middleware. Uses `c.FullPath()` for `path_pattern` label to avoid
cardinality explosion from path parameters (e.g. `/channels/:id`
not `/channels/abc123`).
### Instrumentation points
- `events/ws.go` — `WebSocketConnections.Inc()/Dec()` on connect/disconnect
- `handlers/completion.go` — token counters in `logUsage()`, completion
duration/status in `recordHealth()`
- `handlers/stream_loop.go` — same instrumentation in `recordHealthFn()`
- `health/accumulator.go` — `ProviderStatus` gauge updated in `flush()`
### Route
`/metrics` mounted via `promhttp.Handler()` (no auth, standard scraping).
---
## CS2 — OpenAPI Spec + Swagger UI
### `server/static/openapi.yaml` (new)
Hand-curated OpenAPI 3.0.3 spec covering core API groups:
Auth, Channels, Completions, Health, Metrics, WebSocket Tickets,
Admin (stats, health, usage, dashboard).
### `server/static/swagger.html` (new)
Minimal HTML loading Swagger UI v5 from unpkg CDN, pointing at
`openapi.yaml`. Zero build step.
### Routes
- `GET /api/docs` → serves embedded `swagger.html`
- `GET /api/docs/openapi.yaml` → serves embedded `openapi.yaml`
Both files embedded via `//go:embed` directives.
---
## CS3 — Grafana Dashboard + Alerting Rules + Helm
Static files + Helm templates. No Go code changes.
### `chart/dashboards/switchboard-overview.json` (new)
Grafana dashboard with 10 panels:
- Request rate, error rate, latency percentiles
- WebSocket connections, completion rate/latency by provider
- Tokens/min by model, provider status
- DB connection pool, task executions
Template variables: `$datasource`, `$namespace`, `$pod`.
### `chart/alerting/switchboard-rules.yaml` (new)
Source PrometheusRule with 6 alerts:
| Alert | Condition | Severity |
|-------|-----------|----------|
| SwitchboardPodRestart | Restarts in 1h | warning |
| SwitchboardProviderDown | Status > 2 for 5m | critical |
| SwitchboardDBPoolExhaustion | In-use/open > 80% for 5m | warning |
| SwitchboardHighErrorRate | 5xx rate > 5% for 5m | warning |
| SwitchboardTaskFailureRate | Error rate > 25% for 10m | warning |
| SwitchboardNoCompletions | Zero completions for 15m | critical |
### Helm templates (new, all gated `enabled: false` by default)
- `chart/templates/servicemonitor.yaml` — ServiceMonitor CRD
- `chart/templates/grafana-dashboard-configmap.yaml` — ConfigMap for
Grafana sidecar discovery
- `chart/templates/prometheusrule.yaml` — PrometheusRule CRD
### Helm modifications
- `chart/values.yaml` — `logging` section + `monitoring` section
- `chart/templates/configmap.yaml` — `LOG_FORMAT`, `LOG_LEVEL` env vars
- `chart/templates/services.yaml` — Prometheus scrape annotations
---
## CS4 — Admin Observability Dashboard
Built-in admin page for real-time health without Grafana dependency.
### `server/handlers/dashboard_admin.go` (new)
`GET /api/v1/admin/dashboard` aggregating:
- Provider health summaries (from `HealthStore`)
- 24h usage totals (from `UsageStore`)
- DB pool stats (`database.DB.Stats()`)
- WebSocket connection count (`Hub.ConnCount()`)
- Recent errors (from `AuditStore`)
- Process uptime
### Frontend
- `SCAFFOLDING.dashboard` — stat cards grid + content container
- `ADMIN_LOADERS.dashboard` → `UI.loadAdminDashboard()`
- `ADMIN_SECTIONS.monitoring` — `dashboard` added as first section
- Dashboard auto-refreshes every 30s
- CSS: provider grid, DB pool bar gauge, error list
### Admin template
Monitoring tab now lands on `/admin/dashboard` instead of `/admin/usage`.
---
## CS5 — Wiring, docker-compose, ICD Tests
### docker-compose.yml
Added `LOG_FORMAT` and `LOG_LEVEL` env vars with defaults.
### ICD tests (`crud/observability.js`)
| Test | Assertion |
|------|-----------|
| `GET /metrics` | 200, contains `switchboard_*` metrics |
| `GET /api/docs` | 200, contains `swagger-ui` |
| `GET /api/docs/openapi.yaml` | 200, contains `openapi:` and `paths:` |
| X-Request-Id generation | Response includes 36-char UUID header |
| X-Request-Id passthrough | Client-sent header echoed back |
| `GET /admin/dashboard` | 200, has `uptime`, `ws_connections`, `provider_health` |
---
## Verification
1. `LOG_FORMAT=json docker compose up` → JSON log lines on stdout
2. `curl localhost:8080/metrics | grep switchboard_` → all metrics present
3. `curl -I localhost:8080/api/v1/channels` → `X-Request-Id` header
4. `/api/docs` in browser → Swagger UI renders
5. `/admin` → Monitoring → Dashboard shows live stat cards
6. `helm lint chart/ --set monitoring.serviceMonitor.enabled=true` → passes
7. ICD test suite passes (existing + observability tests)

217
docs/archive/DESIGN-0.34.0.md
View File

@@ -1,217 +0,0 @@
# DESIGN-0.34.0 — Data Portability
Export your data, import it elsewhere, delete your account. Six
changesets covering user export, user import, GDPR delete, admin
team export/import, ChatGPT import, and K8s scheduled backups.
Depends on: v0.33.0 (Observability, current HEAD).
**Design decision:** One new migration (021) — indexes only, no new
tables. The export archive is a ZIP file (`.switchboard` extension)
containing a JSON manifest and per-entity JSON files. Sensitive fields
(password hashes, encrypted keys, storage keys) are stripped at export
time via entity sanitization.
**Archive format:** `.switchboard` ZIP with `format_version=1`:
```
manifest.json
data/users.json
data/channels.json
data/messages.json
data/channel_participants.json
data/channel_models.json
data/channel_cursors.json
data/notes.json
data/note_links.json
data/memories.json
data/projects.json
data/project_channels.json
data/project_knowledge_bases.json
data/project_notes.json
data/workspaces.json
data/workspace_files.json
data/folders.json
data/user_settings.json
data/notification_preferences.json
data/usage_entries.json
data/persona_groups.json
data/persona_group_members.json
data/files.json
files/{file_id}/{filename}
```
---
## What Already Existed
- **Package export** — `.pkg` ZIP format, `package_export.go`
- **Markdown export** — pandoc-based, `export.go`
- **Workspace archive** — ZIP/tar.gz with size limits, `workspace/archive.go`
- **Soft deletes** — `deleted_at` on messages
- **Audit trail** — `audit_logs` table
- **39 store interfaces** covering all entity types
---
## CS0 — Export Format + User Export
Foundation changeset. Defines the archive format and ExportStore interface.
### `server/export/format.go` (new)
Archive types: `Manifest`, `ManifestScope`, `ArchiveWriter`, `ArchiveReader`.
Size limits: 500 MB archive, 100 MB per file, 10K files max.
Key methods: `WriteManifest`, `WriteEntityJSON`, `WriteFile`,
`ReadManifest`, `ReadEntityJSON`, `FileEntries`.
### `server/export/entities.go` (new)
Per-entity sanitization functions. Export types strip sensitive fields:
- `ExportChannel` — removes `provider_config_id`
- `ExportMessage` — no changes needed
- `ExportWorkspace` — removes `root_path`, git credentials
- `ExportUsageEntry` — removes `provider_config_id`
- Users — removes `password_hash`, `encrypted_uek`, `uek_salt`
### `server/store/interfaces.go` (modified)
Added `Export ExportStore` to `Stores` struct. The `ExportStore`
interface provides 21 user-scoped read methods, 20 import methods
(returning `(imported, skipped, error)` counts), 14 team-scoped
read methods, and 4 GDPR methods.
### `server/store/{postgres,sqlite}/export.go` (new)
Full implementations for both dialects (~1400-1500 lines each).
Import uses `INSERT ON CONFLICT DO NOTHING` (PG) / `INSERT OR IGNORE`
(SQLite) for UUID-based dedup.
### `server/handlers/export_data.go` (new)
`DataExportHandler.ExportMyData` — `GET /api/v1/export/me`.
Streams ZIP directly to response using `zip.NewWriter(c.Writer)`.
Includes file blobs from object store.
### Migration 021
Indexes only — `idx_messages_channel_active`, `idx_channels_user_id`,
`idx_files_user_id`. PG version uses partial index
(`WHERE deleted_at IS NULL`).
---
## CS1 — User Import
### `server/handlers/import_data.go` (new)
`DataImportHandler.ImportMyData` — `POST /api/v1/import/me`.
Opens archive, validates manifest `format_version`, reads entities
in FK-dependency order, remaps `user_id` for cross-instance import.
Returns `{imported: {channels: N, ...}, skipped: {...}, errors: [...]}`.
Import order: Folders → Projects → Workspaces → Channels →
Participants/Models/Cursors → Messages → Notes → Note Links →
Memories → Project links → Workspace files → Files + blobs →
Settings → Persona groups.
---
## CS2 — GDPR Account Delete
### `server/handlers/gdpr.go` (new)
`GDPRHandler.DeleteMyAccount` — `DELETE /api/v1/me`.
Requires `{"confirm": "DELETE", "password": "..."}`. Verifies
password, prevents last-admin deletion, then:
1. Soft-deletes messages, archives channels
2. Hard-deletes: notes, memories, workspaces, files, projects, settings
3. Revokes tokens (refresh + WS tickets)
4. Anonymizes user: `deleted-user-{sha256(id)[:12]}`
5. Writes audit log entry
---
## CS3 — Admin Team Export/Import
`DataExportHandler.ExportTeam` — `GET /api/v1/admin/teams/:id/export`
`DataImportHandler.ImportTeam` — `POST /api/v1/admin/teams/:id/import`
Same archive format, scoped to team entities (channels, personas,
knowledge bases, workflows, groups, projects, resource grants).
Team import skips members whose `user_id` doesn't exist on target.
---
## CS4 — ChatGPT Import
### `server/export/chatgpt.go` (new)
Parses ChatGPT `conversations.json` format. Each conversation maps
to one channel (type=direct). The `mapping` DAG is walked depth-first
to produce ordered messages with `parent_id` and `sibling_index`.
Role mapping: `user`→user, `assistant`→assistant, `system`→system,
`tool`→tool. Content: `parts[]` joined with `\n`, non-string parts
skipped. Timestamps: `float64` epoch → `time.Time`.
`DataImportHandler.ImportChatGPT` — `POST /api/v1/import/chatgpt`.
Accepts multipart upload of `conversations.json` or ZIP containing it.
---
## CS5 — K8s Backup + ICD Tests
### `chart/templates/cronjob-backup.yaml` (new)
CronJob running `pg_dump | gzip` on schedule (default daily 02:00 UTC).
Optional S3 upload via AWS CLI. Local retention pruning. Gated on
`backup.enabled=true` and `database.driver=postgres`.
### `chart/templates/pvc-backup.yaml` (new)
Conditional PVC for local backup storage (when `backup.persistence.enabled`).
### `chart/values.yaml` (modified)
New `backup:` section with schedule, retention, S3 config, persistence,
and resource limits.
### ICD Tests
~30 new tests in `crud/portability.js` covering:
- Export download + archive validation (ZIP structure, manifest, entities)
- Sensitive field exclusion verification
- Import re-import (dedup validation)
- Invalid archive rejection
- ChatGPT import (valid, invalid JSON, empty array)
- ChatGPT channel/message verification
- GDPR delete flow (register → create data → delete → verify lockout)
- GDPR validation (missing confirm, wrong password)
- Admin team export/import
---
## New Routes
| Method | Path | Auth | Handler |
|--------|------|------|---------|
| GET | `/api/v1/export/me` | JWT | `DataExportHandler.ExportMyData` |
| POST | `/api/v1/import/me` | JWT | `DataImportHandler.ImportMyData` |
| POST | `/api/v1/import/chatgpt` | JWT | `DataImportHandler.ImportChatGPT` |
| DELETE | `/api/v1/me` | JWT | `GDPRHandler.DeleteMyAccount` |
| GET | `/api/v1/admin/teams/:id/export` | Admin | `DataExportHandler.ExportTeam` |
| POST | `/api/v1/admin/teams/:id/import` | Admin | `DataImportHandler.ImportTeam` |
---
## Verification
1. `cd server && go build ./...` — all changesets compile clean
2. `helm template ./chart --set backup.enabled=true` — CronJob renders
3. ICD test suite: ~615 tests (585 existing + ~30 portability)
4. Manual smoke: export → inspect ZIP → re-import → verify dedup
5. GDPR: delete → verify login fails → inspect anonymized DB record
6. Sensitive fields: no `password_hash`, `encrypted_uek`, `storage_key` in export

148
docs/archive/DESIGN-0.35.0.md
View File

@@ -1,148 +0,0 @@
# DESIGN-0.35.0 — Workflow Product
Transforms the workflow engine from a linear stage-runner into a
product-grade automation tool. Seven changesets covering conditional
routing, AI-triggered routing, data pipeline hooks, progressive forms,
conditional fields, structured review, and a monitoring dashboard.
Depends on: v0.34.0 (Data Portability), v0.31.2 (Team Workflows),
v0.29.0 (Starlark Sandbox).
## Schema Changes
Migrations modified in-place (DB rebuild required):
**005_channels.sql** — add `stage_entered_at` (PG: `TIMESTAMPTZ`,
SQLite: `TEXT`). Tracks when the current stage was entered for SLA
computation. Also add index `idx_channels_workflow_active` on
`(workflow_id, workflow_status)` for monitoring queries.
**018_workflows.sql** — add `sla_seconds INTEGER` to `workflow_stages`.
Add `review_comments` (PG: `JSONB DEFAULT '[]'`, SQLite: `TEXT
DEFAULT '[]'`) to `workflow_assignments`.
No new migration files. No new tables.
## Conditional Routing Engine
**File:** `server/workflow/routing.go`
The existing advance logic hardcoded `nextStage = currentStage + 1`.
The routing engine replaces this with condition evaluation:
1. Parse `transition_rules.conditions[]` from the departing stage
2. Evaluate each condition against accumulated `stage_data`
3. First match → resolve `target_stage` (by name or ordinal)
4. No match → fallback to `ordinal + 1`
**Expression format:** `{field, op, value, target_stage}`. Operators:
`eq`, `neq`, `gt`, `lt`, `gte`, `lte`, `in`, `contains`, `exists`,
`not_exists`. Loose equality via `fmt.Sprintf("%v")` comparison.
Numeric comparison extracts `float64` from both sides.
**Wiring:** `ResolveNextStage()` called in all 4 advance sites:
handler (`workflow_instances.go`), tool (`tools/workflow.go`), forms
auto-advance (`workflow_forms.go`), Starlark module
(`sandbox/workflow_module.go`).
**Backward compatible:** no conditions = `ordinal + 1`.
## AI-Triggered Routing
**Tool:** `workflow_route` — persona calls with `{target_stage, reason}`.
Uses `ResolveStageByName()` (case-insensitive). Records decision in
`stage_data._route_history_latest`. Allows forward and backward jumps
(loop stages for iterative correction).
**Starlark:** `workflow.route(channel_id, target_stage, reason)` for
extension-driven escalation patterns. Requires `workflow.access`.
## on_advance Hook
**File:** `server/handlers/workflow_hooks.go`
Fires synchronously after `AdvanceWorkflowStage` succeeds. Configured
per-stage in `transition_rules.on_advance: {package_id, entry_point}`.
Hook receives: `{stage_data, previous_stage, current_stage, channel_id}`.
Returns: `{stage_data: {...}}` (enriched), `None` (no change), or
`{error: "msg"}` (logged, not rolled back — future: rollback support).
The existing `http` Starlark module (v0.29.1) is available in hooks,
enabling external API enrichment without new infrastructure.
## Progressive Forms
**Model:** `TypedFormTemplate.Fieldsets []FormFieldset` — optional
array of `{label, fields[]}`. When present, top-level `fields` is
replaced by flattened fieldset fields in `ParseTypedFormTemplate()`.
**Frontend:** `workflow.html` renders one fieldset at a time with
step indicator, next/back navigation. All fields submitted together
on the final step.
## Conditional Fields
**Model:** `FormField.Condition *FieldCondition` — `{when, op, value}`.
Operators: `eq` (default), `neq`, `in`, `exists`.
**Server-side:** `ValidateFormData()` calls `evaluateFieldCondition()`
before each field. Hidden fields are skipped (no required check, no
type validation).
**Client-side:** `workflow.html` attaches `change`/`input` listeners
to source fields. Target fields toggle `cond-hidden` CSS class.
## Structured Review
**Review comments:** `review_comments JSONB` on `workflow_assignments`.
`POST /workflow-assignments/:id/comment` appends `{text, user_id,
created_at}`. `GET /workflow-assignments/:id` returns full assignment.
**Review surface:** Side-by-side layout in `workflow.html` — left panel
shows structured `stage_data` table (internal `_`-prefixed keys hidden),
right panel has comment textarea + approve/reject buttons. Keyboard
shortcuts: `Ctrl+Enter` approve, `Ctrl+Shift+Enter` reject.
## Monitoring Dashboard
**File:** `server/handlers/workflow_monitor.go`
**Endpoints:**
- `GET /admin/workflows/monitor/instances` — all active instances
- `GET /admin/workflows/monitor/funnel/:id` — stage counts
- `GET /admin/workflows/monitor/stale?threshold_hours=N`
- `GET /teams/:teamId/workflows/monitor/instances` — team-scoped
**SLA computation:** `stage_entered_at + sla_seconds` vs `NOW()`.
Returned as `sla_remaining_seconds` (negative = breached) and
`sla_breached` boolean. Computed at query time, not stored.
**Store:** New `ListByType(ctx, "workflow")` method on `ChannelStore`
(both PG and SQLite). Monitoring queries iterate active channels,
joining workflow definitions and stage metadata.
**Frontend:** `src/js/workflow-monitor.js` — auto-refresh every 30s,
SLA color indicators (green > 50%, yellow > 20%, red), stale instance
alert section.
## Workflow Branding
`WorkflowPageData.BrandingJSON` passes the workflow's `branding` JSON
to the visitor template. Applied as:
- `--accent` CSS custom property (from `accent_color`)
- Logo image in header (from `logo_url`)
- Tagline text below header (from `tagline`)
Previously branding was only on the landing page; now also on the
active workflow surface.
## New Files
| File | Purpose |
|------|---------|
| `server/workflow/routing.go` | Conditional routing engine |
| `server/handlers/workflow_hooks.go` | on_advance hook dispatch |
| `server/handlers/workflow_monitor.go` | Monitoring dashboard |
| `src/js/workflow-monitor.js` | Admin monitoring frontend |
| `packages/icd-test-runner/js/crud/workflow-product.js` | E2E tests |

498
docs/archive/DESIGN-CM6.md
View File

@@ -1,498 +0,0 @@
# DESIGN: CodeMirror 6 Integration
**Version:** v0.17.2
**Status:** Complete
**Scope:** Add CM6 as a vendored dependency, bundled at Docker build time via esbuild. Three integration surfaces.
---
## Motivation
Three features converge on the same dependency:
1. **Chat input** — live markdown rendering (backtick → code block, bold, etc.)
2. **Extension editor** — syntax-highlighted JavaScript/JSON editing in admin panel (replaces bare `<textarea>`)
3. **Code editing surface** (roadmap) — future extension surface type for code review, snippets, etc.
CM6 is ESM-native and expects a bundler. The build step is contained in the Docker image build — the frontend continues to ship as static assets served by nginx. No runtime build tooling, no change to the developer experience for non-CM6 code.
---
## Architecture
### Build Pipeline
New Docker stage between the existing `vendor` stage and the final `nginx` stage:
```
Stage 1: vendor (existing — marked, purify, mermaid, katex)
Stage 2: cm6-build (NEW — npm install + esbuild → single IIFE bundle)
Stage 3: nginx (existing — copies src/ + vendor/ + cm6 bundle)
```
The CM6 build stage:
```dockerfile
FROM node:20-alpine AS cm6-build
WORKDIR /build
COPY src/editor/package.json src/editor/build.mjs ./
RUN npm install --production=false
RUN node build.mjs
# Output: /build/dist/codemirror.bundle.js (~180-250KB minified)
# /build/dist/codemirror.bundle.css (~5-10KB)
```
Final stage addition:
```dockerfile
COPY --from=cm6-build /build/dist/ /usr/share/nginx/html/vendor/codemirror/
```
### Bundle Structure
A single esbuild entrypoint (`src/editor/index.mjs`) that imports CM6 packages and exposes factory functions on `window.CM`:
```javascript
// src/editor/index.mjs — esbuild entrypoint
import { EditorView, keymap, placeholder, ViewPlugin, ... } from '@codemirror/view';
import { EditorState, ... } from '@codemirror/state';
import { markdown, markdownLanguage } from '@codemirror/lang-markdown';
import { javascript } from '@codemirror/lang-javascript';
import { json } from '@codemirror/lang-json';
import { sql } from '@codemirror/lang-sql';
import { html } from '@codemirror/lang-html';
import { css } from '@codemirror/lang-css';
import { yaml } from '@codemirror/lang-yaml';
import { go } from '@codemirror/lang-go';
import { python } from '@codemirror/lang-python';
import { rust } from '@codemirror/lang-rust';
import { oneDark } from '@codemirror/theme-one-dark';
import { defaultKeymap, indentWithTab } from '@codemirror/commands';
import { bracketMatching, ... } from '@codemirror/language';
import { highlightSelectionMatches, searchKeymap } from '@codemirror/search';
import { autocompletion } from '@codemirror/autocomplete';
import { vim } from '@replit/codemirror-vim';
import { emacs } from '@replit/codemirror-emacs';
// Expose on window for script-tag consumption
window.CM = {
EditorView,
EditorState,
// Factory: chat input (markdown mode, minimal chrome)
chatInput(target, opts = {}) { ... },
// Factory: code editor (full features, language auto-detect)
codeEditor(target, opts = {}) { ... },
// Supported languages for code editor
languages: { markdown, javascript, json, sql, html, css, yaml, go, python, rust },
// Keybinding modes (applied via user preference)
keybindings: { vim, emacs },
};
```
### esbuild Config
```javascript
// src/editor/build.mjs
import { build } from 'esbuild';
await build({
entryPoints: ['index.mjs'],
bundle: true,
minify: true,
format: 'iife',
target: ['es2020'],
outfile: 'dist/codemirror.bundle.js',
// CSS extracted automatically by esbuild
});
```
### package.json (editor only)
```json
{
"private": true,
"type": "module",
"devDependencies": {
"esbuild": "^0.25.0"
},
"dependencies": {
"@codemirror/autocomplete": "^6",
"@codemirror/commands": "^6",
"@codemirror/lang-css": "^6",
"@codemirror/lang-go": "^6",
"@codemirror/lang-html": "^6",
"@codemirror/lang-javascript": "^6",
"@codemirror/lang-json": "^6",
"@codemirror/lang-markdown": "^6",
"@codemirror/lang-sql": "^6",
"@codemirror/lang-python": "^6",
"@codemirror/lang-rust": "^6",
"@codemirror/lang-yaml": "^6",
"@codemirror/language": "^6",
"@codemirror/search": "^6",
"@codemirror/state": "^6",
"@codemirror/theme-one-dark": "^6",
"@codemirror/view": "^6",
"@replit/codemirror-vim": "^6",
"@replit/codemirror-emacs": "^6"
}
}
```
---
## Integration Surfaces
### 1. Chat Input — Markdown Mode
Replace `<textarea id="messageInput">` with a CM6 instance configured for chat composition.
**Behavior:**
- Markdown syntax highlighting as you type (fenced code blocks, bold, italic, links, etc.)
- Shift+Enter inserts newline (existing behavior preserved)
- Enter sends message (existing behavior preserved)
- Auto-growing height (CM6 handles this natively via `EditorView.contentAttributes`)
- Placeholder text: "Send a message..."
- No line numbers, no gutter — minimal chrome
- Tab inserts real tab (indentWithTab) inside code blocks; normal behavior outside
**Factory:**
```javascript
CM.chatInput(document.getElementById('messageInputContainer'), {
placeholder: 'Send a message...',
onSubmit: (text) => sendMessage(text), // Enter key
onChange: (text) => updateInputTokens(text), // live token count
darkMode: true,
});
```
**Migration path:**
The existing code reads `document.getElementById('messageInput').value` in many places. The factory returns an object with a `.getValue()` / `.setValue()` / `.focus()` API that matches the textarea interface. A thin shim keeps all existing callsites working:
```javascript
// After CM.chatInput() creates the editor:
const editor = CM.chatInput(...);
// Shim: existing code that reads .value still works
Object.defineProperty(document.getElementById('messageInput'), 'value', {
get: () => editor.getValue(),
set: (v) => editor.setValue(v),
});
```
Alternatively (cleaner): introduce `ChatInput.getValue()` / `ChatInput.setValue()` / `ChatInput.focus()` and update callsites. There aren't many — `chat.js`, `attachments.js`, `tokens.js`, and a few event handlers.
### 2. Extension Editor — JavaScript/JSON Mode
Replace the two `<textarea>` elements in `editAdminExtension()` (`admin-handlers.js` line 706-710) with CM6 instances.
**Behavior:**
- Manifest textarea → JSON mode with bracket matching, auto-indent
- Script textarea → JavaScript mode with bracket matching, auto-indent
- Line numbers enabled
- Search/replace (Ctrl+F) — currently impossible in a textarea
- Tab key inserts proper indent (replaces the manual keydown handler at line 726-736)
- Themed to match the app's dark/light mode
**Factory:**
```javascript
CM.codeEditor(document.getElementById('extEdit-manifest-container'), {
language: 'json',
value: manifestJSON,
lineNumbers: true,
darkMode: isDark,
});
CM.codeEditor(document.getElementById('extEdit-script-container'), {
language: 'javascript',
value: script,
lineNumbers: true,
darkMode: isDark,
keymap: prefs.editorKeymap || 'standard', // 'standard' | 'vim' | 'emacs'
});
```
**Save integration:** `saveAdminExtension()` currently reads `.value` from the textareas. The CM6 instances expose `.getValue()` — update the save function to call that instead.
### 3. Future Code Surface (Roadmap)
The `CM.codeEditor()` factory with language auto-detection covers this. No additional work needed now — just document that it's available for the extensions surface type when that ships.
---
## Loading Strategy
The CM6 bundle is **not** in `SHELL_FILES` (service worker precache list). It loads on demand:
```html
<!-- index.html: load after core app scripts, before app.js init -->
<script src="vendor/codemirror/codemirror.bundle.js?v=%%APP_VERSION%%"
onerror="console.warn('CM6 not available — falling back to textarea')"></script>
<link rel="stylesheet" href="vendor/codemirror/codemirror.bundle.css?v=%%APP_VERSION%%"
onerror="this.remove()">
```
**Graceful degradation:** If the bundle fails to load (disconnected environment without vendored copy, build issue, etc.), `window.CM` is undefined and all integration points fall back to plain `<textarea>`. Each factory callsite checks:
```javascript
if (window.CM) {
CM.chatInput(container, opts);
} else {
// existing textarea behavior — unchanged
}
```
This means CM6 is a progressive enhancement, not a hard dependency.
---
## SW Cache Considerations
Add `vendor/codemirror/` to the SW exclusion list (same fix as the extensions path discussed earlier):
```javascript
if (url.pathname.includes('/api/') ||
url.pathname.includes('/ws') ||
url.pathname.includes('/branding/') ||
url.pathname.includes('/extensions/') ||
url.pathname.includes('/vendor/codemirror/') ||
event.request.method !== 'GET') {
return;
}
```
Or, more broadly, exclude all of `/vendor/` if you want vendor updates to always bypass the SW cache. The core vendors (marked, purify) are small enough that network-first is fine.
Alternatively, since the bundle is versioned via `?v=%%APP_VERSION%%` and only changes on new builds, it could safely be in `SHELL_FILES` for precaching — unlike extensions which are user-editable at runtime. Either approach works.
---
## Theme Integration
CM6's `oneDark` theme for dark mode. For light mode, the CM6 default is clean and neutral.
Both need CSS variable overrides to match the app's existing `--bg`, `--bg-2`, `--border`, `--text`, `--accent` palette:
```javascript
const switchboardTheme = EditorView.theme({
'&': {
backgroundColor: 'var(--bg)',
color: 'var(--text)',
fontSize: '14px',
},
'.cm-content': {
fontFamily: 'var(--mono)',
caretColor: 'var(--accent)',
},
'.cm-cursor': {
borderLeftColor: 'var(--accent)',
},
'&.cm-focused .cm-selectionBackground, .cm-selectionBackground': {
backgroundColor: 'var(--accent-muted, rgba(99,102,241,0.2))',
},
'.cm-gutters': {
backgroundColor: 'var(--bg-2)',
color: 'var(--text-3)',
borderRight: '1px solid var(--border)',
},
}, { dark: document.body.classList.contains('dark-theme') });
```
This goes into the bundle so it's applied automatically.
---
## File Layout
```
src/
editor/
package.json # CM6 deps + esbuild
build.mjs # esbuild script
index.mjs # bundle entrypoint, exports window.CM
theme.mjs # switchboard theme overrides
chat-input.mjs # chatInput() factory
code-editor.mjs # codeEditor() factory
js/
... # existing app code (unchanged)
vendor/
marked.min.js # existing
purify.min.js # existing
codemirror/ # gitignored — built by Docker
codemirror.bundle.js
codemirror.bundle.css
```
The `src/editor/` directory is self-contained. It has its own `package.json` and `node_modules` (inside Docker only). No npm dependencies bleed into the main `src/js/` code. The build output lands in `vendor/codemirror/` which is gitignored like the other Docker-built vendors.
---
## Dockerfile Changes
```dockerfile
# Stage 1: vendor libs (existing)
FROM node:20-alpine AS vendor
# ... existing marked, purify, mermaid, katex extraction ...
# Stage 2: CM6 bundle (NEW)
FROM node:20-alpine AS cm6-build
WORKDIR /build
COPY src/editor/ ./
RUN npm ci
RUN node build.mjs
# Stage 3: nginx (existing, with additions)
FROM nginx:1-alpine
# ... existing setup ...
COPY src/ /usr/share/nginx/html/
COPY --from=vendor /vendor/ /usr/share/nginx/html/vendor/
COPY --from=cm6-build /build/dist/ /usr/share/nginx/html/vendor/codemirror/
# ... rest unchanged ...
```
The `vendor` and `cm6-build` stages run in parallel (Docker BuildKit), so build time impact is minimal.
---
## Migration Checklist
### Phase 1: Bundle + Extension Editor
- [ ] Create `src/editor/` with package.json, build.mjs, index.mjs
- [ ] Implement `CM.codeEditor()` factory
- [ ] Update Dockerfile.frontend with cm6-build stage
- [ ] Replace extension editor textareas in `admin-handlers.js`
- [ ] Remove manual Tab key handler (CM6 handles it)
- [ ] Verify save/load round-trip with CM6 `.getValue()`
- [ ] Add graceful fallback if `window.CM` undefined
### Phase 2: Chat Input
- [ ] Implement `CM.chatInput()` factory with markdown mode
- [ ] Wire Enter=send, Shift+Enter=newline keybindings
- [ ] Integrate with `updateInputTokens()` via onChange callback
- [ ] Update `chat.js`, `attachments.js`, `tokens.js` to use CM API
- [ ] Auto-grow height behavior matching current textarea
- [ ] Test paste handling (plain text, code, attachments)
- [ ] Test mobile/touch input
### Phase 3: Polish
- [ ] Theme integration with CSS variables
- [ ] Dark/light mode switching (listen for theme toggle)
- [ ] Editor keymap preference UI (Standard / Vim / Emacs) in appearance settings
- [ ] Vim/Emacs modes on code editor + extension editor only (not chat input)
- [ ] Add to `SHELL_FILES` in sw.js (or exclude from SW cache)
- [ ] Update `debug.js` snapshot to include CM6 version
- [ ] Documentation in ARCHITECTURE.md
---
## Bundle Size Estimate
Based on CM6 package sizes (minified + tree-shaken by esbuild):
| Package | Approx Size |
|---------|-------------|
| @codemirror/view + state + commands | ~90KB |
| @codemirror/language + highlight | ~30KB |
| lang-markdown + lang-javascript + lang-json | ~40KB |
| lang-go + lang-sql + lang-html + lang-css + lang-yaml | ~50KB |
| lang-python + lang-rust | ~20KB |
| theme-one-dark | ~5KB |
| search + autocomplete | ~20KB |
| @replit/codemirror-vim + emacs | ~40KB |
| **Total (minified)** | **~295KB** |
| **Gzipped** | **~90KB** |
For comparison: mermaid.min.js is ~1.2MB. This is modest.
---
## Resolved Decisions
1. **Language modes** — Ship Python, Rust, and YAML upfront alongside Go, JS, JSON, SQL, HTML, CSS, and Markdown. Covers the team's stack. Additional modes are a one-line import + rebuild.
2. **Local dev build script** — Provide a `scripts/build-editor.sh` that runs the esbuild step standalone, shared by both Dockerfile.frontend and the unified Dockerfile. Developers can also run it locally to get the CM6 bundle without a full Docker build.
3. **Vim/Emacs keybindings** — Ship both. Exposed as a user preference toggle (default: standard). Applied via `CM.keybindings.vim` / `CM.keybindings.emacs` as CM6 extensions at editor creation time.
---
## Build Script
Shared script used by both Dockerfiles and local dev:
```bash
#!/bin/sh
# scripts/build-editor.sh — Build CM6 bundle
# Usage: ./scripts/build-editor.sh [output-dir]
#
# Defaults to src/vendor/codemirror/ for local dev.
# Dockerfiles pass /build/dist/ as the output dir.
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
EDITOR_DIR="${SCRIPT_DIR}/../src/editor"
OUTPUT_DIR="${1:-${SCRIPT_DIR}/../src/vendor/codemirror}"
cd "${EDITOR_DIR}"
# Install if needed (CI/Docker will already have node_modules)
[ -d node_modules ] || npm ci
mkdir -p "${OUTPUT_DIR}"
node build.mjs --outdir="${OUTPUT_DIR}"
echo "✅ CM6 bundle → ${OUTPUT_DIR}"
```
Dockerfile usage:
```dockerfile
FROM node:20-alpine AS cm6-build
WORKDIR /build
COPY src/editor/ /build/src/editor/
COPY scripts/build-editor.sh /build/scripts/build-editor.sh
RUN cd /build/src/editor && npm ci
RUN sh /build/scripts/build-editor.sh /build/dist
```
---
## Keybinding Preference
Add to user appearance preferences (alongside dark/light theme toggle):
```
Editor keybindings: [Standard ▾] [Vim] [Emacs]
```
Stored in `localStorage` under `cs-appearance`:
```javascript
const prefs = JSON.parse(localStorage.getItem('cs-appearance') || '{}');
// prefs.editorKeymap = 'standard' | 'vim' | 'emacs'
```
Applied at editor creation time:
```javascript
const keymapExt = [];
if (prefs.editorKeymap === 'vim') keymapExt.push(CM.keybindings.vim());
else if (prefs.editorKeymap === 'emacs') keymapExt.push(CM.keybindings.emacs());
CM.codeEditor(target, {
language: 'javascript',
extensions: keymapExt,
// ...
});
```
**Note:** Vim/Emacs modes apply only to the code editor and extension editor surfaces. The chat input always uses standard keybindings to avoid confusing Enter-to-send behavior with modal editing.

773
docs/archive/DESIGN-SURFACES.md
View File

@@ -1,773 +0,0 @@
# DESIGN — Surface & Extension Architecture
**Status:** Accepted (Phases 1–2 implemented in v0.25.1)
**Scope:** Primitives, Components, Surfaces, Extension hooks, Themes
**Depends on:** v0.22.5 (Go template engine), v0.22.7 (ChatPane), v0.25.1 (audit)
**Informs:** v0.22.8+, EXTENSIONS.md rewrite, ARCHITECTURE.md update, ICD-API, ICD-SURFACE
**Note:** For current package/surface authoring, see [PACKAGES.md](PACKAGES.md) and [EXTENSION-SURFACES.md](EXTENSION-SURFACES.md). For workflow surfaces, see [WORKFLOW-PACKAGES.md](WORKFLOW-PACKAGES.md).
---
## Overview
The UI is a four-layer architecture: Primitives, Components, Surfaces,
and Banners. Each layer has a single responsibility and a strict
dependency direction — primitives know nothing, components compose
primitives, surfaces compose components, banners are platform chrome
outside the surface boundary.
Extensions participate at every layer through six defined hooks.
Admin-installed extensions are privileged (full DOM, direct API access,
own routes). Themes are pure CSS custom property overrides that
propagate through every layer uniformly.
This document defines the layer contracts, extension hook system,
display content model, trust boundaries, and theme architecture.
---
## Definitions
A **surface** is a full-page layout — a composition of components with
a route, a data loader, and a boot script. Chat is a surface. The
editor is a surface. A custom triage intake form is a surface.
An **extension** is a package — a manifest plus code plus assets that
participates in the platform through hooks. An extension can do one,
some, or all of:
- Register tools the LLM can call (Hook 5: Tool Bridge)
- Transform content during streaming (Hook 2: Stream Processing)
- Enhance rendered messages (Hook 3: Post-Render)
- Attach visual content to messages (Hook 4: Display Content)
- Inject panels or sections into existing surfaces (Hook 6: Surface Injection)
- Create entirely new surfaces with their own routes (Surface Registration)
The five **core surfaces** (Chat, Editor, Notes, Admin, Settings) exist
without extensions. They are built from the same primitive and component
layers that extensions use, but they are registered directly in Go
rather than through a manifest.
An extension that creates a surface gets its own route (`/s/:slug`),
its own data loader, and the full primitive/component library. An
extension that only registers a post-render hook (like a block
renderer) doesn't create any surfaces at all.
**Examples:**
| Extension | Creates Surface? | Hooks Used |
|-----------|-----------------|------------|
| Mermaid renderer | No | Post-Render |
| KaTeX renderer | No | Post-Render |
| Calculator tool | No | Tool Bridge |
| Image Generator | Yes (`/s/gallery`) | Tool Bridge, Stream Processing, Post-Render, Display Content, Surface |
| Custom Dashboard | Yes (`/s/dashboard`) | Surface, Surface Injection (admin section) |
---
## Existing Extension Mapping
The current extension mechanisms map directly into this architecture:
| Current Mechanism | Architecture Equivalent |
|-------------------|------------------------|
| Block renderers (`ctx.renderers.register()`) | Hook 3: Post-Render |
| Tool bridge (`ctx.tools.register()`) | Hook 5: Tool Bridge |
| `ctx.ui.toast()`, `ctx.ui.openPreview()` | Primitive layer (unchanged API) |
| `ctx.ui.isDark()`, `ctx.ui.isMobile()` | Theme layer queries |
| `ctx.surfaces.getCurrent()` | Returns `window.__SURFACE__` |
The manifest schema gains optional new fields (`hooks`, `surfaces`,
`surface_injections`, `theme`). Existing fields retain their meaning.
`Extensions.boot()` is the single entry point for extension
initialization on every surface. It loads manifests, registers
renderers, and initializes tool bridges — same sequence, available
on every surface rather than only chat.
---
## Layer Model
```
┌──────────────────────────────────────────────┐
│ Banner (top) │ ← Platform config
├──────────────────────────────────────────────┤
│ │
│ Surface │ ← Layout + lifecycle
│ ┌─────────────┬──────────────────────────┐ │
│ │ Component │ Component │ │ ← Domain-aware
│ │ (ChatPane) │ (NoteEditor) │ │
│ │ ┌─────────┐ │ ┌────────┐ ┌──────────┐ │ │
│ │ │Primitive│ │ │Primitv.│ │Primitive │ │ │ ← Atomic UI
│ │ │ (input) │ │ │ (menu) │ │ (toggle) │ │ │
│ │ └─────────┘ │ └────────┘ └──────────┘ │ │
│ └─────────────┴──────────────────────────┘ │
│ │
├──────────────────────────────────────────────┤
│ Banner (bottom) │ ← Platform config
└──────────────────────────────────────────────┘
```
Four layers, strict dependency direction: Primitives know nothing.
Components use Primitives. Surfaces compose Components. Banners are
global chrome outside the surface boundary.
---
## Layer 1: Primitives
Atomic UI elements. No domain logic. A toggle doesn't know if it's
toggling a KB or a theme — it takes a label, a state, and a callback.
Styled entirely via CSS custom properties (the theme contract). Every
primitive reads from the same property namespace, so theme changes
propagate instantly.
### Catalog
| Primitive | Description |
|-----------|-------------|
| `Input` | Text, textarea, number, password, with label + validation |
| `Select` | Dropdown, single or multi |
| `Toggle` | Boolean switch with label |
| `Checkbox` | Checkbox with label |
| `Button` | Text button, variants: primary, secondary, danger, ghost |
| `IconButton` | Icon-only button with tooltip |
| `ColorPicker` | Color input with hex text field |
| `Menu` | Dropdown or context menu, item list with icons + shortcuts |
| `Dialog` | Modal: confirm, prompt, or custom form content |
| `Toast` | Transient notification: success, error, warning, info |
| `Badge` | Inline label: accent, success, danger, warning, muted |
| `Avatar` | Image circle with upload affordance |
| `Tabs` | Tab bar with content switching |
| `FormGroup` | Label + input + validation message + help text |
| `SectionHeader` | Titled section divider |
| `EmptyState` | Placeholder with icon + message + action |
| `Spinner` | Loading indicator |
| `Table` | Sortable, paginated data table |
### Implementation
Each primitive is a factory function that returns a DOM element (or
attaches to an existing one). No classes, no inheritance — just
functions.
```js
// Example — not prescriptive API, just the pattern:
Primitives.toggle({ label: 'Auto-search', value: true, onChange: fn })
Primitives.menu({ anchor: el, items: [...], onSelect: fn })
Primitives.dialog({ title: 'Confirm', body: el, onConfirm: fn })
```
Current locations: `ui-primitives.js` (factory functions, `esc()`,
`createComponentRegistry()`, `componentMixin()`, `showConfirm()`,
`showPrompt()`, `Theme`, `mountAvatarUpload()`)
and Go template components (`model-select.html`, `team-select.html`,
`file-upload.html`). These converge into one coherent set.
### Go Template Primitives
Some primitives need server-rendered initial state (e.g., model-select
needs the model list, team-select needs the team list). These are Go
template partials that render the HTML + initial data, then JS hydrates
interactivity on load.
```html
{{template "model-select" dict "ID" "channelModel" "Models" .Models "Type" "chat"}}
```
The JS primitive attaches to the server-rendered DOM by ID, not by
replacing it.
---
## Layer 2: Components
Domain-aware compositions of primitives. Each component owns its DOM
subtree, manages its own state, and exposes a clean API for the surface
to interact with.
### Catalog
| Component | Primitives Used | Domain |
|-----------|----------------|--------|
| `ChatPane` | Input, Button, Menu, Spinner, Toast | Channels, Messages, Streaming |
| `NoteEditor` | Input (CM6), Menu, Tabs, Badge | Notes, Wikilinks, Folders |
| `FileTree` | Menu (context), Button, Spinner | Workspaces, Files |
| `ModelSelector` | Select, Badge, Spinner | Models, Capabilities, Health |
| `PersonaPicker` | Select, Badge, Avatar | Personas, Scopes |
| `KBPicker` | Toggle, Badge, Select | Knowledge Bases, Discoverability |
| `ProjectSidebar` | Menu, Tabs, Badge, EmptyState | Projects, Channels, DnD |
| `AuditLog` | Table, Select, Badge | Audit, Filtering, Pagination |
| `CodeEditor` | Input (CM6), Select, Tabs | Workspaces, Languages |
| `SettingsForm` | FormGroup, Toggle, Select, Button | User/Admin Settings |
### Instance Pattern
Components use `createComponentRegistry()` and `componentMixin()` from
`ui-primitives.js` for shared lifecycle (instance tracking, event
listener cleanup, destroy):
```js
const ChatPane = {
...createComponentRegistry('ChatPane'),
create(opts) {
const instance = componentMixin({
id: opts.id || 'pane-' + Date.now(),
messagesEl: opts.messagesEl,
inputEl: opts.inputEl,
channelId: opts.channelId || null,
// ... domain-specific state
// Component-specific teardown (called by mixin's destroy())
_cleanup() {
if (this._cmView) { this._cmView.destroy(); this._cmView = null; }
},
}, ChatPane);
ChatPane._register(instance.id, instance);
return instance;
},
// Custom queries beyond .get(id)
forChannel(channelId) { /* ... */ },
};
```
The mixin provides `_on(el, event, handler)` for tracked event binding
and `destroy()` which calls `_cleanup()` then removes all listeners and
unregisters from the registry. Components that need extra teardown define
`_cleanup()`.
Components can be instantiated multiple times on the same page (editor
has a ChatPane, notes has a ChatPane — independent instances).
### Server-Rendered Shell + JS Hydration
Components have a Go template partial for the server-rendered scaffold:
```html
{{template "chat-pane" dict "ID" "editor"}}
```
This renders the DOM structure with predictable IDs. The JS component
attaches to these IDs on DOMContentLoaded. No client-side DOM
construction for the initial layout.
---
## Layer 3: Surfaces
A surface is a full-page layout that composes components. Each surface
declares:
1. **What components it uses** (ChatPane, NoteEditor, FileTree, etc.)
2. **How they're arranged** (CSS grid/flex layout)
3. **What data it needs on load** (Go data loader)
4. **What JS runs on boot** (script block or module)
### Current Surfaces
| Surface | Components | Data Loader |
|---------|-----------|-------------|
| Chat | ChatPane, ProjectSidebar, ModelSelector, PersonaPicker | channels, personas, models, projects |
| Editor | FileTree, CodeEditor, ChatPane (assist) | workspace, files, models |
| Notes | NoteEditor, NoteGraph, ChatPane (assist) | notes, folders, graph |
| Admin | AuditLog, SettingsForm, Table (various) | users, configs, models, health |
| Settings | SettingsForm, KBPicker, PersonaPicker | profile, preferences, policies |
### Surface Registration
Today: hardcoded in Go (`pageEngine.RenderSurface("chat")`).
Future: manifest-driven. An admin-installed extension declares a
surface in its manifest:
```json
{
"surfaces": [{
"id": "triage-intake",
"route": "/s/triage",
"title": "Triage Intake",
"components": ["chat-pane", "settings-form"],
"data_requires": ["personas", "models"],
"script": "surfaces/triage.js",
"auth": "authenticated"
}]
}
```
The page engine reads registered surfaces, generates routes, assembles
data loaders from the declared requirements, and renders a shell
template that loads the surface's script.
**Route namespace:** Admin-created surfaces live under `/s/:slug` to
avoid collision with core routes.
### Data Loaders
Each surface declares what data it needs. The page engine has a
registry of data providers:
```
"personas" → loads personas for the current user
"models" → loads enabled models with health status
"channels" → loads user's channels
"workspace" → loads workspace by :wsId param
"notes" → loads notes list
...
```
The surface's `data_requires` field pulls from this registry. Data is
injected into the page as `window.__PAGE_DATA__` (existing pattern).
The surface's JS reads from there on boot — no waterfall of API calls
on page load.
---
## Layer 4: Banners
The banner is global chrome outside the surface boundary. It exists
at the top, bottom, or both — configured by the platform admin via
`PUT /admin/settings/banner`.
```json
{
"enabled": true,
"text": "DEVELOPMENT",
"position": "both|top|bottom",
"bg": "#007a33",
"fg": "#ffffff"
}
```
The Go template base layout renders banners before and after the
surface content area. CSS custom properties (`--banner-top-height`,
`--banner-bottom-height`, `--banner-bg`, `--banner-fg`) allow surfaces
to account for banner space without knowing banner state.
Banners are not configurable by extensions or themes. They are a
platform-level trust signal.
---
## Themes
A theme is a set of CSS custom property overrides applied to `<html>`.
```css
[data-theme="corporate"] {
--bg: #f5f5f5;
--text: #1a1a1a;
--accent: #0066cc;
--border: #d1d5db;
--font-ui: 'Inter', sans-serif;
--font-code: 'Fira Code', monospace;
--radius: 4px;
/* ... full property set */
}
```
### What a Theme Can Do
- Override any CSS custom property in the theme contract
- Change colors, fonts, border radii, spacing scale
- Switch between light and dark base palettes
- Apply to every primitive, component, and surface uniformly
### What a Theme Cannot Do
- Add or remove DOM elements
- Execute JavaScript
- Modify component behavior or layout
- Override banner appearance (platform chrome)
- Access APIs or user data
### Theme Contract
The set of CSS custom properties that all primitives read from. This
is the stable API between themes and the UI. Properties are namespaced:
```
--bg, --bg-secondary, --bg-tertiary (backgrounds)
--text, --text-secondary, --text-muted (typography)
--accent, --accent-hover, --accent-muted (interactive)
--border, --border-strong (edges)
--success, --warning, --danger (semantic)
--font-ui, --font-code (typefaces)
--radius, --radius-lg (shapes)
--shadow, --shadow-lg (elevation)
--banner-bg, --banner-fg (read-only, set by platform)
```
Themes are admin-installed. An admin uploads a CSS file that declares
a `[data-theme="name"]` rule set. Users can select from installed
themes in Settings → Appearance.
Built-in themes: `dark` (default), `light`. The system preference
auto-detection (`prefers-color-scheme`) continues to work.
---
## Extension Hooks
Extensions interact with the application through a defined set of
hooks. Each hook has a specific trigger point in the lifecycle and
a clear contract for what the extension receives and returns.
### Hook 1: Pre-Completion
**When:** After the user sends a message, before the API request fires.
**Receives:** The completion request object (model, messages, tools, etc.)
**Returns:** Modified request object (or unmodified to pass through).
**Use case:** Inject additional tools, modify system prompt, add context.
```js
ctx.hooks.preCompletion(request => {
request.tools.push(myCustomTool);
return request;
});
```
### Hook 2: Stream Processing
**When:** On each SSE chunk during streaming.
**Receives:** The chunk (content delta, tool_use, tool_result, etc.)
**Returns:** Modified chunk (or null to suppress).
**Use case:** Transform content, intercept tool calls, accumulate data.
```js
ctx.hooks.streamChunk((chunk, context) => {
if (chunk.type === 'tool_result' && chunk.name === 'image_gen') {
context.displayContent.push({ type: 'image', src: chunk.data.url });
return null; // suppress from text content
}
return chunk;
});
```
### Hook 3: Post-Render
**When:** After a message is rendered into the DOM.
**Receives:** The message container element, the message object.
**Returns:** Nothing (mutates DOM in place).
**Use case:** Add action buttons, wrap elements, enhance display.
This is how extensions compose with each other's output. Extension A
produces an image via display content. Extension B's post-render hook
finds `<img>` elements and wraps them with action buttons. They don't
know about each other — they agree on the DOM contract.
```js
ctx.hooks.postRender((containerEl, message) => {
containerEl.querySelectorAll('img[data-display-content]').forEach(img => {
const actions = document.createElement('div');
actions.className = 'image-actions';
actions.innerHTML = '<button data-action="upscale">Upscale</button>';
img.parentElement.appendChild(actions);
});
});
```
### Hook 4: Display Content
**When:** After a completion finishes (all chunks received).
**Receives:** The assistant message object, accumulated display content.
**Returns:** Display content items to attach to the message.
Display content is **message-scoped, rendered inline, but excluded from
the LLM context window**. It's not in `messages.content` and not sent
back on the next turn.
```
Message
├── content (text — goes to LLM)
├── attachments (files — go to LLM via multimodal assembly)
└── display_content[] ← rendered inline, NOT sent to LLM
├── { type: "image", src: "...", extension_id: "img-gen" }
└── { type: "html", content: "<div>...", extension_id: "editor" }
```
**Storage:** Display content is persisted in a `display_content` JSONB
column on the message (or a junction table). It survives page reload
and is included in `GET /channels/:id/path` responses.
### Hook 5: Tool Bridge
**When:** The LLM invokes a tool registered by the extension.
**Receives:** Tool name and input parameters.
**Returns:** Tool result (string or structured).
Existing mechanism — the WebSocket tool bridge from EXTENSIONS.md.
Tool is registered via manifest, exposed to the LLM through the
tools list, and executed client-side (browser tier) or server-side
(starlark/sidecar tiers).
### Hook 6: Surface Injection
**When:** Surface boot (DOMContentLoaded).
**Receives:** The surface ID and available mount points.
**Returns:** Nothing (attaches to mount points).
Extensions declare which surfaces they target and which mount points
they use:
```json
{
"surface_injections": [{
"surface": "chat",
"mount_point": "side-panel",
"component": "my-panel.js"
}, {
"surface": "admin",
"mount_point": "section",
"section_id": "my-admin-section",
"label": "Image Gen Settings"
}]
}
```
Mount points are declared by each surface template:
```html
<div data-mount="side-panel"></div>
<div data-mount="section" data-section-id="..."></div>
```
---
## Trust Model
Two tiers. The admin is the trust boundary.
### Privileged (Admin-Installed)
| Capability | Allowed |
|-----------|---------|
| Full DOM access | Yes |
| Direct API calls (user's JWT) | Yes |
| Modify other extensions' output (post-render) | Yes |
| Create surfaces (own routes) | Yes |
| Register tools | Yes |
| Access all hook types | Yes |
| Go template data loader | Yes |
| Read `window.__PAGE_DATA__` | Yes |
The admin chose to install it. Same trust model as a VS Code extension
or a WordPress plugin — you trust the publisher.
### Sandboxed (User-Installed, Future)
| Capability | Allowed |
|-----------|---------|
| Shadow DOM / iframe only | Yes |
| API calls through message bridge | Yes (proxied, scoped) |
| Modify other extensions' output | No |
| Create surfaces | No |
| Register tools | Limited (user-scoped) |
| Access hooks | Post-render own output only |
| Go template data loader | No |
| Read `window.__PAGE_DATA__` | No |
Post-1.0 scope. Documented here to confirm the architecture
accommodates it without redesign.
---
## Extension Lifecycle
### Boot Sequence
```
1. Page loads (Go template renders surface shell + banner)
2. Primitives initialize (CSS loaded, JS factories available)
3. Components hydrate (attach to server-rendered DOM)
4. Extensions.boot() — idempotent, runs on every surface
a. Load extension manifests
b. Register hooks (pre-completion, post-render, etc.)
c. Register surface injections for current surface
d. Initialize tool bridges
5. Surface-specific JS runs (app.js for chat, editor-surface.js, etc.)
```
`Extensions.boot()` replaces the current chat-specific
`Extensions.loadAll()` / `Extensions.initAll()` pair and runs on
every surface.
### Manifest Declaration
```json
{
"id": "image-generator",
"name": "AI Image Generator",
"version": "1.0.0",
"tier": "browser",
"hooks": {
"pre_completion": "hooks/pre-completion.js",
"stream_chunk": "hooks/stream.js",
"post_render": "hooks/post-render.js"
},
"tools": [{
"name": "generate_image",
"description": "Generate an image from a text description",
"parameters": { ... }
}],
"surfaces": [{
"id": "image-gallery",
"route": "/s/gallery",
"title": "Image Gallery",
"components": ["chat-pane"],
"data_requires": ["channels"],
"script": "surfaces/gallery.js"
}],
"surface_injections": [{
"surface": "chat",
"mount_point": "message-actions",
"script": "injections/image-actions.js"
}],
"theme": null,
"settings_schema": { ... }
}
```
---
## Display Content — Data Model
### Message Extension
```sql
ALTER TABLE messages ADD COLUMN display_content JSONB DEFAULT '[]';
```
Array of display items:
```json
[
{
"type": "image",
"src": "data:image/png;base64,...",
"alt": "A cat in a spacesuit",
"extension_id": "image-generator",
"metadata": { "model": "dall-e-3", "size": "1024x1024" }
},
{
"type": "html",
"content": "<div class='chart'>...</div>",
"extension_id": "data-viz",
"metadata": {}
}
]
```
### API Contract
Display content is included in message objects returned by
`GET /channels/:id/path` and `GET /channels/:id/messages`.
It is **excluded** from the message array sent to the LLM provider
in `POST /chat/completions`. The completion handler strips
`display_content` when assembling the provider request.
### Rendering
During `renderMessages()`, after the message content is rendered,
each `display_content` item is rendered below the text content:
```html
<div class="message-content">
<p>Here's the image you requested:</p>
</div>
<div class="message-display-content">
<img src="..." alt="..." data-display-content data-extension="image-generator">
</div>
```
The `data-display-content` attribute is the DOM contract that
post-render hooks use to discover display content from any extension.
---
## Migration Path
| Current State | Target State |
|---------------|-------------|
| `ui-primitives.js` (merged, with `esc()`, `createComponentRegistry()`, `componentMixin()`) | ✅ Done (v0.25.1) |
| `ChatPane.create()` + 5 other components using shared mixin | ✅ Done (v0.25.1) |
| 5 Go template routes | Surface registry (hardcoded core, manifest-driven extensions) |
| `ctx.ui.replace()` / `ctx.ui.inject()` | Hook 6: Surface Injection with declared mount points |
| `ctx.surfaces.register()` | Manifest `surfaces` field → page engine route registration |
| Block renderers in `Extensions.initAll()` | `Extensions.boot()` on every surface |
| Message content only (text + attachments) | `display_content` column for render-only visual content |
| Block renderer pipeline (mermaid, katex) | Post-render hook + DOM contract (`data-display-content`) |
| `[data-theme]` with CSS variables | Theme contract: stable CSS custom property namespace |
---
## Implementation Sequence
This does not prescribe version numbers. It describes dependency order.
**Phase 1: Primitive consolidation** ✅ (v0.25.1)
- `ui-primitives-additions.js` deleted, survivors merged into
`ui-primitives.js`
- `esc()` centralized (13 private copies eliminated)
- Badge system unified (24 scattered classes → grouped in primitives.css)
- CSS variable names normalized, duplicate selectors resolved
- `showConfirm()` is the sole confirm pattern (bare `confirm()` purged)
**Phase 2: Component formalization** ✅ (v0.25.1)
- `createComponentRegistry()` + `componentMixin()` extracted
- 6 components (ChatPane, ModelSelector, UserMenu, CodeEditor, FileTree,
NoteEditor) use the shared mixin. ~100 lines of boilerplate eliminated.
- Each component has a Go template partial + JS hydration
- `surface-nav.js` deleted; navigation is native in `ui-core.js`
- Vendor scripts (`marked.min.js`, `purify.min.js`) and `ui-format.js`
moved to `base.html` (loaded once, available to all surfaces)
**Phase 3: Extension hooks**
- `Extensions.boot()` on every surface (v0.22.8)
- Pre-completion and post-render hooks
- Display content data model (message column + render pipeline)
- Stream processing hook
**Phase 4: Surface registry**
- Manifest-driven surface registration
- Dynamic route generation in page engine
- Data loader registry
- Mount point declaration and injection
**Phase 5: Theme system**
- Stable CSS custom property contract
- Admin theme upload
- User theme selection in Settings → Appearance
---
## Open Questions
1. **Display content storage:** JSONB column on messages vs. separate
`message_display_content` table. Column is simpler; table allows
independent lifecycle (delete display content without touching
message). Leaning column for KISS.
2. **Display content size limits:** Base64 images in JSONB could get
large. May need blob storage for display content with URL references
instead of inline data. Deferred until real usage patterns emerge.
3. **Extension ordering:** Post-render hooks from multiple extensions
run in what order? Manifest priority field? Installation order?
Alphabetical? Matters when Extension B adds buttons to Extension A's
images — A must render first.
4. **Surface layout persistence:** If a surface supports resizable
panes (editor: file tree width, chat pane width), where is that
persisted? User settings? Per-workspace? Per-surface?
5. **Extension settings storage:** Extensions need per-user config
(API keys for image gen, preferences for behavior). Current
`UpdateUserExtensionSettings` endpoint exists. Is the schema
sufficient or does it need structured validation from
`settings_schema` in the manifest?

7
docs/archive/README.md
View File

@@ -1,7 +0,0 @@
# Archived Design & Changes Docs
Historical DESIGN-*.md (per-version specs) and CHANGES-*.md (migration guides).
**Active docs**: ARCHITECTURE.md, EXTENSIONS.md, ROADMAP.md, CHANGELOG.md.
**Query**: [Git search](https://git.gobha.me/xcaliber/chat-switchboard/search?q=DESIGN)

1227
docs/archive/ROADMAP.md
View File

File diff suppressed because it is too large Load Diff

186
docs/archive/v0171-sqlite-backend.md
View File

@@ -1,186 +0,0 @@
# v0.17.1 — SQLite Backend
## Overview
Adds SQLite as an alternative database backend alongside PostgreSQL. Enables
single-binary deployments, air-gapped environments, edge nodes, and developer
laptops without requiring a PostgreSQL instance.
## Architecture
```
DB_DRIVER env var
│
├─ "postgres" (default) → database.DialectPostgres → postgres.NewStores()
└─ "sqlite" → database.DialectSQLite → sqlite.NewStores()
Both implement the same 19 store.* interfaces unchanged.
```
### Key Design Decisions
| Decision | Rationale |
|----------|-----------|
| Parallel `store/sqlite/` package | Clean separation, no runtime branching in SQL |
| `?` placeholders (not `$N`) | SQLite native, no conversion overhead |
| JSON text arrays (`'["a","b"]'`) | Replace `TEXT[]`/`UUID[]` + `pq.Array` |
| `json_each()` for array membership | Replace `= ANY(array_col)` |
| App-side `store.NewID()` UUIDs | Replace `DEFAULT gen_random_uuid()` |
| `datetime('now')` | Replace `NOW()` / `CURRENT_TIMESTAMP` |
| `LIKE` fallback for search | Replace `tsvector` / `ts_rank` |
| Feature-gated vector search | pgvector → returns error hint on SQLite |
| `excluded.col` in ON CONFLICT | Replace Postgres `$N` reuse in SET clause |
| WAL mode + single writer | Optimal SQLite concurrency for web apps |
## Files Delivered
### Infrastructure
| File | Purpose |
|------|---------|
| `server/config/config.go` | Added `DBDriver` field + `DB_DRIVER` env var |
| `server/database/dialect.go` | `Dialect` type, `IsPostgres()`, `IsSQLite()` |
| `server/database/database.go` | Dual-driver `Connect()`: postgres + sqlite |
| `server/database/migrate.go` | Dialect-aware migration runner |
| `server/database/migrations/sqlite/001_v017_schema.sql` | Full SQLite schema |
| `server/store/id.go` | `store.NewID()` — app-side UUID generation |
### Store Layer (19 stores)
| File | Lines | Key Transforms |
|------|-------|---------------|
| `store/sqlite/helpers.go` | Query builders with `?`, JSON helpers |
| `store/sqlite/stores.go` | `NewStores()` constructor |
| `store/sqlite/user.go` | `store.NewID()`, `datetime('now')` |
| `store/sqlite/provider.go` | JSONB→TEXT, `store.NewID()` |
| `store/sqlite/catalog.go` | JSONB→TEXT scanning |
| `store/sqlite/persona.go` | `json_each()` for group grants |
| `store/sqlite/policy.go` | `excluded.col` in upsert |
| `store/sqlite/user_settings.go` | COALESCE with `excluded.col` |
| `store/sqlite/channel.go` | Tags as JSON text array |
| `store/sqlite/message.go` | `store.NewID()` |
| `store/sqlite/note.go` | LIKE search fallback, JSON tags |
| `store/sqlite/usage.go` | Dynamic `?` filters |
| `store/sqlite/pricing.go` | `excluded.col` upserts |
| `store/sqlite/extension.go` | `store.NewID()` |
| `store/sqlite/attachment.go` | `store.NewID()` |
| `store/sqlite/knowledge_bases.go` | `IN(?)` replaces `ANY()`, vector search gated |
| `store/sqlite/groups.go` | `store.NewID()` |
| `store/sqlite/resource_grants.go` | `json_each()` for array membership |
| `store/sqlite/team.go` | `store.NewID()`, `excluded.col` |
| `store/sqlite/audit.go` | `store.NewID()` |
| `store/sqlite/global_config.go` | `excluded.col` upsert |
### Integration
| File | Purpose |
|------|---------|
| `server/MAIN_GO_PATCH.md` | Shows exact main.go changes needed |
## Pattern Conversion Reference
### Placeholders
```sql
-- Postgres: WHERE id = $1 AND name = $2
-- SQLite: WHERE id = ? AND name = ?
```
### UUID Generation
```go
// Postgres: RETURNING id (gen_random_uuid() DEFAULT)
// SQLite: obj.ID = store.NewID()
// INSERT INTO table (id, ...) VALUES (?, ...)
// RETURNING created_at (id already set)
```
### Array Types
```go
// Postgres: pq.Array(teamIDs) → team_id = ANY($3)
// SQLite: for _, tid := range teamIDs { args = append(args, tid) }
// team_id IN (?,?,?)
//
// Postgres: pq.Array(&tags) → stored as TEXT[]
// SQLite: ArrayToJSON(tags) → stored as '["a","b"]'
// ScanArray(jsonStr) → read back
```
### Array Membership (granted_groups)
```sql
-- Postgres: gm.group_id = ANY(rg.granted_groups)
-- SQLite: JOIN json_each(rg.granted_groups) je ON je.value = gm.group_id
```
### ON CONFLICT Upserts
```sql
-- Postgres: ON CONFLICT (key) DO UPDATE SET value = $2, updated_by = $3
-- SQLite: ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_by = excluded.updated_by
```
### Timestamps
```sql
-- Postgres: NOW(), CURRENT_TIMESTAMP, TIMESTAMPTZ
-- SQLite: datetime('now'), TEXT (ISO 8601)
```
### Full-Text Search
```sql
-- Postgres: search_vector @@ to_tsquery('english', $1)
-- SQLite: (title LIKE ? OR content LIKE ?) -- per word
```
### Vector Search
```
-- Postgres: c.embedding <=> $1::vector (pgvector cosine distance)
-- SQLite: Feature-gated. Returns error hint.
-- KB ingestion works (stores text chunks).
-- Future: sqlite-vec extension support.
```
## Configuration
```bash
# PostgreSQL (default — no changes needed)
DB_DRIVER=postgres
DATABASE_URL=postgres://user:pass@host:5432/switchboard
# SQLite
DB_DRIVER=sqlite
DATABASE_URL=switchboard.db # file path
DATABASE_URL=:memory: # in-memory (testing)
DATABASE_URL=/data/switchboard.db # absolute path
```
Auto-detection: if `DB_DRIVER` is empty, inferred from `DATABASE_URL` format.
## SQLite Pragmas (set automatically)
| Pragma | Value | Purpose |
|--------|-------|---------|
| `journal_mode` | WAL | Concurrent readers |
| `busy_timeout` | 5000ms | Retry on lock |
| `foreign_keys` | ON | Enforce FK constraints |
## Limitations vs PostgreSQL
| Feature | PostgreSQL | SQLite |
|---------|-----------|--------|
| Vector similarity search | ✅ pgvector | ❌ Feature-gated |
| Full-text search | ✅ tsvector/tsquery | ⚠ LIKE fallback |
| Concurrent writes | ✅ MVCC | ⚠ Single writer (WAL) |
| Array columns | ✅ Native | ⚠ JSON text arrays |
| `NUMERIC(12,6)` precision | ✅ Exact | ⚠ REAL (float64) |
| GIN indexes | ✅ Native | ❌ Not available |
| Partial indexes | ✅ Full support | ✅ Supported |
| RETURNING clause | ✅ Full support | ✅ SQLite 3.35+ |
## go.mod Addition
```
require modernc.org/sqlite v1.34.5 // pure-Go, no CGO required
```
## TODO (follow-up)
- [ ] Wire main.go dialect switch (see MAIN_GO_PATCH.md)
- [ ] Add `modernc.org/sqlite` to go.mod
- [ ] CI matrix: run integration tests against both drivers
- [ ] SQLite-specific integration test suite
- [ ] sqlite-vec extension support for vector search (optional)
- [ ] Benchmark: SQLite vs PostgreSQL for typical workloads

52
server/config/config.go
View File

@@ -25,11 +25,6 @@ type Config struct {
// Seed users (dev/test only) — CSV: "user:pass:role,user2:pass2:role2"
SeedUsers string
// Seed providers (dev/test only) — CSV: "provider:api_key[:name]"
// Shortcuts: "openai:sk-xxx", "anthropic:sk-ant-xxx", "openrouter:sk-or-xxx"
// Skips if provider with same name exists (idempotent on restart).
SeedProviders string
// API key encryption (required for v0.9.4+)
// Used to derive AES-256 key for global/team provider API keys.
// Personal keys use per-user UEK (derived from password).
@@ -51,37 +46,6 @@ type Config struct {
S3Prefix string // optional key prefix within bucket (e.g. "switchboard/")
S3ForcePathStyle bool // use path-style URLs (required for MinIO, most self-hosted)
// Extraction pipeline (v0.12.0+)
// EXTRACTION_MODE: "inline" (in-process) or "sidecar" (shared PVC watcher).
// EXTRACTION_CONCURRENCY: max concurrent extraction jobs (default 3).
ExtractionMode string
ExtractionConcurrency int
// Workspace indexing (v0.21.2)
// WORKSPACE_INDEXING_ENABLED: global kill switch for workspace file indexing (default true).
// WORKSPACE_INDEX_CONCURRENCY: max concurrent indexing goroutines (default 2).
WorkspaceIndexingEnabled bool
WorkspaceIndexConcurrency int
// Provider auto-disable (v0.22.4)
// PROVIDER_AUTO_DISABLE_THRESHOLD: consecutive "down" hourly windows before a
// provider is automatically deactivated. Default 3. Set to 0 to disable.
ProviderAutoDisableThreshold int
// Config-file provider types (v0.29.1)
// PROVIDER_TYPES_FILE: path to JSON file defining custom provider types
// (e.g., Ollama, LiteLLM, vLLM). Empty = no custom types.
ProviderTypesFile string
// SESSION_EXPIRY_DAYS: anonymous sessions older than this with no messages
// are cleaned up by the background session sweeper. Default 30. Set to 0
// to disable cleanup.
SessionExpiryDays int
// WORKFLOW_STALE_HOURS: workflow instances with no activity for this many
// hours are marked 'stale'. Default 72 (3 days). Set to 0 to disable.
WorkflowStaleHours int
// Structured logging (v0.33.0)
// LOG_FORMAT: "text" (default, backward-compatible) or "json" (structured).
// LOG_LEVEL: "debug", "info" (default), "warn", "error".
@@ -131,7 +95,6 @@ func Load() *Config {
AdminPassword: getEnv("SWITCHBOARD_ADMIN_PASSWORD", ""),
AdminEmail: getEnv("SWITCHBOARD_ADMIN_EMAIL", ""),
SeedUsers: getEnv("SEED_USERS", ""),
SeedProviders: getEnv("SEED_PROVIDERS", ""),
EncryptionKey: getEnv("ENCRYPTION_KEY", ""),
StorageBackend: getEnv("STORAGE_BACKEND", ""),
@@ -143,19 +106,6 @@ func Load() *Config {
S3SecretKey: getEnv("S3_SECRET_KEY", ""),
S3Prefix: getEnv("S3_PREFIX", ""),
S3ForcePathStyle: getEnv("S3_FORCE_PATH_STYLE", "true") == "true",
ExtractionMode: getEnv("EXTRACTION_MODE", "inline"),
ExtractionConcurrency: getEnvInt("EXTRACTION_CONCURRENCY", 3),
WorkspaceIndexingEnabled: getEnvBool("WORKSPACE_INDEXING_ENABLED", true),
WorkspaceIndexConcurrency: getEnvInt("WORKSPACE_INDEX_CONCURRENCY", 2),
ProviderAutoDisableThreshold: getEnvInt("PROVIDER_AUTO_DISABLE_THRESHOLD", 3),
ProviderTypesFile: getEnv("PROVIDER_TYPES_FILE", ""),
SessionExpiryDays: getEnvInt("SESSION_EXPIRY_DAYS", 30),
WorkflowStaleHours: getEnvInt("WORKFLOW_STALE_HOURS", 72),
LogFormat: getEnv("LOG_FORMAT", "text"),
LogLevel: getEnv("LOG_LEVEL", "info"),
@@ -205,7 +155,7 @@ func resolveDatabaseURL() string {
port := getEnv("POSTGRES_PORT", "5432")
user := getEnv("POSTGRES_USER", "")
pass := getEnv("POSTGRES_PASSWORD", "")
db := getEnv("POSTGRES_DB", "chat_switchboard")
db := getEnv("POSTGRES_DB", "switchboard_core")
ssl := getEnv("POSTGRES_SSLMODE", "disable")
return fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=%s", user, pass, host, port, db, ssl)
}

4
server/handlers/workflows.go
View File

@@ -293,10 +293,6 @@ func (h *WorkflowHandler) Publish(c *gin.Context) {
snapshotStages := make([]stageSnapshot, len(stages))
for i, st := range stages {
snapshotStages[i] = stageSnapshot{WorkflowStage: st}
if st.PersonaID != nil && h.stores.Personas != nil {
grants, _ := h.stores.Personas.GetToolGrants(c.Request.Context(), *st.PersonaID)
snapshotStages[i].ToolGrants = grants
}
}
snapshot := map[string]interface{}{

18
server/main.go
View File

@@ -63,13 +63,6 @@ func main() {
// v0.29.1: Config-file provider types (Ollama, LiteLLM, vLLM, etc.)
if cfg.ProviderTypesFile != "" {
if err != nil {
log.Printf("⚠️ Failed to load custom provider types from %s: %v", cfg.ProviderTypesFile, err)
} else if n > 0 {
log.Printf(" 📦 Loaded %d custom provider type(s) from %s", n, cfg.ProviderTypesFile)
}
}
var stores store.Stores
uekCache := crypto.NewUEKCache()
@@ -568,8 +561,6 @@ func main() {
protected.GET("/profile", settings.GetProfile)
protected.PUT("/profile", settings.UpdateProfile)
protected.POST("/profile/password", settings.ChangePassword)
protected.POST("/profile/avatar", settings.UploadAvatar)
protected.DELETE("/profile/avatar", settings.DeleteAvatar)
protected.GET("/settings", settings.GetSettings)
protected.PUT("/settings", settings.UpdateSettings)
@@ -664,11 +655,6 @@ func main() {
teamScoped.GET("/groups", groupH.ListTeamGroups)
// Team providers
teamScoped.GET("/providers", teams.ListTeamProviders)
teamScoped.POST("/providers", teams.CreateTeamProvider)
teamScoped.PUT("/providers/:id", teams.UpdateTeamProvider)
teamScoped.DELETE("/providers/:id", teams.DeleteTeamProvider)
teamScoped.GET("/providers/:id/models", teams.ListTeamProviderModels)
// Team connections (v0.38.1)
teamScoped.GET("/connections", teams.ListTeamConnections)
@@ -697,8 +683,6 @@ func main() {
teamScoped.PUT("/personas/:id/knowledge-bases", teamPersonas.SetTeamPersonaKBs) // v0.17.0
teamScoped.GET("/personas/:id/tool-grants", teamPersonas.GetTeamPersonaToolGrants) // v0.28.0
teamScoped.PUT("/personas/:id/tool-grants", teamPersonas.SetTeamPersonaToolGrants) // v0.28.0
teamScoped.POST("/personas/:id/avatar", teamPersonas.UploadTeamPersonaAvatar) // v0.28.0
teamScoped.DELETE("/personas/:id/avatar", teamPersonas.DeleteTeamPersonaAvatar) // v0.28.0
teamScoped.GET("/roles", teamRoles.ListTeamRoles)
teamScoped.PUT("/roles/:role", teamRoles.UpdateTeamRole)
@@ -787,8 +771,6 @@ func main() {
admin.POST("/personas", personaAdm.CreateAdminPersona)
admin.PUT("/personas/:id", personaAdm.UpdateAdminPersona)
admin.DELETE("/personas/:id", personaAdm.DeleteAdminPersona)
admin.POST("/personas/:id/avatar", func(c *gin.Context) { handlers.UploadPersonaAvatar(stores.Personas, c) })
admin.DELETE("/personas/:id/avatar", func(c *gin.Context) { handlers.DeletePersonaAvatar(stores.Personas, c) })
admin.GET("/personas/:id/knowledge-bases", personaAdm.GetPersonaKBs) // v0.17.0
admin.PUT("/personas/:id/knowledge-bases", personaAdm.SetPersonaKBs) // v0.17.0
admin.GET("/personas/:id/tool-grants", personaAdm.GetPersonaToolGrants) // v0.25.0

211
server/pages/loaders.go
View File

@@ -10,23 +10,6 @@ import (
"switchboard-core/store"
)
// ── Types for template data ──────────────────
// ModelOption is a flat model for template dropdowns.
type ModelOption struct {
ProviderConfigID string `json:"provider_config_id"`
ProviderName string `json:"provider_name"`
ModelID string `json:"model_id"`
DisplayName string `json:"display_name"`
ModelType string `json:"model_type"`
}
// ProviderOption for template dropdowns.
type ProviderOption struct {
ID string `json:"id"`
Name string `json:"name"`
}
// TeamOption for template dropdowns.
type TeamOption struct {
ID string `json:"id"`
@@ -36,16 +19,6 @@ type TeamOption struct {
IsActive bool `json:"is_active"`
}
// ProviderDetail is an enriched provider for the providers table.
type ProviderDetail struct {
ID string `json:"id"`
Name string `json:"name"`
Provider string `json:"provider"`
Scope string `json:"scope"`
Endpoint string `json:"endpoint"`
ModelCount int `json:"model_count"`
}
// ProviderTypeOption for the provider type dropdown.
type ProviderTypeOption struct {
ID string `json:"id"`
@@ -86,10 +59,7 @@ type RoleSelection struct {
type AdminPageData struct {
Section string `json:"section"`
Category string `json:"category"`
Providers []ProviderOption `json:"providers"`
ProviderDetails []ProviderDetail `json:"provider_details,omitempty"`
ProviderTypes []ProviderTypeOption `json:"provider_types,omitempty"`
Models []ModelOption `json:"models"`
Teams []TeamOption `json:"teams"`
Users []UserRow `json:"users,omitempty"`
Roles []RoleConfig `json:"roles"`
@@ -102,11 +72,6 @@ type ChatPageData struct {
ChatID string `json:"chat_id,omitempty"`
}
// NotesPageData provides context for the notes surface shell.
type NotesPageData struct {
NoteID string `json:"NoteID,omitempty"`
}
// SettingsPageData is what the settings surface templates receive.
// Feature gates (BYOK, personas) moved to sw.auth.policies in v0.37.19.
type SettingsPageData struct {
@@ -122,18 +87,10 @@ type TeamAdminPageData struct {
ConfigSections []ConfigSectionEntry `json:"config_sections,omitempty"` // v0.38.3
}
// ProjectsPageData is what the projects surface receives.
type ProjectsPageData struct {
ProjectID string `json:"project_id"`
}
func (e *Engine) registerLoaders() {
e.RegisterLoader("admin", e.adminLoader)
e.RegisterLoader("chat", e.chatLoader)
e.RegisterLoader("notes", e.notesLoader)
e.RegisterLoader("settings", e.settingsLoader)
e.RegisterLoader("team-admin", e.teamAdminLoader)
e.RegisterLoader("projects", e.projectsLoader)
}
// Used for manifest validation — DataRequires entries must match a key here.
@@ -160,119 +117,6 @@ func (e *Engine) adminLoader(c *gin.Context, s store.Stores) (any, error) {
Category: sectionCategory(section),
}
// ── Providers (global scope) ─────────────
if s.Providers != nil {
configs, err := s.Providers.ListGlobal(ctx)
if err != nil {
log.Printf("[pages/admin] Failed to list providers: %v", err)
} else {
for _, pc := range configs {
name := pc.Name
if name == "" {
name = pc.Provider
}
data.Providers = append(data.Providers, ProviderOption{
ID: pc.ID,
Name: name,
})
}
}
}
// ── Teams ────────────────────────────────
if s.Teams != nil {
teams, err := s.Teams.List(ctx)
if err != nil {
log.Printf("[pages/admin] Failed to list teams: %v", err)
} else {
for _, t := range teams {
data.Teams = append(data.Teams, TeamOption{
ID: t.ID,
Name: t.Name,
Description: t.Description,
MemberCount: t.MemberCount,
IsActive: t.IsActive,
})
}
}
}
// ── Full model catalog with model_type ───
// This is what fixes bug #1: all models from all providers,
// including embedding models, with their model_type intact.
if s.Catalog != nil {
entries, err := s.Catalog.ListAllGlobal(ctx)
if err != nil {
log.Printf("[pages/admin] Failed to list catalog: %v", err)
} else {
provNames := make(map[string]string, len(data.Providers))
for _, p := range data.Providers {
provNames[p.ID] = p.Name
}
for _, entry := range entries {
mt := entry.ModelType
if mt == "" {
mt = "chat"
}
dn := entry.DisplayName
if dn == "" {
dn = entry.ModelID
}
data.Models = append(data.Models, ModelOption{
ProviderConfigID: entry.ProviderConfigID,
ProviderName: provNames[entry.ProviderConfigID],
ModelID: entry.ModelID,
DisplayName: dn,
ModelType: mt,
})
}
}
}
// ── Section-specific data ────────────────
switch section {
case "roles":
data.Roles = e.loadRolesConfig(ctx, s)
case "routing":
if s.RoutingPolicies != nil {
policies, err := s.RoutingPolicies.ListAll(ctx)
if err == nil {
data.Policies = policies
}
}
case "providers":
data.ProviderTypes = loadProviderTypes()
data.ProviderDetails = e.loadProviderDetails(ctx, s, data.Providers, data.Models)
case "users":
data.Users = e.loadUsers(ctx, s)
}
// v0.38.3: Discover extension config sections for admin surface
data.ConfigSections = configSectionsForSurface(ctx, s, "admin")
return data, nil
}
// loadRolesConfig reads current role assignments.
func (e *Engine) loadRolesConfig(ctx context.Context, s store.Stores) []RoleConfig {
roleNames := []string{"utility", "embedding"}
roles := make([]RoleConfig, 0, len(roleNames))
if s.GlobalConfig == nil {
for _, name := range roleNames {
roles = append(roles, RoleConfig{Name: name})
}
return roles
}
raw, err := s.GlobalConfig.Get(ctx, "model_roles")
if err != nil || raw == nil {
for _, name := range roleNames {
roles = append(roles, RoleConfig{Name: name})
}
return roles
}
rolesMap, ok := raw["roles"].(map[string]any)
if !ok {
rolesMap = raw
@@ -300,13 +144,6 @@ func (e *Engine) loadRolesConfig(ctx context.Context, s store.Stores) []RoleConf
return roles
}
// ── Chat loader ──────────────────────────────
func (e *Engine) chatLoader(c *gin.Context, s store.Stores) (any, error) {
chatID := c.Param("chatID")
return &ChatPageData{ChatID: chatID}, nil
}
// ── Admin helpers ────────────────────────────
// sectionCategory maps an admin section to its category tab.
@@ -336,42 +173,6 @@ func loadProviderTypes() []ProviderTypeOption {
return out
}
// loadProviderDetails enriches providers with model counts.
func (e *Engine) loadProviderDetails(ctx context.Context, s store.Stores, provs []ProviderOption, models []ModelOption) []ProviderDetail {
// Count models per provider
counts := make(map[string]int)
for _, m := range models {
counts[m.ProviderConfigID]++
}
// Get full provider configs for endpoint/scope
if s.Providers == nil {
return nil
}
configs, err := s.Providers.ListGlobal(ctx)
if err != nil {
log.Printf("[pages/admin] Failed to list provider details: %v", err)
return nil
}
out := make([]ProviderDetail, 0, len(configs))
for _, pc := range configs {
name := pc.Name
if name == "" {
name = pc.Provider
}
out = append(out, ProviderDetail{
ID: pc.ID,
Name: name,
Provider: pc.Provider,
Scope: pc.Scope,
Endpoint: pc.Endpoint,
ModelCount: counts[pc.ID],
})
}
return out
}
// loadUsers returns the user list for the admin users table.
func (e *Engine) loadUsers(ctx context.Context, s store.Stores) []UserRow {
if s.Users == nil {
@@ -396,13 +197,6 @@ func (e *Engine) loadUsers(ctx context.Context, s store.Stores) []UserRow {
return out
}
// ── Notes loader ─────────────────────────────
func (e *Engine) notesLoader(c *gin.Context, s store.Stores) (any, error) {
noteID := c.Param("noteId")
return &NotesPageData{NoteID: noteID}, nil
}
// ── Settings loader ──────────────────────────
// v0.22.7: Reads feature gates from GlobalConfig to control
// which nav links/tabs are visible (BYOK, User Personas).
@@ -430,11 +224,6 @@ func (e *Engine) settingsLoader(c *gin.Context, s store.Stores) (any, error) {
}, nil
}
func (e *Engine) projectsLoader(c *gin.Context, s store.Stores) (any, error) {
projectID := c.Param("id")
return &ProjectsPageData{ProjectID: projectID}, nil
}
// ── Config section discovery (v0.38.3) ───────
//
// Packages declare a config_section in their manifest. This helper scans

7
server/pages/pages.go
View File

@@ -749,12 +749,7 @@ func (e *Engine) RenderWorkflowLanding() gin.HandlerFunc {
if data.FirstStageMode == "" {
data.FirstStageMode = "custom"
}
if stages[0].PersonaID != nil {
if p, err := e.stores.Personas.GetByID(ctx, *stages[0].PersonaID); err == nil {
data.PersonaName = p.Name
data.PersonaIcon = p.Icon
}
}
// Persona lookup deferred — personas are extensions now
}
// Check for existing active session